编写并将float数组附加到C ++中hdf5文件中的唯一数据集

Question

我正在处理文件数量，文件的每个处理将输出几千个浮点数组，我将所有文件的数据存储在一个大型数据集中的单个hdf5中，以便进一步处理。

目前我很困惑如何将我的数据附加到hdf5文件中。（在上面的代码中注释）在上面的2个for循环中，正如你所看到的，我想一次将1维float数组追加到hdf5中，而不是整个东西。我的数据是以TB为单位，我们只能将数据附加到文件中。

有几个问题：

在这种情况下如何附加数据？我必须使用什么样的功能？
现在，我有fdim [0] = 928347543，我已尝试将HDF5的无限标志放入，但运行时执行抱怨。有没有办法做到这一点？我不想每次计算出的数据;有没有办法简单地继续添加数据，而不关心fdim的价值？

或者这不可能吗？

编辑：

我一直在关注Simon的建议，目前这里是我的更新代码：

hid_t desFi5;
hid_t fid1;
hid_t propList;
hsize_t fdim[2];

desFi5 = H5Fcreate(saveFilePath, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);

fdim[0] = 3;
fdim[1] = 1;//H5S_UNLIMITED;

fid1 = H5Screate_simple(2, fdim, NULL);

cout << "----------------------------------Space done
";

propList = H5Pcreate( H5P_DATASET_CREATE);

H5Pset_layout( propList, H5D_CHUNKED );

int ndims = 2;
hsize_t chunk_dims[2];
chunk_dims[0] = 3;
chunk_dims[1] = 1;

H5Pset_chunk( propList, ndims, chunk_dims );

cout << "----------------------------------Property done
";

hid_t dataset1 = H5Dcreate( desFi5, "des", H5T_NATIVE_FLOAT, fid1, H5P_DEFAULT, propList, H5P_DEFAULT);

cout << "----------------------------------Dataset done
";

bufi = new float*[1];
bufi[0] = new float[3];
bufi[0][0] = 0;
bufi[0][1] = 1;
bufi[0][2] = 2;

//hyperslab
hsize_t start[2] = {0,0};
hsize_t stride[2] = {1,1};
hsize_t count[2] = {1,1};
hsize_t block[2] = {1,3};

H5Sselect_hyperslab( fid1, H5S_SELECT_OR, start, stride, count, block);     
cout << "----------------------------------hyperslab done
";   

H5Dwrite(dataset1, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL, H5P_DEFAULT, *bufi);

fdim[0] = 3;
fdim[1] = H5S_UNLIMITED;    // COMPLAINS HERE
H5Dset_extent( dataset1, fdim );

cout << "----------------------------------extent done
";

//hyperslab2
hsize_t start2[2] = {1,0};
hsize_t stride2[2] = {1,1};
hsize_t count2[2] = {1,1};
hsize_t block2[2] = {1,3};

H5Sselect_hyperslab( fid1, H5S_SELECT_OR, start2, stride2, count2, block2);     
cout << "----------------------------------hyperslab2 done
";  

H5Dwrite(dataset1, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL, H5P_DEFAULT, *bufi);

cout << "----------------------------------H5Dwrite done
";        
H5Dclose(dataset1);
cout << "----------------------------------dataset closed
";   
H5Pclose( propList );   
cout << "----------------------------------property list closed
"; 
H5Sclose(fid1); 
cout << "----------------------------------dataspace fid1 closed
";    
H5Fclose(desFi5);       
cout << "----------------------------------desFi5 closed
";

我目前的输出是：

bash-3.2$ ./hdf5AppendTest.out
----------------------------------Space done
----------------------------------Property done
----------------------------------Dataset done
----------------------------------hyperslab done
HDF5-DIAG: Error detected in HDF5 (1.8.10) thread 0:
  #000: /home/hdftest/snapshots-bin-hdf5_1_8_10/current/src/H5D.c line 1103 in H5Dset_extent(): unable to set extend dataset
    major: Dataset
    minor: Unable to initialize object
  #001: /home/hdftest/snapshots-bin-hdf5_1_8_10/current/src/H5Dint.c line 2179 in H5D__set_extent(): unable to modify size of data space
    major: Dataset
    minor: Unable to initialize object
  #002: /home/hdftest/snapshots-bin-hdf5_1_8_10/current/src/H5S.c line 1874 in H5S_set_extent(): dimension cannot exceed the existing maximal size (new: 18446744073709551615 max: 1)
    major: Dataspace
    minor: Bad value
----------------------------------extent done
----------------------------------hyperslab2 done
----------------------------------H5Dwrite done
----------------------------------dataset closed
----------------------------------property list closed
----------------------------------dataspace fid1 closed
----------------------------------desFi5 closed

目前，我发现使用H5Dset_extent设置无限制仍然会在运行时导致问题。（有问题的函数在上面的代码中用//COMPLAINS HERE标记。）我已经得到了Simon指定的块数据，那么这里的问题是什么？

另一方面，没有H5Dset_extent，我可以编写一个[0,1,2]的测试数组，但是我们怎样才能将输出上面的代码作为测试数组放到文件中：

[0, 1, 2]
[0, 1, 2]
[0, 1, 2]
[0, 1, 2]
...
...

回想一下：这只是一个测试数组，实际数据更大，我无法将整个内容保存在RAM中，所以我必须一次一个地将数据放在第一部分。

编辑2：

我更多地遵循了西蒙的建议。这是关键部分：

hsize_t n = 3, p = 1;
float *bufi_data = new float[n * p];
float ** bufi = new float*[n];
for (hsize_t i = 0; i < n; ++i){
    bufi[i] = &bufi_data[i * n];
}

bufi[0][0] = 0.1;
bufi[0][1] = 0.2;
bufi[0][2] = 0.3;

//hyperslab
hsize_t start[2] = {0,0};
hsize_t count[2] = {3,1};

H5Sselect_hyperslab( fid1, H5S_SELECT_SET, start, NULL, count, NULL);
cout << "----------------------------------hyperslab done
";   

H5Dwrite(dataset1, H5T_NATIVE_FLOAT, H5S_ALL, fid1, H5P_DEFAULT, *bufi);

bufi[0][0] = 0.4;
bufi[0][1] = 0.5;
bufi[0][2] = 0.6;

hsize_t fdimNew[2];
fdimNew[0] = 3;
fdimNew[1] = 2;
H5Dset_extent( dataset1, fdimNew );

cout << "----------------------------------extent done
";

//hyperslab2
hsize_t start2[2] = {0,0}; //PROBLEM
hsize_t count2[2] = {3,1};

H5Sselect_hyperslab( fid1, H5S_SELECT_SET, start2, NULL, count2, NULL);     
cout << "----------------------------------hyperslab2 done
";  

H5Dwrite(dataset1, H5T_NATIVE_FLOAT, H5S_ALL, fid1, H5P_DEFAULT, *bufi);

从上面，我得到了hdf5的以下输出：

0.4 0.5 0.6
  0   0   0

在进一步用start2和count2进行实验后，我发现这些变量只影响bufi的起始指数和递增指数。它根本不会移动我的数据集的写入索引的位置。

回想一下：最终结果必须是：

0.1 0.2 0.3
0.4 0.5 0.6

此外，它必须是bufi而不是*bufi为H5Dwrite，西蒙，因为bufi给了我完全随机的数字。

更新3：

对于Simon建议的选择部分：

hsize_t start[2] = {0, 0};
hsize_t count[2] = {1, 3};

hsize_t start[2] = {1, 0};
hsize_t count[2] = {1, 3};

这些将发出以下错误：

HDF5-DIAG: Error detected in HDF5 (1.8.10) thread 0:
  #000: /home/hdftest/snapshots-bin-hdf5_1_8_10/current/src/H5Dio.c line 245 in H5Dwrite(): file selection+offset not within extent
    major: Dataspace
    minor: Out of range

我想，count[2]应该是{3,1}，而不是{1,3}？而对于start[2]，如果我不将其设置为{0,0}，它将始终大声喊出上面的错误。

你确定这是对的吗？

编写并将float数组附加到C ++中hdf5文件中的唯一数据集

演练working code

First buffer

Second buffer