HDF5-DIAG：在 HDF5 (1.8.11) 中检测到错误

Posted 2023-03-11

【中文标题】HDF5-DIAG：在 HDF5 (1.8.11) 中检测到错误

【英文标题】：HDF5-DIAG: Error detected in HDF5 (1.8.11)

【发布时间】：2015-03-17 21:13:17

【问题描述】：

我正在尝试在 caffe 中加载 hdf5，但它不起作用。我检查了路径，甚至可以使用查看器查看 hdf 文件。一切正常，但 caffe 似乎无法加载。

我使用这样的 python 脚本编写 hdf5，其中 X 和标签是 numpy 数组。

f = h5py.File("facialkp.hd5", "w")
f.create_dataset("data", data=X,  compression="gzip", compression_opts=4)
f.create_dataset("label", data=labels,  compression="gzip", compression_opts=4)

这是整个问题。

root@pbu-OptiPlex-740-Enhanced:/home/pbu/Desktop# ./facialkp_train.sh
I0119 19:55:50.779419  9905 caffe.cpp:103] Use CPU.
I0119 19:55:51.152626  9905 caffe.cpp:107] Starting Optimization
I0119 19:55:51.152817  9905 solver.cpp:32] Initializing solver from parameters: 
test_iter: 1000
test_interval: 1000
base_lr: 0.01
display: 1000
max_iter: 10000
lr_policy: "step"
gamma: 0.1
momentum: 0.9
weight_decay: 0.0005
stepsize: 5000
snapshot: 10000
snapshot_prefix: "/home/pbu/Desktop/tmp"
solver_mode: CPU
net: "/home/pbu/Desktop/facialkp.prototxt"
I0119 19:55:51.152936  9905 solver.cpp:67] Creating training net from net file: /home/pbu/Desktop/facialkp.prototxt
I0119 19:55:51.153211  9905 net.cpp:39] Initializing net from parameters: 
name: "LogReg"
layers 
  top: "data"
  top: "label"
  name: "data"
  type: HDF5_DATA
  hdf5_data_param 
    source: "facialkp.hd5"
    batch_size: 10
  
  include 
    phase: TRAIN
  

layers 
  bottom: "data"
  top: "ip"
  name: "ip"
  type: INNER_PRODUCT
  inner_product_param 
    num_output: 30
  

layers 
  bottom: "ip"
  bottom: "label"
  top: "loss"
  name: "loss"
  type: EUCLIDEAN_LOSS

state 
  phase: TRAIN

I0119 19:55:51.153457  9905 net.cpp:67] Creating Layer data
I0119 19:55:51.153486  9905 net.cpp:356] data -> data
I0119 19:55:51.153524  9905 net.cpp:356] data -> label
I0119 19:55:51.153560  9905 net.cpp:96] Setting up data
I0119 19:55:51.153580  9905 hdf5_data_layer.cpp:57] Loading filename from facialkp.hd5
I0119 19:55:51.171277  9905 hdf5_data_layer.cpp:69] Number of files: 19511
I0119 19:55:51.171358  9905 hdf5_data_layer.cpp:29] Loading HDF5 file�HDF
HDF5-DIAG: Error detected in HDF5 (1.8.11) thread 139901797366336:
  #000: ../../../src/H5F.c line 1586 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: ../../../src/H5F.c line 1275 in H5F_open(): unable to open file: time = Mon Jan 19 19:55:51 2015
, name = '�HDF', tent_flags = 0
    major: File accessibilty
    minor: Unable to open file
  #002: ../../../src/H5FD.c line 987 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
  #003: ../../../src/H5FDsec2.c line 343 in H5FD_sec2_open(): unable to open file: name = '�HDF', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0
    major: File accessibilty
    minor: Unable to open file
E0119 19:55:51.172713  9905 hdf5_data_layer.cpp:32] Failed opening HDF5 file�HDF
I0119 19:55:51.173208  9905 hdf5_data_layer.cpp:81] output data size: 10,0,0,0
I0119 19:55:51.173262  9905 net.cpp:103] Top shape: 10 0 0 0 (0)
I0119 19:55:51.173285  9905 net.cpp:103] Top shape: 10 0 0 0 (0)
I0119 19:55:51.173342  9905 net.cpp:67] Creating Layer ip
I0119 19:55:51.173368  9905 net.cpp:394] ip <- data
I0119 19:55:51.173400  9905 net.cpp:356] ip -> ip
I0119 19:55:51.173432  9905 net.cpp:96] Setting up ip
F0119 19:55:51.173466  9905 blob.cpp:72] Check failed: data_ 
*** Check failure stack trace: ***
    @     0x7f3d674c8daa  (unknown)
    @     0x7f3d674c8ce4  (unknown)
    @     0x7f3d674c86e6  (unknown)
    @     0x7f3d674cb687  (unknown)
    @           0x45ee6e  caffe::Blob<>::mutable_cpu_data()
    @           0x4b4d74  caffe::ConstantFiller<>::Fill()
    @           0x4bb6c9  caffe::InnerProductLayer<>::LayerSetUp()
    @           0x491431  caffe::Net<>::Init()
    @           0x492d3e  caffe::Net<>::Net()
    @           0x458b00  caffe::Solver<>::InitTrainNet()
    @           0x459db6  caffe::Solver<>::Init()
    @           0x459f16  caffe::Solver<>::Solver()
    @           0x419c40  caffe::GetSolver<>()
    @           0x416064  train()
    @           0x410a51  main
    @     0x7f3d630c8ec5  (unknown)
    @           0x414bb7  (unknown)
    @              (nil)  (unknown)
Aborted (core dumped)
root@pbu-OptiPlex-740-En

【问题讨论】：

X和labels的形状是什么？

【参考方案1】：

已解决:)

我创建了一个文本文件，将真实 .hd5 文件的路径放在其中。 caffe prototxt 文件指向文本文件并且可以正常工作:)

 hdf5_data_param 
   source: "train.txt"
   batch_size: 10
  

train.txt 包含行..

facialkp.hd5