caffe代码阅读8: Data_layers的实现细节(各个数据读取层的实现细节) 2016.3.25-28
Posted xizero00
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了caffe代码阅读8: Data_layers的实现细节(各个数据读取层的实现细节) 2016.3.25-28相关的知识,希望对你有一定的参考价值。
一、Data_layers.hpp文件的作用简介
Data_layers.hpp在目前caffe的master分支中已经不能存在了,分散到各个文件中去了。
而之前是存在于cafferoot\include\caffe中。现在已经变成了各个类的名称的头文件了。这里做个提醒
首先给出这个文件中所包含的几个与数据读取有关的类。
分别为:
BaseDataLayer
数据层的基类,继承自通用的类Layer
Batch
Batch实际上就是一个data_和label_类标
BasePrefetchingDataLayer
是预取层的基类,继承自BaseDataLayer和InternalThread,包含能够读取一批数据的能力
DataLayer
DataLayer才是主角,继承自BasePrefetchingDataLayer
使用DataReader来进行数据共享,从而实现并行化
DummyDataLayer
该类是继承自Layer,通过Filler产生数据
HDF5DataLayer
从HDF5中读取,继承自Layer
HDF5OutputLayer
将数据写入到HDF5文件,继承自Layer
ImageDataLayer
从图像文件中读取数据,这个应该比较常用,继承自BasePrefetchingDataLayer
MemoryDataLayer
从内存中读取数据,这里指已经从数据文件或者图像文件中读取到了数据,然后输入到该层,继承自BaseDataLayer
WindowDataLayer
从图像文件的窗口获取数据,需要指定窗口数据文件,继承自BasePrefetchingDataLayer
二、Data_layers文件的的详细介绍
上述类虽然在同一个头文件中进行的定义,但是却都是在不同的cpp文件进行的实现。
下面给出类的实现文件
BaseDataLayer和BasePrefetchingDataLayer
对应于:
base_data_layer.cpp
base_data_layer.cu
DataLayer
对应于:
data_layer.cpp
DummyDataLayer
对应于:
dummy_data_layer.cpp
HDF5DataLayer
HDF5OutputLayer
对应于:
hdf5_data_layer.cpp
hdf5_data_layer.cu
以及
hdf5_output_layer.cpp
hdf5_output_layer.cu
ImageDataLayer
对应于:
image_data_layer.cpp
MemoryDataLayer
对应于:
memory_data_layer.cpp
WindowDataLayer
对应于
window_data_layer.cpp
接下来对这些类进行详细阐述:
(1)BaseDataLayer的类定义以及实现如下:
/** * @brief Provides base for data layers that feed blobs to the Net. * * TODO(dox): thorough documentation for Forward and proto params. * 数据层的基类 */ template <typename Dtype> class BaseDataLayer : public Layer<Dtype> { public: // 显式构造函数 explicit BaseDataLayer(const LayerParameter& param); // LayerSetUp: implements common data layer setup functionality, and calls // DataLayerSetUp to do special data layer setup for individual layer types. // This method may not be overridden except by the BasePrefetchingDataLayer. // 该函数只能被BasePrefetchingDataLayer层进行重载 virtual void LayerSetUp(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top); // Data layers should be shared by multiple solvers in parallel // 数据是否需要给多个并行solver进行共享 virtual inline bool ShareInParallel() const { return true; } // 数据层的初始化 virtual void DataLayerSetUp(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) {} // 数据层是没有输入的(即bottoms),所以reshape只是形式 // Data layers have no bottoms, so reshaping is trivial. virtual void Reshape(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) {} virtual void Backward_cpu(const vector<Blob<Dtype>*>& top, const vector<bool>& propagate_down, const vector<Blob<Dtype>*>& bottom) {} virtual void Backward_gpu(const vector<Blob<Dtype>*>& top, const vector<bool>& propagate_down, const vector<Blob<Dtype>*>& bottom) {} protected: // 对输入的数据进行变换的参数,这其中包括是否需要mirror,是否需要crop // 是否需要减去meanfile,是否需要scale TransformationParameter transform_param_; // 实际执行数据变换类的指针(一个Transform函数加上参数即可完成对数据的变换,参数是数据哈) shared_ptr<DataTransformer<Dtype> > data_transformer_; bool output_labels_; };
具体的实现:
// 构造函数就是初始化数据变换参数 template <typename Dtype> BaseDataLayer<Dtype>::BaseDataLayer(const LayerParameter& param) : Layer<Dtype>(param), transform_param_(param.transform_param()) { } // 初始化的时候根据top的大小来确定,如果是1表明只输出数据,而不输出类标 template <typename Dtype> void BaseDataLayer<Dtype>::LayerSetUp(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) { if (top.size() == 1) { output_labels_ = false; } else { output_labels_ = true; } // 初始化一个DataTransformer实例,便于对数据进行预处理 data_transformer_.reset( new DataTransformer<Dtype>(transform_param_, this->phase_)); // 初始化种子 data_transformer_->InitRand(); // The subclasses should setup the size of bottom and top // 执行数据层的初始化 DataLayerSetUp(bottom, top); }
(2)BasePrefetchingDataLayer类的定义以及实现如下:
BasePrefetchingDataLayer类的定义如下:
// BasePrefetchingDataLayer层是继承于BaseDataLayer的 // 是预取层的基类 template <typename Dtype> class BasePrefetchingDataLayer : public BaseDataLayer<Dtype>, public InternalThread { public: explicit BasePrefetchingDataLayer(const LayerParameter& param); // LayerSetUp: implements common data layer setup functionality, and calls // DataLayerSetUp to do special data layer setup for individual layer types. // This method may not be overridden. void LayerSetUp(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top); virtual void Forward_cpu(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top); virtual void Forward_gpu(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top); // Prefetches batches (asynchronously if to GPU memory) static const int PREFETCH_COUNT = 3; protected: virtual void InternalThreadEntry(); // 多了load_batch函数,该函数是纯虚函数,继承该函数的类都需要实现的 virtual void load_batch(Batch<Dtype>* batch) = 0; // 还有prefetch数组,prefetch_free_,prefetch_full_ Batch<Dtype> prefetch_[PREFETCH_COUNT]; BlockingQueue<Batch<Dtype>*> prefetch_free_; BlockingQueue<Batch<Dtype>*> prefetch_full_; Blob<Dtype> transformed_data_; }; BasePrefetchingDataLayer类的具体实现如下: // 构造函数,初始化预取的队列,free和full template <typename Dtype> BasePrefetchingDataLayer<Dtype>::BasePrefetchingDataLayer( const LayerParameter& param) : BaseDataLayer<Dtype>(param), prefetch_free_(), prefetch_full_() { for (int i = 0; i < PREFETCH_COUNT; ++i) { prefetch_free_.push(&prefetch_[i]); } } // 进行层的初始化 template <typename Dtype> void BasePrefetchingDataLayer<Dtype>::LayerSetUp( const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) { // 首先执行基类BaseDataLayer的层初始化 BaseDataLayer<Dtype>::LayerSetUp(bottom, top); // Before starting the prefetch thread, we make cpu_data and gpu_data // calls so that the prefetch thread does not accidentally make simultaneous // cudaMalloc calls when the main thread is running. In some GPUs this // seems to cause failures if we do not so. // 在开启预取线程的时候,需要让cpu数据和gpu数据分配空间 // 这样才能够避免在某些GPU上出现问题 // 首先是CPU for (int i = 0; i < PREFETCH_COUNT; ++i) { prefetch_[i].data_.mutable_cpu_data(); if (this->output_labels_) { prefetch_[i].label_.mutable_cpu_data(); } } #ifndef CPU_ONLY // 然后是GPU if (Caffe::mode() == Caffe::GPU) { for (int i = 0; i < PREFETCH_COUNT; ++i) { prefetch_[i].data_.mutable_gpu_data(); if (this->output_labels_) { prefetch_[i].label_.mutable_gpu_data(); } } } #endif DLOG(INFO) << "Initializing prefetch"; // 初始化随机数种子 this->data_transformer_->InitRand(); // 开启线程 StartInternalThread(); DLOG(INFO) << "Prefetch initialized."; } // 在StartInternalThread开启线程后就会执行下面自己定义的函数 // 这个就是自己定义的函数,让线程去执行的 template <typename Dtype> void BasePrefetchingDataLayer<Dtype>::InternalThreadEntry() { #ifndef CPU_ONLY cudaStream_t stream; if (Caffe::mode() == Caffe::GPU) { // 创建非阻塞流 CUDA_CHECK(cudaStreamCreateWithFlags(&stream, cudaStreamNonBlocking)); } #endif try { while (!must_stop()) { // 弹出一个batch Batch<Dtype>* batch = prefetch_free_.pop(); // 装载batch load_batch(batch); #ifndef CPU_ONLY if (Caffe::mode() == Caffe::GPU) { // 如果GPU模式开始,则推送到GPU batch->data_.data().get()->async_gpu_push(stream); // 检查是否成功 CUDA_CHECK(cudaStreamSynchronize(stream)); } #endif // 将装好的batch压入full队列 prefetch_full_.push(batch); } } catch (boost::thread_interrupted&) { // Interrupted exception is expected on shutdown } #ifndef CPU_ONLY if (Caffe::mode() == Caffe::GPU) { // 销毁流 CUDA_CHECK(cudaStreamDestroy(stream)); } #endif } template <typename Dtype> void BasePrefetchingDataLayer<Dtype>::Forward_cpu( const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) { // 传递的时候是从full队列中弹出一个数据 Batch<Dtype>* batch = prefetch_full_.pop("Data layer prefetch queue empty"); // Reshape to loaded data. // 根据batch的形状改变数据形状 top[0]->ReshapeLike(batch->data_); // Copy the data // 将batch数据复制到top[0] caffe_copy(batch->data_.count(), batch->data_.cpu_data(), top[0]->mutable_cpu_data()); DLOG(INFO) << "Prefetch copied"; if (this->output_labels_) { // 输出类标的话 // Reshape to loaded labels. // 根据batch中类标的形状改变top[1]的形状 top[1]->ReshapeLike(batch->label_); // Copy the labels. // 复制类标到top[1] caffe_copy(batch->label_.count(), batch->label_.cpu_data(), top[1]->mutable_cpu_data()); } // 将该batch压入free队列 prefetch_free_.push(batch); } // 如果没有GPU的话则在BasePrefetchingDataLayer类中生成一个Forward函数 // 该函数并不前传,而是直接报错 #ifdef CPU_ONLY STUB_GPU_FORWARD(BasePrefetchingDataLayer, Forward); #endif // 初始化层 INSTANTIATE_CLASS(BaseDataLayer); INSTANTIATE_CLASS(BasePrefetchingDataLayer);
(3)DataLayer类的定义以及实现如下:
数据层的主要功能是:
首先给出类的定义
// DataLayer才是主角,继承自BasePrefetchingDataLayer template <typename Dtype> class DataLayer : public BasePrefetchingDataLayer<Dtype> { public: explicit DataLayer(const LayerParameter& param); virtual ~DataLayer(); virtual void DataLayerSetUp(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top); // DataLayer uses DataReader instead for sharing for parallelism // 多了下面几个 virtual inline bool ShareInParallel() const { return false; } virtual inline const char* type() const { return "Data"; } virtual inline int ExactNumBottomBlobs() const { return 0; } virtual inline int MinTopBlobs() const { return 1; } virtual inline int MaxTopBlobs() const { return 2; } protected: virtual void load_batch(Batch<Dtype>* batch); DataReader reader_; };
具体的实现如下:
#ifdef USE_OPENCV #include <opencv2/core/core.hpp> #endif // USE_OPENCV #include <stdint.h> #include <string> #include <vector> #include "caffe/common.hpp" #include "caffe/data_layers.hpp" #include "caffe/layer.hpp" #include "caffe/proto/caffe.pb.h" #include "caffe/util/benchmark.hpp" #include "caffe/util/io.hpp" namespace caffe { // 初始化DataReader,层参数 template <typename Dtype> DataLayer<Dtype>::DataLayer(const LayerParameter& param) : BasePrefetchingDataLayer<Dtype>(param), reader_(param) { } // 析构函数停止内部线程 template <typename Dtype> DataLayer<Dtype>::~DataLayer() { this->StopInternalThread(); } // 数据层的初始化 template <typename Dtype> void DataLayer<Dtype>::DataLayerSetUp(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) { // 从层参数中读取batch_size const int batch_size = this->layer_param_.data_param().batch_size(); // Read a data point, and use it to initialize the top blob. // 从reader_中获取一个数据 Datum& datum = *(reader_.full().peek()); // Use data_transformer to infer the expected blob shape from datum. // 用数据来推断blob的形状存放到top_shape vector<int> top_shape = this->data_transformer_->InferBlobShape(datum); this->transformed_data_.Reshape(top_shape); // Reshape top[0] and prefetch_data according to the batch_size. // 既然获取了数据的形状(channel,height,width),那么这里再设置一下batch_size // top_shape[0]=batch_size // top_shape[1]=channel // top_shape[2]=height // top_shape[3]=width top_shape[0] = batch_size; // 根据形状设置top[0]的形状 top[0]->Reshape(top_shape); // 设置预取数据的形状 for (int i = 0; i < this->PREFETCH_COUNT; ++i) { this->prefetch_[i].data_.Reshape(top_shape); } LOG(INFO) << "output data size: " << top[0]->num() << "," << top[0]->channels() << "," << top[0]->height() << "," << top[0]->width(); // label // 如果输出类标的话则把top[1]的形状也弄一下 if (this->output_labels_) { vector<int> label_shape(1, batch_size); top[1]->Reshape(label_shape); for (int i = 0; i < this->PREFETCH_COUNT; ++i) { this->prefetch_[i].label_.Reshape(label_shape); } } } // This function is called on prefetch thread // 这个函数是在自己定义的线程执行函数内部执行的 template<typename Dtype> void DataLayer<Dtype>::load_batch(Batch<Dtype>* batch) { CPUTimer batch_timer; batch_timer.Start(); double read_time = 0; double trans_time = 0; CPUTimer timer; CHECK(batch->data_.count()); CHECK(this->transformed_data_.count()); // Reshape according to the first datum of each batch // on single input batches allows for inputs of varying dimension. // 意思是像以下这种做法这样的话,每个batch的数据的维度可以不一样 // 从参数文件获取batch_size const int batch_size = this->layer_param_.data_param().batch_size(); // 获取第一个数据 Datum& datum = *(reader_.full().peek()); // Use data_transformer to infer the expected blob shape from datum. // 使用第一个数据推断blob的形状 vector<int> top_shape = this->data_transformer_->InferBlobShape(datum); this->transformed_data_.Reshape(top_shape); // Reshape batch according to the batch_size. top_shape[0] = batch_size; batch->data_.Reshape(top_shape); // top_data存数据 Dtype* top_data = batch->data_.mutable_cpu_data(); Dtype* top_label = NULL; // suppress warnings about uninitialized variables // top_label存类标 if (this->output_labels_) { top_label = batch->label_.mutable_cpu_data(); } // 对这批数据进行处理 for (int item_id = 0; item_id < batch_size; ++item_id) { timer.Start(); // get a datum Datum& datum = *(reader_.full().pop("Waiting for data")); read_time += timer.MicroSeconds(); timer.Start(); // Apply data transformations (mirror, scale, crop...) // 对于给定批的数据获取offset,这里调用的是给定batchid,然后获取offset int offset = batch->data_.offset(item_id); this->transformed_data_.set_cpu_data(top_data + offset); this->data_transformer_->Transform(datum, &(this->transformed_data_)); // Copy label. // 复制类标 if (this->output_labels_) { top_label[item_id] = datum.label(); } // 数据传输时间 trans_time += timer.MicroSeconds(); // 将数据指针压到free队列 reader_.free().push(const_cast<Datum*>(&datum)); } timer.Stop(); batch_timer.Stop(); DLOG(INFO) << "Prefetch batch: " << batch_timer.MilliSeconds() << " ms."; DLOG(INFO) << " Read time: " << read_time / 1000 << " ms."; DLOG(INFO) << "Transform time: " << trans_time / 1000 << " ms."; } INSTANTIATE_CLASS(DataLayer); REGISTER_LAYER_CLASS(Data); } // namespace caffe
(4)DummyDataLayer类的定义与实现介绍:
Dummy数据层的主要功能就是根据所给定的Filler产生数据,然后前向传
首先给出定义
/** * @brief Provides data to the Net generated by a Filler. * * TODO(dox): thorough documentation for Forward and proto params. * 该类是继承自Layer,通过Filler产生数据 */ template <typename Dtype> class DummyDataLayer : public Layer<Dtype> { public: explicit DummyDataLayer(const LayerParameter& param) : Layer<Dtype>(param) {} virtual void LayerSetUp(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top); // Data layers should be shared by multiple solvers in parallel virtual inline bool ShareInParallel() const { return true; } // Data layers have no bottoms, so reshaping is trivial. virtual void Reshape(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) {} virtual inline const char* type() const { return "DummyData"; } virtual inline int ExactNumBottomBlobs() const { return 0; } virtual inline int MinTopBlobs() const { return 1; } protected: virtual void Forward_cpu(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top); virtual void Backward_cpu(const vector<Blob<Dtype>*>& top, const vector<bool>& propagate_down, const vector<Blob<Dtype>*>& bottom) {} virtual void Backward_gpu(const vector<Blob<Dtype>*>& top, const vector<bool>& propagate_down, const vector<Blob<Dtype>*>& bottom) {} vector<shared_ptr<Filler<Dtype> > > fillers_; vector<bool> refill_; };
接下来给出详细的定义:
首先给出FillerParameter的定义,里面指定了值的类型,值是啥,最小是啥,最大是啥,平均值、方差是啥、是否稀疏、以及将扇入个数还是扇出个数还是所有的加起来求均值作为分母
message FillerParameter { // The filler type. optional string type = 1 [default = 'constant']; optional float value = 2 [default = 0]; // the value in constant filler optional float min = 3 [default = 0]; // the min value in uniform filler optional float max = 4 [default = 1]; // the max value in uniform filler optional float mean = 5 [default = 0]; // the mean value in Gaussian filler optional float std = 6 [default = 1]; // the std value in Gaussian filler // The expected number of non-zero output weights for a given input in // Gaussian filler -- the default -1 means don't perform sparsification. optional int32 sparse = 7 [default = -1]; // Normalize the filler variance by fan_in, fan_out, or their average. // Applies to 'xavier' and 'msra' fillers. enum VarianceNorm { FAN_IN = 0; FAN_OUT = 1; AVERAGE = 2; } optional VarianceNorm variance_norm = 8 [default = FAN_IN]; }
再看看该类的参数
</pre><pre name="code" class="plain">// DummyDataLayer fills any number of arbitrarily shaped blobs with random // (or constant) data generated by "Fillers" (see "message FillerParameter"). message DummyDataParameter { // This layer produces N >= 1 top blobs. DummyDataParameter must specify 1 or N // shape fields, and 0, 1 or N data_fillers. // // If 0 data_fillers are specified, ConstantFiller with a value of 0 is used. // If 1 data_filler is specified, it is applied to all top blobs. If N are // specified, the ith is applied to the ith top blob. repeated FillerParameter data_filler = 1; repeated BlobShape shape = 6; // 4D dimensions -- deprecated. Use "shape" instead. repeated uint32 num = 2; repeated uint32 channels = 3; repeated uint32 height = 4; repeated uint32 width = 5; }
接下来给出具体的实现
#include <vector> #include "caffe/filler.hpp" #include "caffe/layer.hpp" #include "caffe/vision_layers.hpp" namespace caffe { template <typename Dtype> void DummyDataLayer<Dtype>::LayerSetUp(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) { // 输出有几个 const int num_top = top.size(); // 获取该层的参数 const DummyDataParameter& param = this->layer_param_.dummy_data_param(); // 有几个filler const int num_data_filler = param.data_filler_size(); // 检查filler的个数,要么为0、1、或者等于输出的个数 CHECK(num_data_filler == 0 || num_data_filler == 1 || num_data_filler == num_top) << "Number of data fillers must be 0, 1 or equal to the number of tops: " << num_top << "; you specified " << num_data_filler << " data fillers."; // 判断是否全部为0 const bool legacy_dims = param.num_size() || param.channels_size() || param.height_size() || param.width_size(); // 下面就是检查参数是不是满足要求,1或者0或者等于num_top if (legacy_dims) {// 如果不是全部为0 CHECK_EQ(0, param.shape_size()) << "Both shape and legacy fields were specified"; // Using deprecated 4D output dim specifiers. CHECK(param.num_size() == 1 || param.num_size() == num_top) << "Must specify 'num' once, or once per top blob " << "(" << num_top << "); specified " << param.num_size() << "."; CHECK(param.channels_size() == 1 || param.channels_size() == num_top) << "Must specify 'channels' once, or once per top blob " << "(" << num_top << "); specified " << param.channels_size() << "."; CHECK(param.height_size() == 1 || param.height_size() == num_top) << "Must specify 'height' once, or once per top blob " << "(" << num_top << "); specified " << param.height_size() << "."; CHECK(param.width_size() == 1 || param.width_size() == num_top) << "Must specify 'width' once, or once per top blob " << "(" << num_top << "); specified " << param.width_size() << "."; } else { CHECK(param.shape_size() == 1 || param.shape_size() == num_top) << "Must specify 'shape' once, or once per top blob " << "(" << num_top << "); specified " << param.shape_size() << "."; } // refill_[i] tells Forward i whether or not to actually refill top Blob i. // If refill_[i] is false, Forward does nothing for Blob i. We use this to // avoid wastefully refilling "constant" Blobs in every forward pass. // We first fill refill_ in with the INVERSE of its final values. // The first time we run Forward from the LayerSetUp method, we'll fill only // Blobs for which refill_ is normally false. These Blobs will never be // filled again. // refill_表明是不是需要填充Blob,如果refill_[i]=false,那么就不会Blob i做任何事 // refill_.clear(); fillers_.clear(); // 要么是0,要么是1 if (num_data_filler <= 1) { // 定义了生成数据的参数 // 比如均值、方差等,详细请看其定义 FillerParameter filler_param; if (num_data_filler == 0) { // 如果没有指定,那么就是常数值填充 filler_param.set_type("constant"); filler_param.set_value(0); } else { // 否则复制filler到filler_param filler_param.CopyFrom(param.data_filler(0)); } // Refill on each iteration iff not using a constant filler, // but use the inverse of this rule for the first run. // 如果 refill_.resize(1); refill_[0] = (strcmp(filler_param.type().c_str(), "constant") == 0); fillers_.resize(1); // 实例化填充器 fillers_[0].reset(GetFiller<Dtype>(filler_param)); } else {// 如果等于=num_top refill_.resize(num_top); fillers_.resize(num_top); for (int i = 0; i < num_top; ++i) { fillers_[i].reset(GetFiller<Dtype>(param.data_filler(i))); // Refill on each iteration iff not using a constant filler, // but use the inverse of this rule for the first run. refill_[i] = (strcmp(param.data_filler(i).type().c_str(), "constant") == 0); } } // 改变形状 for (int i = 0; i < num_top; ++i) { if (legacy_dims) { const int num = (param.num_size() == 1) ? param.num(0) : param.num(i); const int channels = (param.channels_size() == 1) ? param.channels(0) : param.channels(i); const int height = (param.height_size() == 1) ? param.height(0) : param.height(i); const int width = (param.width_size() == 1) ? param.width(0) : param.width(i); top[i]->Reshape(num, channels, height, width); } else { const int shape_index = (param.shape_size() == 1) ? 0 : i; top[i]->Reshape(param.shape(shape_index)); } } // Run Forward once, with refill_ inverted, to fill the constant Blobs. // 执行forward_cpu this->Forward(bottom, top); // Invert the inverted refill_ values to refill the desired (non-constant) // Blobs in every usual forward pass. for (int i = 0; i < refill_.size(); ++i) { refill_[i] = !refill_[i]; } } // Forward里调用了该函数 template <typename Dtype> void DummyDataLayer<Dtype>::Forward_cpu(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) { // 调用fillers_来进行錐ill for (int i = 0; i < top.size(); ++i) { const int filler_id = (fillers_.size() > 1) ? i : 0; if (refill_[filler_id]) { fillers_[filler_id]->Fill(top[i]); } } } // 初始化类 // 注册类 INSTANTIATE_CLASS(DummyDataLayer); REGISTER_LAYER_CLASS(DummyData); } // namespace caffe
(5)HDF5DataLayer类的定义以及实现如下:
HDF5数据层的主要功能是从给定的HDF5文件列表读取数据,然后设置top,即向前传播的数据。
首先给出类的定义:
template <typename Dtype> class HDF5DataLayer : public Layer<Dtype> { public: explicit HDF5DataLayer(const LayerParameter& param) : Layer<Dtype>(param) {} virtual ~HDF5DataLayer(); virtual void LayerSetUp(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top); // Data layers should be shared by multiple solvers in parallel virtual inline bool ShareInParallel() const { return true; } // Data layers have no bottoms, so reshaping is trivial. virtual void Reshape(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) {} virtual inline const char* type() const { return "HDF5Data"; } virtual inline int ExactNumBottomBlobs() const { return 0; } virtual inline int MinTopBlobs() const { return 1; } protected: virtual void Forward_cpu(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top); virtual void Forward_gpu(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top); virtual void Backward_cpu(const vector<Blob<Dtype>*>& top, const vector<bool>& propagate_down, const vector<Blob<Dtype>*>& bottom) {} virtual void Backward_gpu(const vector<Blob<Dtype>*>& top, const vector<bool>& propagate_down, const vector<Blob<Dtype>*>& bottom) {} // 从HDF5文件读取数据 virtual void LoadHDF5FileData(const char* filename); std::vector<std::string> hdf_filenames_; unsigned int num_files_; unsigned int current_file_; hsize_t current_row_; std::vector<shared_ptr<Blob<Dtype> > > hdf_blobs_; // 存放的是数据的索引,可以对索引进行shuffle std::vector<unsigned int> data_permutation_; // 存放的是文件名字的索引,可以对索引进行shuffle std::vector<unsigned int> file_permutation_; };
接下来给出类的具体实现:
给出实现之前先给出HDF5的操作
头文件:
#ifndef CAFFE_UTIL_HDF5_H_ #define CAFFE_UTIL_HDF5_H_ #include <string> #include "hdf5.h" #include "hdf5_hl.h" #include "caffe/blob.hpp" namespace caffe { // 获取HDF5文件的信息以及数据的维度 template <typename Dtype> void hdf5_load_nd_dataset_helper( hid_t file_id, const char* dataset_name_, int min_dim, int max_dim, Blob<Dtype>* blob); // float类型的获取数据维度和信息的包裹函数 template <typename Dtype> void hdf5_load_nd_dataset( hid_t file_id, const char* dataset_name_, int min_dim, int max_dim, Blob<Dtype>* blob); // double类型的获取数据维度和信息的包裹函数 template <typename Dtype> void hdf5_save_nd_dataset( const hid_t file_id, const string& dataset_name, const Blob<Dtype>& blob, bool write_diff = false); // 读取int和存储int,读取字符串和存储字符串到文件 int hdf5_load_int(hid_t loc_id, const string& dataset_name); void hdf5_save_int(hid_t loc_id, const string& dataset_name, int i); string hdf5_load_string(hid_t loc_id, const string& dataset_name); void hdf5_save_string(hid_t loc_id, const string& dataset_name, const string& s); // 获取链接数 int hdf5_get_num_links(hid_t loc_id); // 根据名字找到索引 string hdf5_get_name_by_idx(hid_t loc_id, int idx); } // namespace caffe #endif // CAFFE_UTIL_HDF5_H_
cpp文件:
#include "caffe/util/hdf5.hpp" #include <string> #include <vector> namespace caffe { // Verifies format of data stored in HDF5 file and reshapes blob accordingly. // 获取HDF5文件的信息以及数据的维度 template <typename Dtype> void hdf5_load_nd_dataset_helper( hid_t file_id, const char* dataset_name_, int min_dim, int max_dim, Blob<Dtype>* blob) { // Verify that the dataset exists. // 检查是否存在 CHECK(H5LTfind_dataset(file_id, dataset_name_)) << "Failed to find HDF5 dataset " << dataset_name_; // Verify that the number of dimensions is in the accepted range. herr_t status; int ndims; // 获取数据维度 status = H5LTget_dataset_ndims(file_id, dataset_name_, &ndims); CHECK_GE(status, 0) << "Failed to get dataset ndims for " << dataset_name_; CHECK_GE(ndims, min_dim); CHECK_LE(ndims, max_dim); // Verify that the data format is what we expect: float or double. std::vector<hsize_t> dims(ndims); H5T_class_t class_; // 获取数据信息 status = H5LTget_dataset_info( file_id, dataset_name_, dims.data(), &class_, NULL); CHECK_GE(status, 0) << "Failed to get dataset info for " << dataset_name_; switch (class_) { case H5T_FLOAT: LOG_FIRST_N(INFO, 1) << "Datatype class: H5T_FLOAT"; break; case H5T_INTEGER: LOG_FIRST_N(INFO, 1) << "Datatype class: H5T_INTEGER"; break; case H5T_TIME: LOG(FATAL) << "Unsupported datatype class: H5T_TIME"; case H5T_STRING: LOG(FATAL) << "Unsupported datatype class: H5T_STRING"; case H5T_BITFIELD: LOG(FATAL) << "Unsupported datatype class: H5T_BITFIELD"; case H5T_OPAQUE: LOG(FATAL) << "Unsupported datatype class: H5T_OPAQUE"; case H5T_COMPOUND: LOG(FATAL) << "Unsupported datatype class: H5T_COMPOUND"; case H5T_REFERENCE: LOG(FATAL) << "Unsupported datatype class: H5T_REFERENCE"; case H5T_ENUM: LOG(FATAL) << "Unsupported datatype class: H5T_ENUM"; case H5T_VLEN: LOG(FATAL) << "Unsupported datatype class: H5T_VLEN"; case H5T_ARRAY: LOG(FATAL) << "Unsupported datatype class: H5T_ARRAY"; default: LOG(FATAL) << "Datatype class unknown"; } // 设置blob的维度 vector<int> blob_dims(dims.size()); for (int i = 0; i < dims.size(); ++i) { blob_dims[i] = dims[i]; } blob->Reshape(blob_dims); } // float类型的获取数据维度和信息的包裹函数 template <> void hdf5_load_nd_dataset<float>(hid_t file_id, const char* dataset_name_, int min_dim, int max_dim, Blob<float>* blob) { hdf5_load_nd_dataset_helper(file_id, dataset_name_, min_dim, max_dim, blob); herr_t status = H5LTread_dataset_float( file_id, dataset_name_, blob->mutable_cpu_data()); CHECK_GE(status, 0) << "Failed to read float dataset " << dataset_name_; } // double类型的获取数据维度和信息的包裹函数 template <> void hdf5_load_nd_dataset<double>(hid_t file_id, const char* dataset_name_, int min_dim, int max_dim, Blob<double>* blob) { hdf5_load_nd_dataset_helper(file_id, dataset_name_, min_dim, max_dim, blob); herr_t status = H5LTread_dataset_double( file_id, dataset_name_, blob->mutable_cpu_data()); CHECK_GE(status, 0) << "Failed to read double dataset " << dataset_name_; } // 存放float类型到hdf5文件 template <> void hdf5_save_nd_dataset<float>( const hid_t file_id, const string& dataset_name, const Blob<float>& blob, bool write_diff) { // blob信息放到dims int num_axes = blob.num_axes(); hsize_t *dims = new hsize_t[num_axes]; for (int i = 0; i < num_axes; ++i) { dims[i] = blob.shape(i); } // 获取数据指针 const float* data; if (write_diff) { data = blob.cpu_diff(); } else { data = blob.cpu_data(); } // 存放数据到hdf5 herr_t status = H5LTmake_dataset_float( file_id, dataset_name.c_str(), num_axes, dims, data); CHECK_GE(status, 0) << "Failed to make float dataset " << dataset_name; delete[] dims; } // 存放double类型到hdf5文件 template <> void hdf5_save_nd_dataset<double>( hid_t file_id, const string& dataset_name, const Blob<double>& blob, bool write_diff) { int num_axes = blob.num_axes(); hsize_t *dims = new hsize_t[num_axes]; for (int i = 0; i < num_axes; ++i) { dims[i] = blob.shape(i); } const double* data; if (write_diff) { data = blob.cpu_diff(); } else { data = blob.cpu_data(); } herr_t status = H5LTmake_dataset_double( file_id, dataset_name.c_str(), num_axes, dims, data); CHECK_GE(status, 0) << "Failed to make double dataset " << dataset_name; delete[] dims; } // 读取string到字符串 string hdf5_load_string(hid_t loc_id, const string& dataset_name) { // Get size of dataset size_t size; H5T_class_t class_; herr_t status = H5LTget_dataset_info(loc_id, dataset_name.c_str(), NULL, &class_, &size); CHECK_GE(status, 0) << "Failed to get dataset info for " << dataset_name; char *buf = new char[size]; status = H5LTread_dataset_string(loc_id, dataset_name.c_str(), buf); CHECK_GE(status, 0) << "Failed to load int dataset with name " << dataset_name; string val(buf); delete[] buf; return val; } // 保存string到字符串 void hdf5_save_string(hid_t loc_id, const string& dataset_name, const string& s) { herr_t status = H5LTmake_dataset_string(loc_id, dataset_name.c_str(), s.c_str()); CHECK_GE(status, 0) << "Failed to save string dataset with name " << dataset_name; } // 载入int类型 int hdf5_load_int(hid_t loc_id, const string& dataset_name) { int val; herr_t status = H5LTread_dataset_int(loc_id, dataset_name.c_str(), &val); CHECK_GE(status, 0) << "Failed to load int dataset with name " << dataset_name; return val; } // 存储int类型 void hdf5_save_int(hid_t loc_id, const string& dataset_name, int i) { hsize_t one = 1; herr_t status = H5LTmake_dataset_int(loc_id, dataset_name.c_str(), 1, &one, &i); CHECK_GE(status, 0) << "Failed to save int dataset with name " << dataset_name; } // 获取链接数 int hdf5_get_num_links(hid_t loc_id) { H5G_info_t info; herr_t status = H5Gget_info(loc_id, &info); CHECK_GE(status, 0) << "Error while counting HDF5 links."; return info.nlinks; } // 通过名字找到索引 string hdf5_get_name_by_idx(hid_t loc_id, int idx) { ssize_t str_size = H5Lget_name_by_idx( loc_id, ".", H5_INDEX_NAME, H5_ITER_NATIVE, idx, NULL, 0, H5P_DEFAULT); CHECK_GE(str_size, 0) << "Error retrieving HDF5 dataset at index " << idx; char *c_str = new char[str_size+1]; ssize_t status = H5Lget_name_by_idx( loc_id, ".", H5_INDEX_NAME, H5_ITER_NATIVE, idx, c_str, str_size+1, H5P_DEFAULT); CHECK_GE(status, 0) << "Error retrieving HDF5 dataset at index " << idx; string result(c_str); delete[] c_str; return result; } } // namespace caffe 给出具体实现: /* TODO: - load file in a separate thread ("prefetch") - can be smarter about the memcpy call instead of doing it row-by-row :: use util functions caffe_copy, and Blob->offset() :: don't forget to update hdf5_daa_layer.cu accordingly - add ability to shuffle filenames if flag is set */ #include <fstream> // NOLINT(readability/streams) #include <string> #include <vector> #include "hdf5.h" #include "hdf5_hl.h" #include "stdint.h" #include "caffe/data_layers.hpp" #include "caffe/layer.hpp" #include "caffe/util/hdf5.hpp" namespace caffe { template <typename Dtype> HDF5DataLayer<Dtype>::~HDF5DataLayer<Dtype>() { } // Load data and label from HDF5 filename into the class property blobs. // 读取HDF5文件数据到hdf_blobs template <typename Dtype> void HDF5DataLayer<Dtype>::LoadHDF5FileData(const char* filename) { DLOG(INFO) << "Loading HDF5 file: " << filename; // 打开文件 hid_t file_id = H5Fopen(filename, H5F_ACC_RDONLY, H5P_DEFAULT); if (file_id < 0) { LOG(FATAL) << "Failed opening HDF5 file: " << filename; } int top_size = this->layer_param_.top_size(); hdf_blobs_.resize(top_size); const int MIN_DATA_DIM = 1; const int MAX_DATA_DIM = INT_MAX; for (int i = 0; i < top_size; ++i) { hdf_blobs_[i] = shared_ptr<Blob<Dtype> >(new Blob<Dtype>()); // message LayerParameter { // optional string name = 1; // the layer name // optional string type = 2; // the layer type // repeated string bottom = 3; // the name of each bottom blob // repeated string top = 4; // the name of each top blob hdf5_load_nd_dataset(file_id, this->layer_param_.top(i).c_str(), MIN_DATA_DIM, MAX_DATA_DIM, hdf_blobs_[i].get()); } herr_t status = H5Fclose(file_id); CHECK_GE(status, 0) << "Failed to close HDF5 file: " << filename; // MinTopBlobs==1 guarantees at least one top blob CHECK_GE(hdf_blobs_[0]->num_axes(), 1) << "Input must have at least 1 axis."; const int num = hdf_blobs_[0]->shape(0); for (int i = 1; i < top_size; ++i) { CHECK_EQ(hdf_blobs_[i]->shape(0), num); } // Default to identity permutation. data_permutation_.clear(); data_permutation_.resize(hdf_blobs_[0]->shape(0)); for (int i = 0; i < hdf_blobs_[0]->shape(0); i++) data_permutation_[i] = i; // Shuffle if needed. // 将数据索引映射表进行shuffle if (this->layer_param_.hdf5_data_param().shuffle()) { std::random_shuffle(data_permutation_.begin(), data_permutation_.end()); DLOG(INFO) << "Successully loaded " << hdf_blobs_[0]->shape(0) << " rows (shuffled)"; } else { DLOG(INFO) << "Successully loaded " << hdf_blobs_[0]->shape(0) << " rows"; } } // 主要的功能就是读取HDF5文件,并且设置top blob的形状 template <typename Dtype> void HDF5DataLayer<Dtype>::LayerSetUp(const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) { // Refuse transformation parameters since HDF5 is totally generic. CHECK(!this->layer_param_.has_transform_param()) << this->type() << " does not transform data."; // Read the source to parse the filenames. // 读取HDF列表文件 const string& source = this->layer_param_.hdf5_data_param().source(); LOG(INFO) << "Loading list of HDF5 filenames from: " << source; hdf_filenames_.clear(); std::ifstream source_file(source.c_str()); if (source_file.is_open()) { std::string line; while (source_file >> line) { hdf_filenames_.push_back(line); } } else { LOG(FATAL) << "Failed to open source file: " << source; } source_file.close(); num_files_ = hdf_filenames_.size(); current_file_ = 0; LOG(INFO) << "Number of HDF5 files: " << num_files_; CHECK_GE(num_files_, 1) << "Must have at least 1 HDF5 filename listed in " << source; file_permutation_.clear(); file_permutation_.resize(num_files_); // 文件名字是否shuffle // Default to identity permutation. for (int i = 0; i < num_files_; i++) { file_permutation_[i] = i; } // Shuffle if needed. if (this->layer_param_.hdf5_data_param().shuffle()) { std::random_shuffle(file_permutation_.begin(), file_permutation_.end()); } // Load the first HDF5 file and initialize the line counter. // 从给定的文件名列表中的第一个文件名读取数据到hdf_blobs LoadHDF5FileData(hdf_filenames_[file_permutation_[current_file_]].c_str()); // 设置行指针 current以上是关于caffe代码阅读8: Data_layers的实现细节(各个数据读取层的实现细节) 2016.3.25-28的主要内容,如果未能解决你的问题,请参考以下文章
caffe代码阅读3:data_readerinternalthread以及blocking_queue的实现细节-2016.3.15
caffe代码阅读3:data_readerinternalthread以及blocking_queue的实现细节-2016.3.15