搭建caffe环境及训练模型
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了搭建caffe环境及训练模型相关的知识，希望对你有一定的参考价值。
Mac OS搭建caffe环境
一、相关软件安装
1、 安装CUDA8
（1） 安装xcode及命令行
（2） 安装CUDA的dmg包
（3） 配置环境变量并source ~/.bash_profile
export CUDA_ROOT=/Developer/NVIDIA/CUDA-8.0
export PATH=$CUDA_ROOT/bin${PATH:+:${PATH}}
export DYLD_LIBRARY_PATH=$CUDA_ROOT/lib${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}
export LD_LIBRARY_PATH=$CUDA_ROOT/lib:$LD_LIBRARY_PATH
（4） 执行nvcc –V验证安装
2、 安装必要的包
brew install –vd snappy leveldb gflags glog szip lmdb
brew tap homebrew/science
brew install hdf5 opencv
3、 brew edit opencv并修改两行
-DPYTHON_LIBRARY=#{py_prefix}/lib/libpython2.7.dylib
-DPYTHON_INCLUDE_DIR=#{py_prefix}/include/python2.7
4、 安装protobuf和boost-python
brew install --build-from-source --with-python –vd protobuf
brew install --build-from-source -vd boost boost-python
二、caffe安装
1、下载caffe
git clone https://github.com/BVLC/caffe.git
2、打开Makefile.config, 配置BLAS
BLAS_INCLUDE := /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/Headers
BLAS_LIB := /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A
去掉CPU_ONLY的注释
去掉USE_LEVELDB := 0的注释
修改PYTHON_INCLUDE的目录为python2.7的目录
三、编译caffe
make all
make test
make runtest
make pycaffe
make pytest
make distribute
会有一些warning, 需要安装什么包就安装什么包
1)“dot” not found in path问题需要安装graphviz，不用pip用brew安装
2) make pytest出现了如下问题:
注释掉/python/caffe/proto/caffe_pb2.py 所有 “syntax = proto2 “
3）需要进入python文件夹执行python才能import caffe
4）安装scikit-image库
5）pip不好使，就pip install –U pip
Centos 7安装Caffe
一、安装CUDA
wget https://developer.nvidia.com/compute/cuda/8.0/Prod2/local_installers/cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64-rpm
rpm -i cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64-rpm
yum clean all
yum install cuda
设置CUDA环境变量：
export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:/usr/local/cuda-8.0/extras/CUPTI/lib64:$LD_LIBRARY_PATH（指定共享库，编译opencv、cudasample等）
export CUDA_HOME=/usr/local/cuda-8.0/（指定CUDA安装路径）
export PATH=/usr/local/cuda-8.0/bin:$PATH（在任意环境下调用cuda可执行文件）
更新运行库缓存：
ldconfig
检验CUDA是否安装成功：
nvcc --version
GPU支持：
安装Nvidia驱动：
根据GPU型号从相应网站下载驱动，例如使用NVIDIA Tesla M60，从NVIDIA网站选择对应的型号和操作系统，CUDA Toolkit版本，下载驱动文件，如NVIDIA-Linux-x86_64-375.66.run，运行驱动文件，根据提示安装：
sh NVIDIA-Linux-x86_64-375.66.run
通过NVIDIA命令工具nvidia-smi检验GPU情况:
nvidia-smi
安装cuDNN：
从NVIDIA网站下载cuDNN 安装包，根据GPU及CUDA版本选择对应cuDNN版本，下载cuDNN v5.1 for CUDA8.0,解压拷贝到CUDA安装目录
cp include/* /usr/local/cuda/include
cp lib64/* /usr/local/cuda/lib64
二、yum安装依赖包
yum install protobuf-devel leveldb-devel snappy-devel opencv-devel boost-devel hdf5-devel gflags-devel glog-devel lmdb-devel atlas-devel
n cd /usr/lib64/atlas
n sudo ln -sv libsatlas.so.3.10 libcblas.so
n sudo ln -sv libsatlas.so.3.10 libatlas.so
（如果有需要，可安装Openblas或者MKL作为atlas的备用）
二、非yum手动安装依赖包
安装 gflags：
git clone https://github.com/gflags/gflags
cd gflags
mkdir build && cd build
export CXXFLAGS="-fPIC" && cmake .. -DCMAKE_INSTALL_PREFIX=/data2/shixi_wuhao3/local -DBUILD_SHARED_LIBS=ON
make VERBOSE=1
sudo make install
安装 glog：
git clone https://github.com/google/glog
cd glog
./configure
make
automake --add-missing
sudo make install
安装 lmdb：
git clone https://github.com/LMDB/lmdb
cd lmdb/libraries/liblmdb
make
sudo make install
安装 hdf5：
wget https://support.hdfgroup.org/ftp/HDF5/current18/src/hdf5-1.8.18.tar.gz
tar -xvf hdf5-1.8.18.tar.gz
cd hdf5-1.8.18
./configure --prefix=/usr/local
make
sudo make install
安装protobuf
tar -xvf protobuf
cd protobuf
./configure --prefix=/usr/local/protobuf
make
make check
make install
安装boost
wget http://sourceforge.net/projects/boost/files/boost/1.59.0/boost_1_59_0.tar.gz
tar -zxvf boost_1_59_0.tar.gz
cd boost_1_59_0
./bootstrap.sh --prefix=path/to/install
./b2 install
这会把boost安装到path/to/install，若不指定，默认将安装到/usr/local目录，但需要权限。
安装opencv
leveldb要和snappy一起安装
三、下载caffe
git clone https://github.com/BVLC/caffe
四、安装Python依赖包（注意python2和python3的）
for req in $(cat caffe/python/requirements.txt); do pip install $req; done
五、编辑Makefile.config 文件，根据情况修改配置：
BLAS := atlas
BLAS_INCLUDE := /usr/include/atlas
BLAS_LIB := /usr/lib64/atlas
PYTHON_INCLUDE := /usr/local/include/python2.7 \
/usr/local/lib/python2.7/site-packages/numpy/core/include
PYTHON_LIB := /usr/lib64
USE_CUDNN := 1（使用cuDNN）
（去掉CPU_ONLY的注释——只用CPU）
（去掉USE_LEVELDB := 0的注释——使用LevelDB）
根据情况确定是否修改Makefile.cofig中的相关位置路径
## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!
# cuDNN acceleration switch (uncomment to build with cuDNN).
# USE_CUDNN := 1
"CuDNN是NVIDIA专门针对Deep Learning框架设计的一套GPU计算加速库，用于实现高性能的并行计算,在有GPU并且安装CuDNN的情况下可以打开即将注释去掉。"
# CPU-only switch (uncomment to build without GPU support).
#CPU_ONLY := 1
"表示是否用GPU,如果只有CPU这里要打开"
# uncomment to disable IO dependencies and corresponding data layers
USE_OPENCV := 1
"因为要用到OpenCV库所以要打开，下面这两个选项表示是选择Caffe的数据管理第三方库，两者都不打开 Caffe默认用的是LMDB，这两者均是嵌入式数据库管理系统编程库。"
# USE_LEVELDB := 0
# USE_LMDB := 0
# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
# You should not set this flag if you will be reading LMDBs with any
# possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1
"当需要读取LMDB文件时可以取消注释，默认不打开。"
# Uncomment if you‘re using OpenCV 3
OPENCV_VERSION := 2.4.10 （我的2.4.5）
"用pkg-config --modversion opencv命令查看opencv版本"
# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++
"linux系统默认使用g++编译器，OSX则是clang++。"
# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
"CUDA的安装目录"
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr
# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
-gencode arch=compute_20,code=sm_21 \
-gencode arch=compute_30,code=sm_30 \
-gencode arch=compute_35,code=sm_35 \
-gencode arch=compute_50,code=sm_50 \
-gencode arch=compute_50,code=compute_50
"这些参数需要根据GPU的计算能力
(http://blog.csdn.net/jiajunlee/article/details/52067962)来进行设置，6.0以下的版本不支持×_50的计算能力。"
# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := open
"如果用的是ATLAS计算库则赋值atlas，MKL计算库则用mkl赋值，OpenBlas则赋值open。"
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
BLAS_INCLUDE := /usr/local/OpenBlas/include
BLAS_LIB := /usr/local/OpenBlas/lib
"blas库安装目录"
# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib
"如果不是安装在标准路径则要指明"
# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app
"matlab安装库的目录"
# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
PYTHON_INCLUDE := /usr/include/python2.7 \
/usr/lib/python2.7/dist-packages/numpy/core/include
"python安装目录"
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it‘s in root.
# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
# $(ANACONDA_HOME)/include/python2.7 \
# $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \
# Uncomment to use Python 3 (default is Python 2)
# PYTHON_LIBRARIES := boost_python3 python3.5m
# PYTHON_INCLUDE := /usr/include/python3.5m \
# /usr/lib/python3.5/dist-packages/numpy/core/include
# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /usr/lib
"python库位置"
# PYTHON_LIB := $(ANACONDA_HOME)/lib
# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c ‘import numpy.core; print(numpy.core.__file__)‘))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib
# Uncomment to support layers written in Python (will link against Python libs)
WITH_PYTHON_LAYER := 1
# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib
# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib
# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1
# N.B. both build and distribute dirs are cleared on `make clean`
BUILD_DIR := build
DISTRIBUTE_DIR := distribute
# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1
# The ID of the GPU that ‘make runtest‘ will use to run unit tests.
TEST_GPUID := 0
"所用的GPU的ID编号"
# enable pretty build (comment to see full commands)
Q ?= @
六、编译及测试：
make all
make test
make runtest
测试成功，结果如下：
编译caffe的python环境及测试：
make pycaffe
make pytest
这样进入python交互行import caffe成功即可
http://caffe.berkeleyvision.org/installation.html
pip安装future库
安装最新版的cudnn（美团有链接，可能要注册）
cmake要设置CPU参数OFF, cmake -DUSE_CUDA=OFF ..
git要加参数recursive
make install时要加sudo
在site-package目录下编写.pth来使得caffe在任何目录下都可以import
以上是关于搭建caffe环境及训练模型的主要内容，如果未能解决你的问题，请参考以下文章
windows环境Caffe安装配置步骤（无GPU）及mnist训练