L版之后master ceph编译问题总结

Posted ygtff

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了L版之后master ceph编译问题总结相关的知识，希望对你有一定的参考价值。

  • run-make-check.sh报错

    --  we are x84_64

    -- Performing Test not_arch_x32

    -- Performing Test not_arch_x32 - Failed

    --  we are x32; no yasm for you

    -- Performing Test COMPILER_SUPPORTS_CXX17

    -- Performing Test COMPILER_SUPPORTS_CXX17 - Failed

    CMake Error at src/CMakeLists.txt:149 (message):

      The compiler /usr/bin/c++ has no C++17 support.

    解决：
1. 社区已经考虑到Centos下的gcc、cmake版本比较低，不支持C++17，因此在install-deps.sh中处理：
   1. yum install centos-release-scl  
   2. yum install devtoolset-7
   3. scl enable devtoolset-7 bash
   这样，就可以使用高版本的cmake和gcc了。也就是使用的开发环境，不影响机器上的运行gcc、cmake版本。
2. 参考： ...

     

  • Dpdk编译报错

    In file included from /opt/rh/devtoolset-6/root/usr/lib/gcc/x86_64-redhat-linux/6.3.1/include/x86intrin.h:39:0,

                     from /root/work/source/submit/ceph/build/src/dpdk/include/rte_vect.h:56,

                     from /root/work/source/submit/ceph/build/src/dpdk/include/rte_memcpy.h:46,

                     from /root/work/source/submit/ceph/build/src/dpdk/include/rte_mempool.h:79,

                     from env.c:41:

    /root/work/source/submit/ceph/build/src/dpdk/include/rte_memcpy.h: In function 'rte_memcpy_generic':

    /opt/rh/devtoolset-6/root/usr/lib/gcc/x86_64-redhat-linux/6.3.1/include/tmmintrin.h:185:1: error: inlining failed in call to always_inline '_mm_alignr_epi8': target specific option mismatch

     _mm_alignr_epi8(__m128i __X, __m128i __Y, const int __N)

    Google了半天，很多人都有遇到dpdk的这个问题，但是没看到解决方法，有人说是vm的问题。没有深入的找这个解决办法。
    就准备直接关闭dpdk的编译，但是一看buidl/CMakeCache.txt发现是关的： WITH_DPDK:BOOL=OFF，这就怪异了，关了DPDK的编译，为什么还会编译，肯定还有其他地方直接就编译了。
    于是去ceph源码顶级目录下看CMakeLists.txt找DPDK，也没什么，也是判断with_dpdk，然后再处理的。
    继续搜一下spdk，是打开的，然后会build_spdk,这个build_spdk是在cmake/modules/BuildSPDK.cmake文件里，然后在文件一开始就看到了如下代码：
    if(NOT TARGET dpdk-ext)
      include(BuildDPDK)   
      build_dpdk()         
    endif()                
    原来是在这里打开了dpdk的编译，这下就清楚了，直接关闭spdk的编译项：
    在CMakeLists.cc中将
    if(CMAKE_SYSTEM_PROCESSOR MATCHES "i386|i686|amd64|x86_64|AMD64|aarch64")分支下原来的ON，改成OFF
        option(WITH_SPDK "Enable SPDK" OFF)
    重新再build目录下
    [root@ygt build]# cmake -DBOOST_J=8 -DWITH_CCACHE=ON ..

    尝试解决dpdk编译问题的链接：【TODO：解决该问题】

    https://stackoverflow.com/questions/47443627/error-inlining-failed-in-call-to-always-inline-mm-aesenc-si128-target-speci

    https://stackoverflow.com/questions/35772562/inlining-failed-in-call-to-always-inline-m128i-mm-cvtepu8-epi32-m128i-t

    https://stackoverflow.com/questions/47587561/gcc-compilation-error-inlining-failed-in-call-to-always-inline-even-after-set

    https://www.mail-archive.com/users@dpdk.org/msg02677.html
     

  • Leveldb报错

    ../../lib/libos.a(LevelDBStore.cc.o): In function `LevelDBStore::repair(std::ostream&)':

    /root/work/source/submit/ceph/src/kv/LevelDBStore.cc:208: undefined reference to `leveldb::RepairDB(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, leveldb::Options const&)'

    /root/work/source/submit/ceph/src/kv/LevelDBStore.cc:212: undefined reference to `leveldb::Status::ToString[abi:cxx11]() const'

    ../../lib/libos.a(LevelDBStore.cc.o): In function `LevelDBStore::do_open(std::ostream&, bool)':

    /root/work/source/submit/ceph/src/kv/LevelDBStore.cc:134: undefined reference to `leveldb::DB::Open(leveldb::Options const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, leveldb::DB**)'

    /root/work/source/submit/ceph/src/kv/LevelDBStore.cc:137: undefined reference to `leveldb::Status::ToString[abi:cxx11]() const'

    ../../lib/libos.a(LevelDBStore.cc.o): In function `LevelDBStore::_test_init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':

    /root/work/source/submit/ceph/src/kv/LevelDBStore.cc:167: undefined reference to `leveldb::DB::Open(leveldb::Options const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, leveldb::DB**)'

    leveldb也已经安装了，leveldb动态库在/usr/lib64下也有。leveldb的很多头文件在/usr/include下也有。为什么失败呢？

    从网上找了一段leveldb的测试程序：直接通过这段程序来测试leveldb是否可用。排除掉是ceph的bug。

    [root@ygt c++]# cat leveldb_test.cc

    #include <iostream> 

    #include <leveldb/db.h> 

    //#include <assert> 

       

    int main (int argc,char * argv[]) 

     

        leveldb::DB* db; 

        leveldb::Options options; 

     

        options.create_if_missing = true; 

        std::string dbpath = "testdb"; 

     

        leveldb::Status status = leveldb::DB::Open(options, dbpath, &db); 

        assert (status.ok ()); 

     

        std::string key = "test"; 

        std::string val = "test_value"; 

     

        status = db->Put(leveldb::WriteOptions (), key, val); 

        val.clear (); 

        status = db->Get(leveldb::ReadOptions (), key, &val); 

                       

        std::cout << key << ": " << val << std::endl; 

    编译：也报同样的错：

    [root@ygt c++]# g++  -lleveldb -lpthread leveldb_test.cc

    /tmp/cc7BQ3E6.o: In function `main':

    leveldb_test.cc:(.text+0x7e): undefined reference to `leveldb::DB::Open(leveldb::Options const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, leveldb::DB**)'

    collect2: error: ld returned 1 exit status

    这个时候就怀疑是leveldb版本过低，当前是1.12，于是看了一下github上leveldb的版本，是差了好多版本。

    我没有直接去下载最新版本的leveldb，还是下载了1.12版本的leveldb源码：因为我觉得可能还是和当前的这个leveldb是低版本gcc编译造成的问题。

    [root@ygt source]# git clone git@github.com:google/leveldb.git

    编译：

    [root@ygt leveldb]# make

    编译完成后：找一下动态库

    [root@ygt leveldb]# cd out-shared/

    [root@ygt out-shared]# ls

    c_test  db  db_bench  helpers  libleveldb.so  libleveldb.so.1  libleveldb.so.1.20  port  table  util

     

    将新编译的动态库替换掉原来的库：

    [root@ygt out-shared]# cp libleveldb.so* /usr/lib64/

    [root@ygt lib64]# ll libleveldb.so*

    -rwxr-xr-x. 1 root root 423152 Mar  8 02:22 libleveldb.so

    -rwxr-xr-x. 1 root root 423152 Mar  8 02:22 libleveldb.so.1

    -rwxr-xr-x. 1 root root 348928 May 17  2016 libleveldb.so.1.0.7

    -rwxr-xr-x. 1 root root 423152 Mar  8 02:22 libleveldb.so.1.20

     

    [root@ygt lib64]# rm libleveldb.so libleveldb.so.1

    [root@ygt lib64]# ll libleveldb.so*

    -rwxr-xr-x. 1 root root 348928 May 17  2016 libleveldb.so.1.0.7

    -rwxr-xr-x. 1 root root 423152 Mar  8 02:22 libleveldb.so.1.20

     

    [root@ygt lib64]# ln -s libleveldb.so.1.20 libleveldb.so

    [root@ygt lib64]# ln -s libleveldb.so.1.20 libleveldb.so.1

    [root@ygt lib64]# ll libleveldb.so*

    lrwxrwxrwx. 1 root root     18 Mar  8 02:23 libleveldb.so -> libleveldb.so.1.20

    lrwxrwxrwx. 1 root root     18 Mar  8 02:23 libleveldb.so.1 -> libleveldb.so.1.20

    -rwxr-xr-x. 1 root root 348928 May 17  2016 libleveldb.so.1.0.7

    -rwxr-xr-x. 1 root root 423152 Mar  8 02:22 libleveldb.so.1.20

     

    保险起见，把include下的头文件也替换掉：

    [root@ygt include]# cp -r leveldb/ /usr/include/

     

    使用leveldb测试程序验证一下：（编译通过）

    [root@ygt c++]# g++ leveldb_test.cc -lleveldb -lpthread

附：(Ceph安装了这些，但是并没有去设置source只是提示了而已 . . .)

./run-make-check.sh

Getting requirements for /tmp/install-deps.16680/ceph.spec

 --> Already installed : 1:java-1.8.0-openjdk-devel-1.8.0.141-1.b16.el7_3.x86_64

 --> Already installed : sharutils-4.13.3-8.el7.x86_64

 --> Already installed : checkpolicy-2.5-4.el7.x86_64

 --> Already installed : selinux-policy-devel-3.13.1-102.el7_3.16.noarch

 --> Already installed : bc-1.06.95-13.el7.x86_64

 --> Already installed : gperf-3.0.4-8.el7.x86_64

 --> Already installed : cmake-2.8.12.2-2.el7.x86_64

 --> Already installed : cryptsetup-1.7.2-1.el7.x86_64

 --> Already installed : fuse-devel-2.9.2-7.el7.x86_64

 --> Already installed : devtoolset-7-gcc-c++-7.2.1-1.el7.sc1.x86_64

 --> Already installed : gdbm-1.10-8.el7.x86_64

 --> Already installed : gperftools-devel-2.4-8.el7.x86_64

 --> Already installed : jq-1.5-1.el7.x86_64

 --> Already installed : leveldb-devel-1.12.0-11.el7.x86_64

 --> Already installed : libaio-devel-0.3.109-13.el7.x86_64

 --> Already installed : libblkid-devel-2.23.2-33.el7_3.2.x86_64

 --> Already installed : libcurl-devel-7.29.0-35.el7.centos.x86_64

 --> Already installed : systemd-devel-219-30.el7_3.9.x86_64

 --> Already installed : libtool-2.4.2-22.el7_3.x86_64

 --> Already installed : libxml2-devel-2.9.1-6.el7_2.3.x86_64

 --> Already installed : libuuid-devel-2.23.2-33.el7_3.2.x86_64

 --> Already installed : 1:make-3.82-23.el7.x86_64

 --> Already installed : parted-3.1-28.el7.x86_64

 --> Already installed : 4:perl-5.16.3-291.el7.x86_64

 --> Already installed : 1:pkgconfig-0.27.1-4.el7.x86_64

 --> Already installed : procps-ng-3.3.10-10.el7.x86_64

 --> Already installed : python-2.7.5-48.el7.x86_64

 --> Already installed : python-devel-2.7.5-48.el7.x86_64

 --> Already installed : python-nose-1.3.0-3.el7.noarch

 --> Already installed : python-requests-2.6.0-1.el7_1.noarch

 --> Already installed : python-virtualenv-1.10.1-3.el7.noarch

 --> Already installed : snappy-devel-1.1.0-3.el7.x86_64

 --> Already installed : socat-1.7.3.2-2.el7.x86_64

 --> Already installed : systemd-219-30.el7_3.9.x86_64

 --> Already installed : util-linux-2.23.2-33.el7_3.2.x86_64

 --> Already installed : 1:valgrind-devel-3.11.0-24.el7.x86_64

 --> Already installed : which-2.20-7.el7.x86_64

 --> Already installed : xfsprogs-4.5.0-10.el7_3.x86_64

 --> Already installed : xfsprogs-devel-4.5.0-10.el7_3.x86_64

 --> Already installed : xmlstarlet-1.6.1-1.el7.x86_64

 --> Already installed : yasm-1.2.0-4.el7.x86_64

 --> Already installed : boost-random-1.53.0-26.el7.x86_64

 --> Already installed : btrfs-progs-4.4.1-1.el7.x86_64

 --> Already installed : nss-devel-3.28.4-1.2.el7_3.x86_64

 --> Already installed : keyutils-libs-devel-1.5.8-3.el7.x86_64

 --> Already installed : rdma-core-devel-13-7.el7.x86_64

 --> Already installed : openldap-devel-2.4.40-13.el7.x86_64

 --> Already installed : 1:openssl-devel-1.0.2k-8.el7.x86_64

 --> Already installed : CUnit-devel-2.1.3-13.el7.x86_64

 --> Already installed : redhat-lsb-core-4.1-27.el7.centos.1.x86_64

 --> Already installed : Cython-0.19-3.el7.x86_64

 --> Already installed : python-prettytable-0.7.2-3.el7.noarch

 --> Already installed : python-sphinx-1.1.3-11.el7.noarch

 --> Already installed : python34-devel-3.4.5-5.el7.x86_64

 --> Already installed : python34-setuptools-19.2-3.el7.noarch

 --> Already installed : python34-Cython-0.23.5-1.el7.x86_64

 --> Already installed : python-cherrypy-3.2.2-4.el7.noarch

 --> Already installed : python-pecan-0.4.5-2.el7.noarch

 --> Already installed : python-werkzeug-0.9.1-2.el7.noarch

 --> Already installed : lttng-ust-devel-2.4.1-4.el7.x86_64

 --> Already installed : libbabeltrace-devel-1.2.4-3.el7.x86_64

 --> Already installed : expat-devel-2.1.0-10.el7_3.x86_64

 --> Already installed : redhat-rpm-config-9.1.0-72.el7.centos.noarch

 --> Already installed : junit-4.11-8.el7.noarch

No uninstalled build requires

Your GCC is too old. Please run following command to add DTS to your environment:

 

scl enable devtoolset-7 bash

 

Or add following line to the end of ~/.bashrc to add it permanently:

 

source scl_source enable devtoolset-7

 

see https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/ for more details.

以上是关于L版之后master ceph编译问题总结的主要内容，如果未能解决你的问题，请参考以下文章

Ceph编译：L版本及其之后的版本

使用ceph-deploy安装L版开源存储

Ceph

Ceph对象存储RGW对接企业级网盘OwnCloud三步走

Ceph手动修复mon 集群

1,ceph