使用内核态的RBD

Posted ygtff

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了使用内核态的RBD相关的知识，希望对你有一定的参考价值。

使用内核态的RBD模块（正常情况）

  • 创建测试RBD Image：
[root@ceph-node1 tmp]# rbd create rbd/bench-image --size 1G
[root@ceph-node1 tmp]# rbd ls -p rbd
bench-image
test-image
[root@ceph-node1 tmp]# rbd info rbd/bench-image
rbd image 'bench-image':
	size 1024 MB in 256 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.5ddff6b8b4567
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	flags:
  • 在一台虚拟机中使用这个Image：
[root@ceph-node1 ~]# rbd map bench-iamge
rbd: sysfs write failed
In some cases useful info is found in syslog - try "dmesg | tail" or so.
rbd: map failed: (2) No such file or directory
  • 查看系统Log：
[root@ceph-node1 ~]# dmesg | tail
Oct 18 02:30:24 ceph-node1 kernel: libceph: client374111 fsid cbc99ef9-fbc3-41ad-a726-47359f8d84b3
Oct 18 02:30:24 ceph-node1 kernel: rbd: image bench-image: image uses unsupported features: 0x38

从日志可以看出，是RBD Image的有些feature，当前内核不支持，所以需要disable这些feature，rbd支持disable功能。

  • 内核及系统版本
[root@ceph-node1 ~]# uname -r
3.10.0-514.26.2.el7.x86_64
[root@ceph-node1 ~]# cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core)
  • Disable RBD Image feature
[root@ceph-node1 ~]# rbd feature disable bench-image fast-diff
[root@ceph-node1 ~]# rbd feature disable bench-image object-map
[root@ceph-node1 ~]# rbd feature disable bench-image exclusive-lock
[root@ceph-node1 ~]# rbd feature disable bench-image deep-flatten

上面这些feature也可以通过一条命令直接disable，不过需要注意feature先后顺序。

  • 查看当前Image的feature
[root@ceph-node1 ~]# rbd info bench-image
rbd image 'bench-image':
	size 1024 MB in 256 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.5ddff6b8b4567
	format: 2
	features: layering
	flags:
  • 继续使用RBD Image
[root@ceph-node1 ~]# rbd map bench-image --id admin
/dev/rbd0

使用RBD内核模块 (通过上面的disable feature，依然不成功，可以尝试如下方法)

  • 在另外一个Ceph集群中，通过上述方法，依然不能使用RBD内核模块
[root@nova10 ~]# uname -r
3.10.0-327.el7.x86_64
[root@nova10 ~]# cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core) 

[root@storage04 ~]# rbd info volumes/bench-image
rbd image 'bench-image':
	size 10240 MB in 2560 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.2adef66b8b4567
	format: 2
	features: layering
	flags: 

[root@nova10 ~]# rbd map volumes/bench-image --id admin
^C

[root@nova10 ~]# tailf /var/log/messages
Oct 18 15:04:41 nova10 kernel: libceph: mon1 192.168.34.12:6789 feature set mismatch, my 102b84a842a42 < server's 40102b84a842a42, missing 400000000000000
Oct 18 15:04:41 nova10 kernel: libceph: mon1 192.168.34.12:6789 missing required protocol features
Oct 18 15:07:24 nova10 kernel: libceph: mon0 192.168.34.11:6789 feature set mismatch, my 102b84a842a42 < server's 40102b84a842a42, missing 400000000000000
Oct 18 15:07:24 nova10 kernel: libceph: mon0 192.168.34.11:6789 missing required protocol features

使用rbd-nbd模块

  • 安装rbd-nbd
[root@ceph-node1 ~]# yum install rbd-nbd
Installed:
  rbd-nbd.x86_64 1:10.2.9-2.el7.centos                                                                                                                                                  
Complete!
  • 使用rbd-nbd
[root@ceph-node1 ~]# rbd-nbd map test-image
modprobe: FATAL: Module nbd not found.
rbd-nbd: failed to find unused device

也不成功，报错：没有nbd模块
  • 我们只能自己编译nbd模块，编译过程在后面会详细说明。
  • 当成功加了nbd模块之后，使用rbd-nbd
[root@ceph-node1 ~]# rbd-nbd map test-image
/dev/nbd0

[root@ceph-node1 ~]# rbd-nbd list-mapped
/dev/nbd0

[root@ceph-node1 ~]# rbd showmapped
id pool image       snap device    
0  rbd  bench-image -    /dev/rbd0

编译nbd模块

  • 查看当前系统内核
[root@ceph-node1 ~]# uname -r
3.10.0-514.26.2.el7.x86_64
  • 下载内核源码包
[root@ceph-node1 ~]# wget http://vault.centos.org/7.3.1611/os/Source/SPackages/kernel-3.10.0-514.el7.src.rpm

Centos的内核包地址：（注意替换相应的系统版本） http://vault.centos.org/7.3.1611/os/Source/SPackages/

  • 安装依赖包
yum install rpm-build m4 net-tools bc xmlto asciidoc hmaccalc newt-devel perl pesign elfutils-devel binutils-devel bison audit-libs-devel numactl-devel pciutils-devel ncurses-devel libtiff perl-ExtUtils-Embed python-devel java-devel -y
  • 安装内核源码包：
[root@ceph-node1 ~]$ rpm -ivh kernel-3.10.0-514.el7.src.rpm
  • 准备编译
[root@ceph-node1 ~]$ echo '%_topdir %(echo $HOME)/rpmbuild' > ~/.rpmmacros
[root@ceph-node1 ~]$ cd ~/rpmbuild/SPECS
[root@ceph-node1 ~]$ rpmbuild -bp --target=$(uname -m) kernel.spec
  • 编译

    • 在内核源码中选择编译nbd模块

      [root@ceph-node1 ~]$ cd ~/rpmbuild/BUILD/kernel-3.10.0-514.el7/linux-3.10.0-514.el7.centos.x86_64
[root@ceph-node1 linux-3.10.0-514.el7.centos.x86_64]$ make menuconfig
  在弹出的窗口中选择Device Driver -> Block devices -> Network block device support，并在“Network block device support”中输入“M”，保存退出

    • 编译

      [root@ceph-node1 linux-3.10.0-514.el7.centos.x86_64]$ make prepare && make modules_prepare && make
[root@ceph-node1 linux-3.10.0-514.el7.centos.x86_64]$ make M=drivers/block -j4

    • 问题

      编译的时候遇到如下错误：

      drivers/block/nbd.c: 在函数‘__nbd_ioctl’中:
drivers/block/nbd.c:619:19: 错误：‘REQ_TYPE_SPECIAL’未声明(在此函数内第一次使用)
sreq.cmd_type = REQ_TYPE_SPECIAL;
               ^
drivers/block/nbd.c:619:19: 附注：每个未声明的标识符在其出现的函数内只报告一次
make[1]: *** [drivers/block/nbd.o] 错误 1
make: *** [_module_drivers/block] 错误 2

      解决：

      [root@ceph-node1 linux-3.10.0-514.el7.centos.x86_64]$ vim drivers/block/nbd.c
将sreq.cmd_type = REQ_TYPE_SPECIAL;最后那个宏是7，可以改成： sreq.cmd_type = 7; 然后继续编译
  • 加载nbd内核模块