Ceph常用操作维护手册

Posted John08

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Ceph常用操作维护手册相关的知识,希望对你有一定的参考价值。

一、更换磁盘

1、  确认需要更换的磁盘对应的盘符、OSD ID;

root@Ceph-B**A**-04-S2-60:~# df -h |grep 17

/dev/sdp3       1.1T  566G  551G  51% /var/lib/ceph/osd/ceph-17

*sdp为服务器上对应的盘符;

*ceph-17 17OSD对应的ID

 

2、  确认磁盘在服务器上对应的槽位;

root@Ceph-B**A**-04-S2-60:~# ll /dev/disk/by-path/ |grep sdp

lrwxrwxrwx 1 root root    9 Aug 17 19:17 pci-0000:02:00.0-scsi-0:0:18:0 -> ../../sdp

lrwxrwxrwx 1 root root   10 Aug 17 19:17 pci-0000:02:00.0-scsi-0:0:18:0-part1 -> ../../sdp1

lrwxrwxrwx 1 root root   10 Aug 17 19:17 pci-0000:02:00.0-scsi-0:0:18:0-part2 -> ../../sdp2

lrwxrwxrwx 1 root root   10 Aug 17 19:17 pci-0000:02:00.0-scsi-0:0:18:0-part3 -> ../../sdp3

*根据盘符找到服务器上对应的磁盘槽位,scsi-0:0:18:0标识第18号槽位;

 

3、  确认主机的SN;

root@Ceph-B**A**-04-S2-60:~# dmidecode -s system-serial-number

HVNHB**

 

4、  在Crushmap设置故障OSD OUT标志;

ceph osd out 17

*设置OUT标识,告诉集群不要再往这个OSD磁盘上读写数据;

 

5、  查看集群状态;

ceph –s

*确认OSD设置OUT标志后,集群有没有出现异常;

 

6、  关闭OSD对应的实例进程;

stop cep-osd id=17

 

7、  从Crushmap中移除OSD;

ceph osd crush remove osd.17

 

8、  删除集群中OSD认证信息;

ceph auth del osd.17

 *清除旧的认证key信息,新的磁盘加入集群后会重新生产认证密钥;

 

9、  从集群中移除OSD;

ceph osd rm 17

 

10、     卸载OSD磁盘挂载点;

umount /var/lib/ceph/osd/ceph-17

 

11、     IDC维保服务商更换新的磁盘;

 

*根据确认的服务器SN和对应的磁盘槽位,将旧的磁盘取出,更换新的磁盘;

*登入IDRAC将磁盘调整为非RAID模式;

 

12、     初始化新的磁盘;

ceph-deploy disk zap Ceph-BXXAXX-04-S2-60.domain.tld:sdp

 *更换新的磁盘后,有可能磁盘的盘符会变掉,初始化新的磁盘前,最好确认清楚(建议lsblk确认);

 

13、     OSD进程初始化;

ceph-deploy osd create Ceph-BXXAXX-04-S2-60.domain.tld:sdp

 *如果更换的是旧盘,这一步操作有可能失败报错,此时最好用dd(或者分区工具)摧毁磁盘上残留的Raid或分区表信息,然后重新初始化尝试;

 

14、     调整Crushmap;

ceph osd crush add osd.17 1.17 root=sata pod=sata-pod-01 chassis=sata-chassis-01 host=Ceph-B**A**-04-S2-60-sata

*新的OSD权重和位置信息,可以使用ceph osd tree确认服务器上其他OSD的位置和权重做参考;

 

15、     优化挂载参数;

mount -o remount,nobarrier /var/lib/ceph/osd/ceph-17

 *根据需要做优化调整,主要是优化filestore做后端存储引擎时,文件系统层面的提升;

*ceph –s关注集群在数据回填(backfill)中,集群读写IO;

 

16、     完成。

 

 

二、 关机、重启

1、  查看确认Ceph集群状态;

ceph –s

*主要是看Ceph集群当前的状态,有没有pg在做scrub,有没有OSD down、out,数据有没有在做大量的backfill,remap等;

 

2、  设置OSD NOOUT标志;

ceph osd set nooout

 

3、  关闭本机上所有的OSD进程服务;

stop ceph-osd-all

*等待所有的OSD进程关闭,使用ps aux |grep ceph确认;

 

4、  关机或重启;

sync;sync;shutdown –h now

or

sync;sync;reboot

*如果更换、扩容内存,关机后可以操作;

*如果迁移服务器,关机之后可以迁移(在服务器开启前,网络要保证和迁移前相同);

 

5、  开机或重启后,启动OSD进程服务;

start ceph-osd-all-starter

 

6、  取消OSD NOOUT标志;

ceph osd unset noout

 

7、  查看集群状态;

ceph –s

*确认集群恢复到正常状态;

 

8、  完成。

 

 

三、PG异常状态修复

常见状态说明

状态(state)

说明(description)

Active

Ceph可正常处理此pg请求

Clean

PG内所有的对象都被正确的复制了对应的份数

Down

一个包含必备数据的副本离线,所以PG也离线了

Degraded

PG中的一些对象还没有被复制到规定的份数

Inconsistent

Ceph检测到PG中对象的一份或多份数据不一致

Peering

PG正在互联过程中

Recovering

Ceph正在迁移/同步对象和其副本

Incomplete

Ceph 探测到某一PG可能丢失了写入信息,或者没有健康的副本

Stale

PG状态未知,从PG mapping更新后Monitor一直没有收到更新

 

1、 查找Inconsistent pg并修复

      ceph pg dump | grep insconsistent

      or

      ceph health detail

     ceph pg repair {pgnum}

 

*如果修复不好,可以尝试ceph pg scrub {pgnum}或ceph pg deep-scrub {pgnum}

 

2、 查找stale pg 并修复

     ceph pg dump |grep stale

     

      ceph pg map {pgname}

      ceph pg force_create_pg {pgnum}

 

 

四、对象存储帐户管理

1、 创建帐户;

root@node1:~# radosgw-admin user create --display-name="Johnny wick" --uid=Johnny

{

    "user_id": "Johnny",

    "display_name": "Johnny wick",

    "email": "",

    "suspended": 0,

    "max_buckets": 1000,

    "auid": 0,

    "subusers": [],

    "keys": [

        {

            "user": "Johnny",

            "access_key": "S54GYGAB21NYEF765DSV",

            "secret_key": "i6AswOcSF5xaeLm4sahRDpYpClpVTYe6FURWuDrU"

        }

    ],

    "swift_keys": [],

    "caps": [],

    "op_mask": "read, write, delete",

    "default_placement": "",

    "placement_tags": [],

    "bucket_quota": {

        "enabled": true,

        "max_size_kb": -1,

        "max_objects": 100

    },

    "user_quota": {

        "enabled": false,

        "max_size_kb": -1,

        "max_objects": -1

    },

    "temp_url_keys": []

}

 

2、 停用帐户;

root@node1:~# radosgw-admin user suspend --uid=Johnny

{

    "user_id": "Johnny",

    "display_name": "Johnny wick",

    "email": "",

    "suspended": 1,

    "max_buckets": 1000,

    "auid": 0,

    "subusers": [],

    "keys": [

        {

            "user": "Johnny",

            "access_key": "S54GYGAB21NYEF765DSV",

            "secret_key": "i6AswOcSF5xaeLm4sahRDpYpClpVTYe6FURWuDrU"

        }

    ],

    "swift_keys": [],

    "caps": [],

    "op_mask": "read, write, delete",

    "default_placement": "",

    "placement_tags": [],

    "bucket_quota": {

        "enabled": true,

        "max_size_kb": -1,

        "max_objects": 100

    },

    "user_quota": {

        "enabled": false,

        "max_size_kb": -1,

        "max_objects": -1

    },

    "temp_url_keys": []

}

 

3、 启用帐户;

root@node1:~# radosgw-admin user enable --uid=Johnny

{

    "user_id": "Johnny",

    "display_name": "Johnny wick",

    "email": "",

    "suspended": 0,

    "max_buckets": 1000,

    "auid": 0,

    "subusers": [],

    "keys": [

        {

            "user": "Johnny",

            "access_key": "S54GYGAB21NYEF765DSV",

            "secret_key": "i6AswOcSF5xaeLm4sahRDpYpClpVTYe6FURWuDrU"

        }

    ],

    "swift_keys": [],

    "caps": [],

    "op_mask": "read, write, delete",

    "default_placement": "",

    "placement_tags": [],

    "bucket_quota": {

        "enabled": true,

        "max_size_kb": -1,

        "max_objects": 100

    },

    "user_quota": {

        "enabled": false,

        "max_size_kb": -1,

        "max_objects": -1

    },

   "temp_url_keys": []

}

 

4、 删除帐户;

root@node1:~# radosgw-admin user rm --uid=Johnny

root@node1:~# radosgw-admin user info --uid=Johnny

could not fetch user info: no user info saved

 

5、 查看帐户;

root@node1:~# radosgw-admin user info --uid=Johnny

{

    "user_id": "Johnny",

    "display_name": "Johnny wick",

    "email": "",

    "suspended": 0,

    "max_buckets": 1000,

    "auid": 0,

    "subusers": [],

    "keys": [

        {

            "user": "Johnny",

            "access_key": "B6WXEJ94UG9V68VFD3UK",

            "secret_key": "zaSwdDGSIWtWKYRp5dXQJymGXuzQrYa9pnaK4QxM"

        }

    ],

    "swift_keys": [],

    "caps": [],

    "op_mask": "read, write, delete",

    "default_placement": "",

    "placement_tags": [],

    "bucket_quota": {

        "enabled": true,

        "max_size_kb": -1,

        "max_objects": 100

    },

    "user_quota": {

        "enabled": false,

        "max_size_kb": -1,

        "max_objects": -1

    },

    "temp_url_keys": []

}

6、 设置用户配额;

root@node1:~# radosgw-admin quota set --quota-scope=user --uid=Johnny --max-objects=2048 --max-size=204800

root@node1:~# radosgw-admin user info --uid=Johnny

{

    "user_id": "Johnny",

    "display_name": "Johnny wick",

    "email": "",

    "suspended": 0,

    "max_buckets": 1000,

    "auid": 0,

    "subusers": [],

    "keys": [

        {

            "user": "Johnny",

            "access_key": "B6WXEJ94UG9V68VFD3UK",

            "secret_key": "zaSwdDGSIWtWKYRp5dXQJymGXuzQrYa9pnaK4QxM"

        }

    ],

    "swift_keys": [],

    "caps": [],

    "op_mask": "read, write, delete",

    "default_placement": "",

    "placement_tags": [],

    "bucket_quota": {

        "enabled": true,

        "max_size_kb": -1,

        "max_objects": 100

    },

    "user_quota": {

       "enabled": false,

       "max_size_kb": 200,

       "max_objects": 2048

   },

    "temp_url_keys": []

}

 

root@node1:~# radosgw-admin quota enable --quota-scope=user --uid=Johnny

root@node1:~# radosgw-admin user info --uid=Johnny

{

    "user_id": "Johnny",

    "display_name": "Johnny wick",

    "email": "",

    "suspended": 0,

    "max_buckets": 1000,

    "auid": 0,

    "subusers": [],

    "keys": [

        {

            "user": "Johnny",

            "access_key": "B6WXEJ94UG9V68VFD3UK",

            "secret_key": "zaSwdDGSIWtWKYRp5dXQJymGXuzQrYa9pnaK4QxM"

        }

    ],

    "swift_keys": [],

    "caps": [],

    "op_mask": "read, write, delete",

    "default_placement": "",

    "placement_tags": [],

    "bucket_quota": {

        "enabled": true,

        "max_size_kb": -1,

        "max_objects": 100

    },

    "user_quota": {

        "enabled": true,

        "max_size_kb": 200,

        "max_objects": 2048

    },

    "temp_url_keys": []

}

 

7、 设置bucket配额;

root@node1:~# radosgw-admin quota set quota-scope=bucket --uid=Johnny --bucket=bucket-test01 --max-objects=512 --max-size=51200

root@node1:~# radosgw-admin quota set quota-scope=bucket --uid=Johnny --bucket=bucket-test02 --max-objects=1024 --max-size=102400

root@node1:~# radosgw-admin quota enable --quota-scope=bucket --uid=Johnny --bucket=bucket-test01

root@node1:~# radosgw-admin quota enable --quota-scope=bucket --uid=Johnny --bucket=bucket-test02

root@node1:~# radosgw-admin bucket stats --bucket=bucket-test02

{

    "bucket": "bucket-test02",

    "pool": ".rgw.buckets",

    "index_pool": ".rgw.buckets.index",

    "id": "default.124124.2",

    "marker": "default.124124.2",

    "owner": "Johnny",

    "ver": "0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1,11#1,12#1,13#1,14#1,15#1,16#1,17#1,18#1,19#1,20#1,21#1,22#1,23#1,24#1,25#1,26#1,27#1,28#1,29#1,30#1,31#1,32#1,33#1,34#1,35#1,36#1,37#1,38#1,39#1,40#1,41#1,42#1,43#1,44#1,45#1,46#1,47#1,48#1,49#1,50#1,51#1,52#1,53#1,54#1,55#1,56#1,57#1,58#1,59#1,60#1,61#1,62#1,63#1,64#1,65#1,66#1,67#1,68#1,69#1,70#1,71#1,72#1,73#1,74#1,75#1,76#1,77#1,78#1,79#1,80#1,81#1,82#1,83#1,84#1,85#1,86#1,87#1,88#1,89#1,90#1,91#1,92#1,93#1,94#1,95#1,96#1,97#1,98#1,99#1,100#1,101#1,102#1,103#1,104#1,105#1,106#1,107#1,108#1,109#1,110#1,111#1,112#1,113#1,114#1,115#1,116#1,117#1,118#1,119#1,120#1,121#1,122#1,123#1,124#1,125#1,126#1,127#1",

    "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0",

    "mtime": "2021-04-14 10:33:46.000000",

    "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#",

    "usage": {},

    "bucket_quota": {

       "enabled": true,

       "max_size_kb": 100,

       "max_objects": 1024

   }

}

root@node1:~# radosgw-admin bucket stats --bucket=bucket-test01

{

    "bucket": "bucket-test01",

    "pool": ".rgw.buckets",

    "index_pool": ".rgw.buckets.index",

    "id": "default.124118.1",

    "marker": "default.124118.1",

    "owner": "Johnny",

    "ver": "0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1,11#1,12#1,13#1,14#1,15#1,16#1,17#1,18#1,19#1,20#1,21#1,22#1,23#1,24#1,25#1,26#1,27#1,28#1,29#1,30#1,31#1,32#1,33#1,34#1,35#1,36#1,37#1,38#1,39#1,40#1,41#1,42#1,43#1,44#1,45#1,46#1,47#1,48#1,49#1,50#1,51#1,52#1,53#1,54#1,55#1,56#1,57#1,58#1,59#1,60#1,61#1,62#1,63#1,64#1,65#1,66#1,67#1,68#1,69#1,70#1,71#1,72#1,73#1,74#1,75#1,76#1,77#1,78#1,79#1,80#1,81#1,82#1,83#1,84#1,85#1,86#1,87#1,88#1,89#1,90#1,91#1,92#1,93#1,94#1,95#1,96#1,97#1,98#1,99#1,100#1,101#1,102#1,103#1,104#1,105#1,106#1,107#1,108#1,109#1,110#1,111#1,112#1,113#1,114#1,115#1,116#1,117#1,118#1,119#1,120#1,121#1,122#1,123#1,124#1,125#1,126#1,127#1",

    "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0",

    "mtime": "2021-04-14 10:33:45.000000",

    "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#",

    "usage": {},

    "bucket_quota": {

       "enabled": true,

       "max_size_kb": 50,

       "max_objects": 512

   }

}

8、 禁用配额

root@node1:~# radosgw-admin quota disable --quota-scope=user --uid=Johnny

root@node1:~# radosgw-admin user info --uid=Johnny

{

    "user_id": "Johnny",

    "display_name": "Johnny wick",

    "email": "",

    "suspended": 0,

    "max_buckets": 1000,

    "auid": 0,

    "subusers": [],

    "keys": [

        {

            "user": "Johnny",

          

以上是关于Ceph常用操作维护手册的主要内容,如果未能解决你的问题,请参考以下文章

Ceph运维存储 命令管理操作

Ceph运维ceph 日常运维

(转)Ceph常规操作及常见问题梳理

ceph radosgw-admin的操作

ceph集群常用命令

VIM常用操作手册