Ceph常用操作维护手册
Posted John08
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Ceph常用操作维护手册相关的知识，希望对你有一定的参考价值。
一、更换磁盘
1、 确认需要更换的磁盘对应的盘符、OSD ID；
root@Ceph-B**A**-04-S2-60:~# df -h |grep 17
/dev/sdp3 1.1T 566G 551G 51% /var/lib/ceph/osd/ceph-17
*sdp为服务器上对应的盘符；
*ceph-17 17为OSD对应的ID；
2、 确认磁盘在服务器上对应的槽位；
root@Ceph-B**A**-04-S2-60:~# ll /dev/disk/by-path/ |grep sdp
lrwxrwxrwx 1 root root 9 Aug 17 19:17 pci-0000:02:00.0-scsi-0:0:18:0 -> ../../sdp
lrwxrwxrwx 1 root root 10 Aug 17 19:17 pci-0000:02:00.0-scsi-0:0:18:0-part1 -> ../../sdp1
lrwxrwxrwx 1 root root 10 Aug 17 19:17 pci-0000:02:00.0-scsi-0:0:18:0-part2 -> ../../sdp2
lrwxrwxrwx 1 root root 10 Aug 17 19:17 pci-0000:02:00.0-scsi-0:0:18:0-part3 -> ../../sdp3
*根据盘符找到服务器上对应的磁盘槽位，scsi-0:0:18:0标识第18号槽位；
3、 确认主机的SN；
root@Ceph-B**A**-04-S2-60:~# dmidecode -s system-serial-number
HVNHB**
4、 在Crushmap设置故障OSD OUT标志；
ceph osd out 17
*设置OUT标识，告诉集群不要再往这个OSD磁盘上读写数据；
5、 查看集群状态；
ceph –s
*确认OSD设置OUT标志后，集群有没有出现异常；
6、 关闭OSD对应的实例进程；
stop cep-osd id=17
7、 从Crushmap中移除OSD；
ceph osd crush remove osd.17
8、 删除集群中OSD认证信息；
ceph auth del osd.17
*清除旧的认证key信息，新的磁盘加入集群后会重新生产认证密钥；
9、 从集群中移除OSD；
ceph osd rm 17
10、 卸载OSD磁盘挂载点；
umount /var/lib/ceph/osd/ceph-17
11、 IDC维保服务商更换新的磁盘；
*根据确认的服务器SN和对应的磁盘槽位，将旧的磁盘取出，更换新的磁盘；
*登入IDRAC将磁盘调整为非RAID模式；
12、 初始化新的磁盘；
ceph-deploy disk zap Ceph-BXXAXX-04-S2-60.domain.tld:sdp
*更换新的磁盘后，有可能磁盘的盘符会变掉，初始化新的磁盘前，最好确认清楚（建议lsblk确认）；
13、 OSD进程初始化；
ceph-deploy osd create Ceph-BXXAXX-04-S2-60.domain.tld:sdp
*如果更换的是旧盘，这一步操作有可能失败报错，此时最好用dd（或者分区工具）摧毁磁盘上残留的Raid或分区表信息，然后重新初始化尝试；
14、 调整Crushmap；
ceph osd crush add osd.17 1.17 root=sata pod=sata-pod-01 chassis=sata-chassis-01 host=Ceph-B**A**-04-S2-60-sata
*新的OSD权重和位置信息，可以使用ceph osd tree确认服务器上其他OSD的位置和权重做参考；
15、 优化挂载参数；
mount -o remount,nobarrier /var/lib/ceph/osd/ceph-17
*根据需要做优化调整，主要是优化filestore做后端存储引擎时，文件系统层面的提升；
*ceph –s关注集群在数据回填（backfill）中，集群读写IO；
16、 完成。
二、 关机、重启
1、 查看确认Ceph集群状态；
ceph –s
*主要是看Ceph集群当前的状态，有没有pg在做scrub，有没有OSD down、out，数据有没有在做大量的backfill，remap等；
2、 设置OSD NOOUT标志；
ceph osd set nooout
3、 关闭本机上所有的OSD进程服务；
stop ceph-osd-all
*等待所有的OSD进程关闭，使用ps aux |grep ceph确认；
4、 关机或重启；
sync；sync；shutdown –h now
or
sync;sync;reboot
*如果更换、扩容内存，关机后可以操作；
*如果迁移服务器，关机之后可以迁移（在服务器开启前，网络要保证和迁移前相同）；
5、 开机或重启后，启动OSD进程服务；
start ceph-osd-all-starter
6、 取消OSD NOOUT标志；
ceph osd unset noout
7、 查看集群状态；
ceph –s
*确认集群恢复到正常状态；
8、 完成。
三、PG异常状态修复
常见状态说明
|
状态（state） |
说明(description) |
|
Active |
Ceph可正常处理此pg请求 |
|
Clean |
PG内所有的对象都被正确的复制了对应的份数 |
|
Down |
一个包含必备数据的副本离线，所以PG也离线了 |
|
Degraded |
PG中的一些对象还没有被复制到规定的份数 |
|
Inconsistent |
Ceph检测到PG中对象的一份或多份数据不一致 |
|
Peering |
PG正在互联过程中 |
|
Recovering |
Ceph正在迁移/同步对象和其副本 |
|
Incomplete |
Ceph 探测到某一PG可能丢失了写入信息，或者没有健康的副本 |
|
Stale |
PG状态未知，从PG mapping更新后Monitor一直没有收到更新 |
1、 查找Inconsistent pg并修复
ceph pg dump | grep insconsistent
or
ceph health detail
ceph pg repair {pgnum}
*如果修复不好，可以尝试ceph pg scrub {pgnum}或ceph pg deep-scrub {pgnum}
2、 查找stale pg 并修复
ceph pg dump |grep stale
ceph pg map {pgname}
ceph pg force_create_pg {pgnum}
四、对象存储帐户管理
1、 创建帐户；
root@node1:~# radosgw-admin user create --display-name="Johnny wick" --uid=Johnny
{
"user_id": "Johnny",
"display_name": "Johnny wick",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "Johnny",
"access_key": "S54GYGAB21NYEF765DSV",
"secret_key": "i6AswOcSF5xaeLm4sahRDpYpClpVTYe6FURWuDrU"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": true,
"max_size_kb": -1,
"max_objects": 100
},
"user_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"temp_url_keys": []
}
2、 停用帐户；
root@node1:~# radosgw-admin user suspend --uid=Johnny
{
"user_id": "Johnny",
"display_name": "Johnny wick",
"email": "",
"suspended": 1,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "Johnny",
"access_key": "S54GYGAB21NYEF765DSV",
"secret_key": "i6AswOcSF5xaeLm4sahRDpYpClpVTYe6FURWuDrU"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": true,
"max_size_kb": -1,
"max_objects": 100
},
"user_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"temp_url_keys": []
}
3、 启用帐户；
root@node1:~# radosgw-admin user enable --uid=Johnny
{
"user_id": "Johnny",
"display_name": "Johnny wick",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "Johnny",
"access_key": "S54GYGAB21NYEF765DSV",
"secret_key": "i6AswOcSF5xaeLm4sahRDpYpClpVTYe6FURWuDrU"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": true,
"max_size_kb": -1,
"max_objects": 100
},
"user_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"temp_url_keys": []
}
4、 删除帐户；
root@node1:~# radosgw-admin user rm --uid=Johnny
root@node1:~# radosgw-admin user info --uid=Johnny
could not fetch user info: no user info saved
5、 查看帐户；
root@node1:~# radosgw-admin user info --uid=Johnny
{
"user_id": "Johnny",
"display_name": "Johnny wick",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "Johnny",
"access_key": "B6WXEJ94UG9V68VFD3UK",
"secret_key": "zaSwdDGSIWtWKYRp5dXQJymGXuzQrYa9pnaK4QxM"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": true,
"max_size_kb": -1,
"max_objects": 100
},
"user_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"temp_url_keys": []
}
6、 设置用户配额；
root@node1:~# radosgw-admin quota set --quota-scope=user --uid=Johnny --max-objects=2048 --max-size=204800
root@node1:~# radosgw-admin user info --uid=Johnny
{
"user_id": "Johnny",
"display_name": "Johnny wick",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "Johnny",
"access_key": "B6WXEJ94UG9V68VFD3UK",
"secret_key": "zaSwdDGSIWtWKYRp5dXQJymGXuzQrYa9pnaK4QxM"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": true,
"max_size_kb": -1,
"max_objects": 100
},
"user_quota": {
"enabled": false,
"max_size_kb": 200,
"max_objects": 2048
},
"temp_url_keys": []
}
root@node1:~# radosgw-admin quota enable --quota-scope=user --uid=Johnny
root@node1:~# radosgw-admin user info --uid=Johnny
{
"user_id": "Johnny",
"display_name": "Johnny wick",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "Johnny",
"access_key": "B6WXEJ94UG9V68VFD3UK",
"secret_key": "zaSwdDGSIWtWKYRp5dXQJymGXuzQrYa9pnaK4QxM"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": true,
"max_size_kb": -1,
"max_objects": 100
},
"user_quota": {
"enabled": true,
"max_size_kb": 200,
"max_objects": 2048
},
"temp_url_keys": []
}
7、 设置bucket配额；
root@node1:~# radosgw-admin quota set quota-scope=bucket --uid=Johnny --bucket=bucket-test01 --max-objects=512 --max-size=51200
root@node1:~# radosgw-admin quota set quota-scope=bucket --uid=Johnny --bucket=bucket-test02 --max-objects=1024 --max-size=102400
root@node1:~# radosgw-admin quota enable --quota-scope=bucket --uid=Johnny --bucket=bucket-test01
root@node1:~# radosgw-admin quota enable --quota-scope=bucket --uid=Johnny --bucket=bucket-test02
root@node1:~# radosgw-admin bucket stats --bucket=bucket-test02
{
"bucket": "bucket-test02",
"pool": ".rgw.buckets",
"index_pool": ".rgw.buckets.index",
"id": "default.124124.2",
"marker": "default.124124.2",
"owner": "Johnny",
"ver": "0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1,11#1,12#1,13#1,14#1,15#1,16#1,17#1,18#1,19#1,20#1,21#1,22#1,23#1,24#1,25#1,26#1,27#1,28#1,29#1,30#1,31#1,32#1,33#1,34#1,35#1,36#1,37#1,38#1,39#1,40#1,41#1,42#1,43#1,44#1,45#1,46#1,47#1,48#1,49#1,50#1,51#1,52#1,53#1,54#1,55#1,56#1,57#1,58#1,59#1,60#1,61#1,62#1,63#1,64#1,65#1,66#1,67#1,68#1,69#1,70#1,71#1,72#1,73#1,74#1,75#1,76#1,77#1,78#1,79#1,80#1,81#1,82#1,83#1,84#1,85#1,86#1,87#1,88#1,89#1,90#1,91#1,92#1,93#1,94#1,95#1,96#1,97#1,98#1,99#1,100#1,101#1,102#1,103#1,104#1,105#1,106#1,107#1,108#1,109#1,110#1,111#1,112#1,113#1,114#1,115#1,116#1,117#1,118#1,119#1,120#1,121#1,122#1,123#1,124#1,125#1,126#1,127#1",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0",
"mtime": "2021-04-14 10:33:46.000000",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#",
"usage": {},
"bucket_quota": {
"enabled": true,
"max_size_kb": 100,
"max_objects": 1024
}
}
root@node1:~# radosgw-admin bucket stats --bucket=bucket-test01
{
"bucket": "bucket-test01",
"pool": ".rgw.buckets",
"index_pool": ".rgw.buckets.index",
"id": "default.124118.1",
"marker": "default.124118.1",
"owner": "Johnny",
"ver": "0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1,11#1,12#1,13#1,14#1,15#1,16#1,17#1,18#1,19#1,20#1,21#1,22#1,23#1,24#1,25#1,26#1,27#1,28#1,29#1,30#1,31#1,32#1,33#1,34#1,35#1,36#1,37#1,38#1,39#1,40#1,41#1,42#1,43#1,44#1,45#1,46#1,47#1,48#1,49#1,50#1,51#1,52#1,53#1,54#1,55#1,56#1,57#1,58#1,59#1,60#1,61#1,62#1,63#1,64#1,65#1,66#1,67#1,68#1,69#1,70#1,71#1,72#1,73#1,74#1,75#1,76#1,77#1,78#1,79#1,80#1,81#1,82#1,83#1,84#1,85#1,86#1,87#1,88#1,89#1,90#1,91#1,92#1,93#1,94#1,95#1,96#1,97#1,98#1,99#1,100#1,101#1,102#1,103#1,104#1,105#1,106#1,107#1,108#1,109#1,110#1,111#1,112#1,113#1,114#1,115#1,116#1,117#1,118#1,119#1,120#1,121#1,122#1,123#1,124#1,125#1,126#1,127#1",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0",
"mtime": "2021-04-14 10:33:45.000000",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#",
"usage": {},
"bucket_quota": {
"enabled": true,
"max_size_kb": 50,
"max_objects": 512
}
}
8、 禁用配额
root@node1:~# radosgw-admin quota disable --quota-scope=user --uid=Johnny
root@node1:~# radosgw-admin user info --uid=Johnny
{
"user_id": "Johnny",
"display_name": "Johnny wick",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "Johnny",
以上是关于Ceph常用操作维护手册的主要内容，如果未能解决你的问题，请参考以下文章