鏇挎崲OSD鎿嶄綔鐨勪紭鍖栦笌鍒嗘瀽

Posted 2022-09-06

鏍囩锛?a href='http://www.mamicode.com/so/1/%e6%b3%a8%e6%84%8f' title='娉ㄦ剰'>娉ㄦ剰   grep   get   roo   鍒ゆ柇   lin   鍓湰   orm   add   

http://www.zphj1987.com/2016/09/19/%E6%9B%BF%E6%8D%A2OSD%E6%93%8D%E4%BD%9C%E7%9A%84%E4%BC%98%E5%8C%96%E4%B8%8E%E5%88%86%E6%9E%90/

鍓嶈█

涔嬪墠鏈夊啓杩囦竴绡?a href="http://www.zphj1987.com/2016/01/12/%E5%88%A0%E9%99%A4osd%E7%9A%84%E6%AD%A3%E7%A1%AE%E6%96%B9%E5%BC%8F/" rel="external" target="_blank">鍒犻櫎OSD鐨勬纭柟寮?/a>锛岄噷闈㈠彧鏄畝鍗曠殑璁蹭簡涓嬪垹闄ょ殑鏂瑰紡鎬庢牱鑳藉噺灏戣縼绉婚噺锛屾湰绡囧睘浜庝竴涓墿灞曪紝璁茶堪浜?Ceph 杩愮淮褰撲腑缁忓父鍑虹幇鐨勫潖鐩樻彁鎹㈢洏鐨勬楠ょ殑浼樺寲

鍩虹鐜涓ゅ彴涓绘満姣忓彴涓绘満8涓?OSD锛屼竴鍏?16 涓?OSD锛屽壇鏈缃负2锛孭G 鏁拌缃负800锛岃绠椾笅鏉ュ钩鍧囨瘡涓?OSD 涓婄殑 P G鏁扮洰涓?00涓紝鏈瘒灏嗛€氳繃鏁版嵁鏉ュ垎鏋愪笉鍚岀殑澶勭悊鏂规硶鐨勫樊鍒?/p>

寮€濮嬫祴璇曞墠鍏堟妸鐜璁剧疆涓?nbsp;noout锛岀劧鍚庨€氳繃鍋滄 OSD 鏉ユā鎷?OSD 鍑虹幇浜嗗紓甯革紝涔嬪悗杩涜涓嶅悓澶勭悊鏂规硶

娴嬭瘯涓夌鏂规硶

棣栧厛 out 涓€涓?OSD锛岀劧鍚庡墧闄?OSD锛岀劧鍚庡鍔?OSD

1. 鍋滄鎸囧畾 OSD 杩涚▼
2. out 鎸囧畾 OSD
3. crush remove 鎸囧畾 OSD
4. 澧炲姞涓€涓柊鐨?OSD

涓€鑸敓浜х幆澧冧細璁剧疆涓?nbsp;noout锛屽綋鐒朵笉璁剧疆涔熷彲浠ワ紝閭ｅ氨浜ょ粰绋嬪簭鍘绘帶鍒惰妭鐐圭殑 out锛岄粯璁ゆ槸鍦ㄨ繘绋嬪仠姝㈠悗鐨勪簲鍒嗛挓锛屾€讳箣杩欎釜鍦版柟濡傛灉鏈?out 瑙﹀彂锛屼笉绠℃槸浜轰负瑙﹀彂锛岃繕鏄嚜鍔ㄨЕ鍙戞暟鎹祦鏄竴瀹氱殑锛屾垜浠繖閲屼负浜嗕究浜庢祴璇曪紝浣跨敤鐨勬槸浜轰负瑙﹀彂锛屼笂闈㈡彁鍒扮殑棰勫埗鐜灏辨槸璁剧疆鐨?nbsp;noout

寮€濮嬫祴璇曞墠鑾峰彇鏈€鍘熷鐨勫垎甯?/p>

[root@lab8106 ~]# ceph pg dump pgs|awk 鈥榹print $1,$15鈥榺grep -v pg > pg1.txt

 

鑾峰彇褰撳墠鐨?PG 鍒嗗竷,淇濆瓨鍒版枃浠秔g1.txt锛岃繖涓?PG 鍒嗗竷璁板綍鏄?PG 鎵€鍦ㄧ殑 OSD锛岃褰曚笅鏉ワ紝鏂逛究鍚庨潰杩涜姣旇緝锛屼粠鑰屽緱鍑洪渶瑕佽縼绉荤殑鏁版嵁

鍋滄鎸囧畾鐨?OSD 杩涚▼

[root@lab8106 ~]# systemctl stop ceph-osd@15

鍋滄杩涚▼骞朵笉浼氳Е鍙戣縼绉伙紝鍙細寮曡捣 PG 鐘舵€佺殑鍙樺寲锛屾瘮濡傚師鏉ヤ富 PG 鍦ㄥ仠姝㈢殑 OSD 涓婏紝閭ｄ箞鍋滄鎺?OSD 浠ュ悗锛屽師鏉ョ殑鍓湰鐨勯偅涓?PG 灏变細瑙掕壊鍗囩骇涓轰富 PG 浜?/p>

out 鎺変竴涓?OSD

[root@lab8106 ~]# ceph osd out 15

鍦ㄨЕ鍙?out 浠ュ墠锛屽綋鍓嶇殑 PG 鐘舵€佸簲璇ユ湁 active+undersized+degraded,瑙﹀彂 out 浠ュ悗锛屾墍鏈夌殑 PG 鐨勭姸鎬佸簲璇ヤ細鎱㈡參鍙樻垚 active+clean,绛夊緟闆嗙兢姝ｅ父鍚庯紝鍐嶆鏌ヨ褰撳墠鐨?PG 鍒嗗竷鐘舵€?/p>

[root@lab8106 ~]# ceph pg dump pgs|awk 鈥榹print $1,$15鈥榺grep -v pg > pg2.txt

 

淇濆瓨褰撳墠鐨?PG 鍒嗗竷涓簆g2.txt
姣旇緝 out 鍓嶅悗鐨?PG 鐨勫彉鍖栨儏鍐碉紝涓嬮潰鏄瘮杈冨叿浣撶殑鍙樺寲鎯呭喌锛屽彧鍒楀嚭鍙樺寲鐨勯儴鍒?/p>

[root@lab8106 ~]# diff -y -W 100 pg1.txt pg2.txt --suppress-common-lines

 

杩欓噷鎴戜滑鍏冲績鐨勬槸鍙樺姩鐨勬暟鐩紝鍙粺璁″彉鍔ㄧ殑 PG 鐨勬暟鐩?/p>

[root@lab8106 ~]# diff -y -W 100 pg1.txt pg2.txt --suppress-common-lines|wc -l 102

 

绗竴娆?out 浠ュ悗鏈?02涓?PG 鐨勫彉鍔?杩欎釜鏁板瓧璁颁綇锛屽悗闈㈢殑缁熻浼氱敤鍒?/p>

浠?crush 閲岄潰鍒犻櫎 OSD

[root@lab8106 ~]# ceph osd crush remove osd.15

crush 鍒犻櫎浠ュ悗鍚屾牱浼氳Е鍙戣縼绉伙紝绛夊緟 PG 鐨勫潎琛★紝涔熷氨鏄叏閮ㄥ彉鎴?nbsp;active+clean 鐘舵€?/p>

[root@lab8106 ~]# ceph pg dump pgs|awk 鈥榹print $1,$15鈥榺grep -v pg > pg3.txt

 

鑾峰彇褰撳墠鐨?PG 鍒嗗竷鐨勭姸鎬?br />鐜板湪鏉ユ瘮杈?crush remove 鍓嶅悗鐨?PG 鍙樺姩

[root@lab8106 ~]# diff -y -W 100 pg2.txt pg3.txt --suppress-common-lines|wc -l 137

 

鎴戜滑閲嶆柊鍔犱笂鏂扮殑 OSD

[root@lab8106 ~]# ceph-deploy osd prepare lab8107:/dev/sdi [root@lab8106 ~]# ceph-deploy osd activate lab8107:/dev/sdi1

 

鍔犲畬浠ュ悗缁熻褰撳墠鐨勬柊鐨?PG 鐘舵€?/p>

[root@lab8106 ~]# ceph pg dump pgs|awk 鈥榹print $1,$15鈥榺grep -v pg > pg4.txt

 

姣旇緝鍓嶅悗鐨勫彉鍖?/p>

[root@lab8106 ~]# diff -y -W 100 pg3.txt pg4.txt --suppress-common-lines|wc -l 167

 

鏁翠釜鏇挎崲娴佺▼瀹屾瘯锛岀粺璁′笂闈㈢殑 PG 鎬荤殑鍙樺姩

102 +137 +167 = 406

涔熷氨鏄寜杩欎釜鏂规硶鐨勫彉鍔ㄤ负406涓?PG锛屽洜涓烘槸鍙湁鍙屼富鏈猴紝閲岄潰鍙兘瀛樺湪鏌愪簺鏀惧ぇ闂锛岃繖閲屼笉鍋氭繁鍏ョ殑璁ㄨ锛屽洜涓烘垜鐨勪笁缁勬祴璇曠幆澧冮兘鏄竴鏍风殑鎯呭喌锛屽彧鍋氭í鍚戞瘮杈冿紝鍘熺悊鐩搁€氾紝杩欓噷鏄敤鏁版嵁鏉ュ垎鏋愬嚭宸埆

鍏坈rush reweight 0 锛岀劧鍚巓ut锛岀劧鍚庡啀澧炲姞osd

棣栧厛鎭㈠鐜涓烘祴璇曞墠鐨勭幆澧?/p>

[root@lab8106 ~]# ceph pg dump pgs|awk 鈥榹print $1,$15鈥榺grep -v pg > 2pg1.txt

 

璁板綍鏈€鍘熷鐨?PG 鍒嗗竷鎯呭喌

crush reweight 鎸囧畾OSD

[root@lab8106 ~]# ceph osd crush reweight osd.16 0 reweighted item id 16 name 鈥榦sd.16鈥?to 0 in crush map

绛夊緟骞宠　浜嗕互鍚庤褰曞綋鍓嶇殑 PG 鍒嗗竷鐘舵€?/p>

[root@lab8106 ~]# ceph pg dump pgs|awk 鈥榹print $1,$15鈥榺grep -v pg > 2pg2.txt dumped pgs in format plain

 

姣旇緝鍓嶅悗鐨勫彉鍔?/p>

[root@lab8106 ~]# diff -y -W 100 2pg1.txt 2pg2.txt --suppress-common-lines|wc -l 166

 

crush remove 鎸囧畾 OSD

[root@lab8106 ~]# ceph osd crush remove osd.16 removed item id 16 name 鈥榦sd.16鈥?from crush map

杩欎釜鍦版柟鍥犱负涓婇潰 crush 宸茬粡鏄?浜嗘墍浠ュ垹闄や篃涓嶄細寮曡捣 PG 鍙樺姩
鐒跺悗鐩存帴 ceph osd rm osd.16 鍚屾牱娌℃湁 PG 鍙樺姩

澧炲姞鏂扮殑 OSD

[root@lab8106 ~]#ceph-deploy osd prepare lab8107:/dev/sdi [root@lab8106 ~]#ceph-deploy osd activate lab8107:/dev/sdi1

绛夊緟骞宠　浠ュ悗鑾峰彇褰撳墠鐨?PG 鍒嗗竷

[root@lab8106 ceph]# ceph pg dump pgs|awk 鈥榹print $1,$15鈥榺grep -v pg > 2pg3.txt

 

鏉ユ瘮杈冨墠鍚庣殑鍙樺寲

[root@lab8106 ~]# diff -y -W 100 2pg2.txt 2pg3.txt --suppress-common-lines|wc -l 159

 

鎬荤殑 PG 鍙樺姩涓?/p>

166+159=325

寮€濮嬪仛norebalance锛岀劧鍚庡仛crush remove锛岀劧鍚庡仛add

鎭㈠鐜涓哄垵濮嬬幆澧冿紝鐒跺悗鑾峰彇褰撳墠鐨?PG 鍒嗗竷

[root@lab8106 ~]# ceph pg dump pgs|awk 鈥榹print $1,$15鈥榺grep -v pg > 3pg1.txt dumped pgs in format plain

 

[root@lab8106 ~]# ceph osd set norebalance
set norebalance
[root@lab8106 ~]# ceph osd set nobackfill
set nobackfill
[root@lab8106 ~]# ceph osd set norecover
set norecover

 

crush reweight 鎸囧畾 OSD

[root@lab8106 ~]# ceph osd crush reweight osd.15 0 reweighted item id 15 name 鈥榦sd.15鈥?to 0 in crush map

杩欎釜鍦版柟鍥犱负宸茬粡鍋氫簡涓婇潰鐨勬爣璁帮紝鎵€浠ュ彧浼氬嚭鐜扮姸鎬佸彉鍖栵紝鑰屾病鏈夌湡姝ｇ殑杩佺Щ锛屾垜浠篃鍏堢粺璁′竴涓?/p>

[root@lab8106 ~]# ceph pg dump pgs|awk 鈥榹print $1,$15鈥榺grep -v pg > 3pg2.txt [root@lab8106 ~]# diff -y -W 100 3pg1.txt 3pg2.txt --suppress-common-lines|wc -l 158

 

娉ㄦ剰杩欓噷鍙槸璁＄畻浜嗭紝骞舵病鏈夌湡姝ｇ殑鏁版嵁鍙樺姩锛屽彲浠ラ€氳繃鐩戞帶涓ゅ彴鐨勪富鏈虹殑缃戠粶娴侀噺鏉ュ垽鏂?鎵€浠ヨ繖閲岀殑鍙樺姩骞朵笉鐢ㄨ绠楀埌闇€瑕佽縼绉荤殑 PG 鏁扮洰褰撲腑

crush remove 鎸囧畾 OSD

[root@lab8106 ~]#ceph osd crush remove osd.15

鍒犻櫎鎸囧畾鐨?OSD

鍒犻櫎浠ュ悗鍚屾牱鏄病鏈?PG 鐨勫彉鍔ㄧ殑

ceph osd rm osd.15

 

杩欎釜鍦版柟鏈変釜灏忓湴鏂归渶瑕佹敞鎰忎竴涓嬶紝涓嶅仛 ceph auth del osd.15 鎶?5鐨勭紪鍙风暀鐫€锛岃繖鏍峰ソ鍒ゆ柇鍓嶅悗鐨?PG 鐨勫彉鍖栵紝涓嶇劧鐩稿悓鐨勭紪鍙凤紝灏辨棤娉曞垽鏂槸涓嶆槸鍋氫簡杩佺Щ浜?/p>

澧炲姞鏂扮殑 OSD

[root@lab8106 ~]#ceph-deploy osd prepare lab8107:/dev/sdi [root@lab8106 ~]#ceph-deploy osd activate lab8107:/dev/sdi1

鎴戠殑鐜涓嬶紝鏂板鐨?OSD 鐨勭紪鍙蜂负16浜?/p>

瑙ｉ櫎鍚勭鏍囪

鎴戜滑鏀惧紑涓婇潰鐨勮缃紝鐪嬩笅鏁版嵁鐨勫彉鍔ㄦ儏鍐?/p>

[root@lab8106 ceph]# ceph osd unset norebalance unset norebalance [root@lab8106 ceph]# ceph osd unset nobackfill unset nobackfill [root@lab8106 ceph]# ceph osd unset norecover unset norecover

 

璁剧疆瀹屼簡鍚庢暟鎹墠鐪熸寮€濮嬪彉鍔ㄤ簡锛屽彲浠ラ€氳繃瑙傚療缃戝崱娴侀噺鐪嬪埌锛屾潵鐪嬩笅鏈€缁坧g鍙樺寲

[root@lab8106 ceph]# ceph pg dump pgs|awk 鈥榹print $1,$15鈥榺grep -v pg > 3pg3.txt dumped pgs in format plain [root@lab8106 ~]# diff -y -W 100 3pg1.txt 3pg3.txt --suppress-common-lines|wc -l 195

 

杩欓噷鎴戜滑鍙渶瑕佽窡鏈€寮€濮嬬殑 PG 鍒嗗竷鐘跺喌杩涜姣旇緝灏卞彲浠ヤ簡锛屽洜涓轰腑闂寸殑鐘舵€佸疄闄呬笂閮芥病鏈夊仛鏁版嵁鐨勮縼绉伙紝鎵€浠ヤ笉闇€瑕佺粺璁¤繘鍘伙紝鍙互鐪嬪埌杩欎釜鍦版柟鍔ㄤ簡195涓?PG
鎬诲叡鐨?PG 杩佺Щ閲忎负

195

鎬荤粨

浠庢垜鑷繁鐨勬搷浣滅粡楠屾潵鐪嬶紝鏈€寮€濮嬫槸鐢ㄧ殑绗竴绉嶆柟娉曪紝鍚庨潰灏辩敤绗簩绉嶆柟娉曞噺灏戜簡涓€閮ㄥ垎杩佺Щ閲忥紝鏈€杩戠湅鍒拌祫鏂欏啓鍋氬墧闄SD鐨勬椂鍊欏彲浠ュ叧闂縼绉婚槻姝㈡棤鏁堢殑杩囧鐨勮縼绉伙紝鐒跺悗灏辨祴璇曚簡涓€涓嬶紝纭疄鑳藉鍑忓皯涓嶅皯鐨勮縼绉婚噺锛岃繖涓噺灏戝湪鏌愪簺鍦烘櫙涓嬭繕鏄緢濂界殑锛屽綋鐒跺鏋滀笉澶啛鎮夛紝鐢ㄥ摢涓€绉嶉兘鍙互锛屾渶缁堣兘杈惧埌鐨勭洰鐨勬槸涓€鏍风殑

闄勫姞

鏈変汉闂埌涓€涓棶棰橈紝涓轰粈涔堟寜鐓ц繖涓祦绋嬫搷浣滅殑鏃跺€欙紝浼氬嚭鐜皊low request锛熷湪杩涜浜嗕竴娆￠獙璇佸悗锛屽彂鐜板湪杩佺Щ杩囩▼涓殑璇锋眰璺緞杩樻槸寰堥暱鐨勶紝鎵€浠ュ嚭鐜皊low request杩樻槸寰堝鏄撶殑

鍋囧鎴戜滑鏈変笁涓猳sd锛屽垎鍒负0,1,2锛岄噷闈㈡湁鍚勭鐨勫垎甯冿紝鎴戜滑鍦ㄨ涪鎺変竴涓猳sd.2鍚庯紝鍙兘鍑虹幇鐨勪竴涓儏鍐垫槸
鏌愪釜PG(0.3b)鐨刐2,0]鍒嗗竷鍙樻垚浜哰1,0]
鑰屾鏃跺悗鍙扮殑osd.1鐨凱G锛?.3b锛夎繖涓洰褰曢噷闈㈢殑鍐呭瀹為檯鏄┖鐨勶紝濡傛灉杩欎釜鏃跺€欙紝鍓嶇鐨勮姹備竴涓璞℃濂芥槸鍒嗗竷鍦?.3b杩欎釜PG涓婄殑鏃跺€欙紝鍚庡彴闇€瑕佸厛灏唎sd.0涓婇潰鐨勮繖涓?.3b鐨勫璞″啓鍏ュ埌osd.1鐨?.3b鐨刾g閲岄潰鍘伙紝鐒跺悗鍐嶅幓鍝嶅簲瀹㈡埛绔殑璇锋眰锛岃嚜鐒惰矾寰勫氨闀夸簡锛屽鏋滆繖鏍风殑璇锋眰涓€澶氾紝鍝嶅簲鍓嶅彴鐨勬€ц兘灏辨湁闂浜嗭紝澧炲姞鑺傜偣鐨勬椂鍊欏悓鐞?/p>

璇锋眰鍒拌繖绉嶇┖PG鐨勫璞★紝PG鐨勭姸鎬佷細杩欐牱鍙樺寲锛?/p>

浠巃ctive+degraded 鍙樻垚active+recovery_wait+degraded

杩佺Щ鐨勬暟鎹噺鏄竴瀹氱殑锛岃繖涓湅鏄姹傜殑鏃跺€欏疄鏃惰縼绉荤劧鍚庡搷搴旇繕鏄彁鍓嶈縼绉伙紝鐒跺悗鍝嶅簲锛屾墍浠ヨ繖涓腑闂存搷浣滆繃绋嬪敖閲忕殑蹇殑瀹屾垚锛岀劧鍚庡ソ杩佺Щ瀹屽搷搴斿墠绔殑璇锋眰