閲嶆俯澶ф暟鎹?--Hbase閮ㄧ讲浠ュ強鏋舵瀯鍒嗘瀽

Posted 2021-04-18 浜掓壇绋嬪簭

杩欑瘒鏂囩珷涓昏浠嬬粛涓€涓嬶紝Hbase鏄粈涔堬紝鑳藉共浠€涔堬紝骞茬殑鎬庝箞鏍枫€備互鍙奌base鐨勫畨瑁呴厤缃拰璋冭瘯绛夈€傛€荤殑鏉ヨHbase鍦ㄥぇ鏁版嵁鐨勭敓鎬佺郴缁熶腑鍗犳湁寰堥噸瑕佺殑浣嶇疆锛屽浜嶩base鐨勫涔犺繕闇€瑕佹洿鍔犳繁鍏ユ墠琛屻€傚浜庡ぇ鏁版嵁鐨勫唴瀹规垜閮藉缓璁鐪嬬湅瀹樻柟鏂囨。锛屼互鍓嶆垜鎬昏寰楄嚜宸辩湅涓嶆噦鍝簺楂樺ぇ涓婄殑锛屽叾瀹炰笉鐒躲€傪煒戰煒岎煒屾劅瑙夊畼鏂规枃妗ｅ啓鐨勯兘寰堜翰姘戙€?/p>

Hbase

HBase鏄竴涓垎甯冨紡鐨勩€侀潰鍚戝垪鐨勫紑婧愭暟鎹簱锛岃鎶€鏈潵婧愪簬 Fay Chang 鎵€鎾板啓鐨凣oogle璁烘枃鈥淏igtable锛氫竴涓粨鏋勫寲鏁版嵁鐨勫垎甯冨紡瀛樺偍绯荤粺鈥濄€傚氨鍍廈igtable鍒╃敤浜咷oogle鏂囦欢绯荤粺锛團ile System锛夋墍鎻愪緵鐨勫垎甯冨紡鏁版嵁瀛樺偍涓€鏍凤紝HBase鍦℉adoop涔嬩笂鎻愪緵浜嗙被浼间簬Bigtable鐨勮兘鍔涖€侶Base鏄疉pache鐨凥adoop椤圭洰鐨勫瓙椤圭洰銆侶Base涓嶅悓浜庝竴鑸殑鍏崇郴鏁版嵁搴擄紝瀹冩槸涓€涓€傚悎浜庨潪缁撴瀯鍖栨暟鎹瓨鍌ㄧ殑鏁版嵁搴撱€傚彟涓€涓笉鍚岀殑鏄疕Base鍩轰簬鍒楃殑鑰屼笉鏄熀浜庤鐨勬ā寮忋€?- 鐧惧害鐧剧

绠€鍗曠殑璇碒Base灏辨槸涓€涓垎甯冨紡鐨勫彲鎵╁睍鐨勫ぇ鏁版嵁閲忕殑闈炲叧绯诲瀷鏁版嵁搴擄紙NoSQL锛夈€傚畠鍏锋湁涓€鑸殑鍏崇郴鍨嬫暟鎹?Oracle/mysql鐨勫熀纭€鍔熻兘濡傦細

• 瀛樺偍鏁版嵁

• 妫€绱㈡暟鎹?/span>

• 鏌ヨ鏁版嵁

Hbase鑳藉仛鐨勪粈涔?/strong>

Hbase 浣滀负涓€涓潪鍏崇郴鍨嬫暟鎹簱锛岄鍏堝氨鏄鏁版嵁鐨勫熀纭€鐨勫瓨鍌紝鍏舵鏄彲浠ユ彁渚涗紶缁熷叧绯诲瀷鏁版嵁搴撹揪涓嶅埌鐨勪嚎绾ф暟鎹噺瀹炴椂鐨勬煡璇€?/p>

• 娴烽噺鏁版嵁鐨勫瓨鍌?/span>

• 娴烽噺鏁版嵁鐨勬煡璇?/span>

Hbase鍦ㄤ笂浜跨殑鏁版嵁琛ㄩ噷鑳藉仛鍒扮绾у埆鏌ヨ锛堝疄鏃舵煡璇級

Hbase浣跨敤鍦烘櫙

• 浜ら€氭暟鎹?/span>

• 璐﹀崟鏁版嵁

• 娓告垙鏁版嵁

• 鐢靛晢浜ゆ槗鏁版嵁

涓€鍒囧ぇ鏁版嵁閲忕殑锛屽瀹炴椂鏌ヨ渚濊禆楂樼殑鐨勫満鏅兘鑳借В鍐炽€?/p>

Hbase鍦ㄧ敓鎬佸湀涓殑浣嶇疆

鍥句腑鍙锛孒base鏄熀浜嶩DFS鐨勶紝涔熷氨鏄Hbase鐨勬暟鎹槸瀛樺偍鍦℉DFS涓婇潰鐨勶紝瀹冨彲浠ュ緢濂藉湴缁ф壙MapReduce鍜孒ive鏉ヨ繘琛屾暟鎹殑澶勭悊銆?/p>

Hbase鏁版嵁瀛樺偍鏂瑰紡瀛樺偍

Hbase鏄潪鍏崇郴鍨嬫暟鎹簱涓吀鍨嬬殑鍒楀紡瀛樺偍鏁版嵁搴?/p>

Hbase鏁版嵁鐨勭粍鎴愬厓绱?/strong>

Hbase涓暟鎹槸浠ュ瓧鑺傛暟缁勮繘琛屽瓨鍌紝涓嶅瓨鍦ㄦ暟鎹被鍨嬨€傞粯璁ゆ暟鎹笁浠斤紙HDFS锛夊苟涓旀瘡鏉℃暟鎹湁鍞竴鐨勬爣璇嗙銆?/p>

rowkey + columnfamily锛堝垪钄燂級 + column01锛堝垪鍚嶏級  + timestamp锛堟椂闂存埑锛?nbsp;锛?nbsp; value 锛堝€硷級  => cell (鍗曞厓)

• Row Key: 琛岄敭锛孴able鐨勪富閿紝Table涓殑璁板綍鎸夌収Row Key鎺掑簭

• Timestamp: 鏃堕棿鎴筹紝姣忔鏁版嵁鎿嶄綔瀵瑰簲鐨勬椂闂存埑锛屽彲浠ョ湅浣滄槸鏁版嵁鐨剉ersion number

• Column Family锛氬垪绨囷紝Table鍦ㄦ按骞虫柟鍚戞湁涓€涓垨鑰呭涓狢olumn Family缁勬垚锛屼竴涓狢olumn Family涓彲浠ョ敱浠绘剰澶氫釜Column缁勬垚锛屽嵆Column Family鏀寔鍔ㄦ€佹墿灞曪紝鏃犻渶棰勫厛瀹氫箟Column鐨勬暟閲忎互鍙婄被鍨嬶紝鎵€鏈塁olumn鍧囦互浜岃繘鍒舵牸寮忓瓨鍌紝鐢ㄦ埛闇€瑕佽嚜琛岃繘琛岀被鍨嬭浆鎹€?/span>

涓轰粈涔圚base鑳借繖涔堝揩鍛紵

鍏抽敭鐐瑰湪浜庤〃涓璻owkey鐨勮璁°€?/p>

1銆乭base鏄彲鍒掑垎鎴愬涓猺egion銆?br>2銆乺owkey鏄帓濂藉簭浜嗙殑銆?br>3銆佹暟鎹槸鎸夊垪瀛樺偍鐨勩€?/p>

• 棣栧厛锛岃兘蹇€熸壘鍒拌鎵€鍦ㄧ殑region(鍒嗗尯)锛屽亣璁捐〃鏈?0浜挎潯璁板綍锛屽崰绌洪棿1TB, 鍒嗗垪鎴愪簡500涓猺egion, 1涓猺egion鍗?涓狦. 鏈€澶氳鍙?G鐨勮褰曪紝灏辫兘鎵惧埌瀵瑰簲璁板綍

• 鍏舵锛屾槸鎸夊垪瀛樺偍鐨勶紝鍏跺疄鏄垪鏃忥紝鍋囪鍒嗕负3涓垪鏃忥紝姣忎釜鍒楁棌灏辨槸666M锛?濡傛灉瑕佹煡璇㈢殑涓滆タ鍦ㄥ叾涓?涓垪鏃忎笂锛?涓垪鏃忓寘鍚?涓垨鑰呭涓狧StoreFile锛屽亣璁句竴涓狧StoreFile鏄?28M锛?璇ュ垪鏃忓寘鍚?涓狧StoreFile鍦ㄧ鐩樹笂. 鍓╀笅鐨勫湪鍐呭瓨涓€?/span>

• 鍐嶆锛屾槸鎺掑ソ搴忎簡鐨勶紝浣犺鐨勮褰曟湁鍙兘鍦ㄦ渶鍓嶉潰锛屼篃鏈夊彲鑳藉湪鏈€鍚庨潰锛屽亣璁惧湪涓棿锛屾垜浠彧闇€閬嶅巻2.5涓狧StoreFile鍏?00M

• 鏈€鍚庯紝姣忎釜HStoreFile(HFile鐨勫皝瑁?锛屾槸浠ラ敭鍊煎锛坘ey-value锛夋柟寮忓瓨鍌紝鍙閬嶅巻涓€涓釜鏁版嵁鍧椾腑鐨刱ey鐨勪綅缃紝骞跺垽鏂鍚堟潯浠跺彲浠ヤ簡銆?涓€鑸琸ey鏄湁闄愮殑闀垮害锛屽亣璁捐窡value鏄?:19锛堝拷鐣File涓婂叾瀹冨潡锛夛紝鏈€缁堝彧闇€瑕?5M灏卞彲鑾峰彇鐨勫搴旂殑璁板綍锛屾寜鐓х鐩樼殑璁块棶100M/S锛屽彧闇€0.15绉掋€?鍔犱笂鍧楃紦瀛樻満鍒讹紙LRU鍘熷垯锛夛紝浼氬彇寰楁洿楂樼殑鏁堢巼銆?/span>

Hbase鐨勬暟鎹ā鍨?/strong>

Hbase鏋舵瀯

鍥句腑鍙HDFS锛孼ookeeper銆侶base鏈夊灏戜釜RS锛屽叾绠＄悊鐨勫唴瀹归兘瀛樺湪ZK鐨刏Node涓婇潰锛孼K鍙互甯姪鐢ㄦ埛鎵惧埌鏌愬紶琛ㄥ湪鍝釜RS锛屽仛涓€涓绱㈠姛鑳姐€傛灦鏋勭殑璇︾粏锛岃寰€鍚庨潰鐪嬨€?/p>

Hbase瀹夎閮ㄧ讲

• 瑙ｅ帇瀹夎锛屼笉蹇呭璇?/span>

• 閰嶇疆hbase-env.sh 閰嶇疆JDK

• 閰嶇疆hbase-site.xml 璁剧疆HDFS涓奾base鏁版嵁瀛樺偍鐩綍锛?/p>

鍚屾椂鍏抽棴榛樿ZK绠＄悊锛岀敱鎴戜滑鍚姩鑷繁瀹夎鐨刏K銆傛敞閲婃帀灏辫銆?/p>

hbase-env.sh

# export HBASE_MANAGES_ZK=true

• 淇敼 regionservers

* zookeeper
* namenode
* datanode
* HM

• 璁块棶Web椤甸潰 绔彛鍙?0010

鍚勪釜鐩綍鐨勫惈涔夛紙瀹樻柟鏂囨。锛夛細

1./.tmp 杩欎釜鐩綍鐢ㄦ潵瀛樺偍涓存椂鏂囦欢锛屽綋瀵硅〃杩涜鎿嶄綔鐨勬椂鍊欙紝棣栧厛浼氬皢琛ㄧЩ鍔ㄥ埌璇ョ洰褰曚笅锛岀劧鍚庡啀杩涜鎿嶄綔銆?nbsp;
2./WALs WAL鎰忎负Write ahead log鐢ㄦ潵鍋氱伨闅炬仮澶嶃€傝HLog瀹炰緥绠＄悊鐨刉AL鏂囦欢銆?nbsp;
3./data hbase 鐨勬牳蹇冪洰褰曪紝绯荤粺浼氶缃袱涓?nbsp;namespace 鍗筹細hbase鍜?span class="mq-117">default銆?data/hbase 瀛樺偍浜嗗瓨鍌ㄤ簡 HBase 鐨?nbsp;namespace銆乵eta 涓や釜绯荤粺绾ц〃銆?span class="mq-119">namespace 涓瓨鍌ㄤ簡 HBase 涓殑鎵€鏈?nbsp;namespace 淇℃伅锛屽寘鎷缃殑hbase 鍜?nbsp;default銆?data/default/ 瀛樺偍鎵€鏈夌敤鎴锋暟鎹〃/data/default/琛ㄥ悕銆?nbsp;
4./hbase.id 瀛樺偍闆嗙兢鍞竴鐨?nbsp;cluster id 鍙凤紝鏄竴涓?nbsp;uuid銆?nbsp;
5./hbase.version 鍚屾牱涔熸槸涓€涓枃浠讹紝瀛樺偍闆嗙兢鐨勭増鏈彿锛岃矊浼兼槸鍔犲瘑鐨勶紝鐪嬩笉鍒帮紝鍙兘閫氳繃web-ui 鎵嶈兘姝ｇ‘鏄剧ず鍑烘潵銆?nbsp;6./oldWALs 褰?WALs 涓殑HLog鏂囦欢琚寔涔呭寲鍒板瓨鍌ㄦ枃浠朵腑锛屼笉鍐嶉渶瑕佹棩蹇楁枃浠舵椂锛屽畠浠細琚Щ鍔ㄥ埌/oldWALs鐩綍銆?br>

娉ㄦ剰 锛氬鏋滄湁闇€瑕佺殑璇濆氨鎹adoop鐨刯ar锛屾浛鎹adoop鐨勭増鏈€?/p>

Hbase Shell浣跨敤

• list 鏌ョ湅琛?/span>

• help 鈥樷€欐煡鐪嬪府鍔╃粏鑺?help 鈥榗reate鈥?/span>

• 鍒涘缓琛?create 鈥榰ser鈥欙紝鈥榠nfo鈥?/span>

• 鎻忚堪琛?describe 鈥榰ser鈥?/span>

• 鎻掓暟鎹?put 鈥榰ser鈥?鈥?001鈥?鈥榠nfo:name鈥?鈥榵ianglei鈥?锛堣〃鍚嶏紝rowkey锛屽垪钄?鍒楀悕锛屽€硷級

• 鏌ョ湅鏁版嵁锛堜笁绉嶆柟寮忥級

• 渚濇嵁rowkey鏌ヨ锛屾渶蹇殑

• 浠巑emstore鍐欏埌storefile flush 鈥榰ser鈥?/span>

• 鎵嬪姩compact compact 鈥榰ser鈥?/span>

鏌ョ湅鏁版嵁

get

鑼冨洿鏌ヨ锛堢敤鐨勬渶澶氾級
scan 鈥榰ser鈥?{STARTROW=>鈥?002鈥檥

鍏ㄨ〃鎵弿
scan

鍒犻櫎鏁版嵁

delete 鈥榰ser鈥?鈥?001鈥?鈥榠nfo:name鈥?br>deleteall 鈥榰ser鈥?鈥?001鈥?/p>

鍒犻櫎鎵€鏈夋暟鎹?/p>

鍒犻櫎琛?/p>

鍒犻櫎鍓嶏紝蹇呴』鍏坉isable

琛ㄧ殑鍚敤绂佺敤
绂佺敤

鍚敤

Hbase 鐗╃悊妯″瀷

璇磋Hbase涓暟鎹瓨鍌ㄧ殑鐗╃悊妯″瀷銆傛暟鎹槸鎬庝箞鍦℉base涓瓨鍌ㄧ殑銆傜畝鍗曟潵璇存槸鎸塺owkey鏉ヨ繘琛屽姩鎬佸垎鍖猴紙region锛夊瓨鍌ㄧ殑銆?/p>

缁嗚妭璇存槑锛?/p>

姣忎釜columnfamily瀛樺偍鍦℉DFS涓婄殑涓€涓崟鐙枃浠朵腑锛汯ey 鍜?Version number鍦ㄦ瘡涓?column family涓潎鐢变竴浠斤紱绌哄€间笉浼氳淇濆瓨銆?/p>

HBase鏁版嵁鍐欏叆娴佺▼

put 鈥?gt; cell - wal(棰勫啓鏃ュ織) -> hdfs- memstore锛堟孩鍐欙級- storefile -> hdfs

缁嗚妭鎻忚堪

• Client閫氳繃Zookeeper鐨勮皟搴︼紝鍚慠egionServer鍙戝嚭鍐欐暟鎹姹傦紝鍦≧egion涓啓鏁版嵁锛?/span>

• 鏁版嵁琚啓鍏egion鐨凪emStore锛岀煡閬揗emStore杈惧埌棰勮闃€鍊?鍗矼emStore婊?锛?/span>

• MemStore涓殑鏁版嵁琚獸lush鎴愪竴涓猄toreFile锛?/span>

• 闅忕潃StoreFile鏂囦欢鐨勪笉鏂澶氾紝褰撳叾鏁伴噺澧為暱鍒颁竴瀹氶榾鍊煎悗锛岃Е鍙慍ompact鍚堝苟鎿嶄綔锛屽皢澶氫釜StoreFile鍚堝苟鎴愪竴涓猄toreFile锛屽悓鏃惰繘琛岀増鏈悎骞跺拰鏁版嵁鍒犻櫎锛?/span>

• StoreFiles閫氳繃涓嶆柇鐨凜ompact鍚堝苟鎿嶄綔锛岄€愭褰㈡垚瓒婃潵瓒婂ぇ鐨凷toreFile锛?/span>

• 鍗曚釜StoreFile澶у皬瓒呰繃涓€瀹氶榾鍊煎悗锛岃Е鍙慡plit鎿嶄綔锛屾妸褰撳墠Region Split鎴?涓柊鐨凴egion銆傜埗Region浼氫笅绾匡紝鏂癝plit鍑虹殑2涓瓙Region浼氳HMaster鍒嗛厤鍒扮浉搴旂殑RegionServer涓婏紝浣垮緱鍘熷厛1涓猂egion鐨勫帇鍔涘緱浠ュ垎娴佸埌2涓猂egion涓娿€?/span>

鍙互鐪嬪嚭HBase鍙湁澧炴坊鏁版嵁锛屾墍鏈夌殑鏇存柊鍜屽垹闄ゆ搷浣滈兘鏄湪鍚庣画鐨凜ompact鍘嗙▼涓妇琛岀殑锛屼娇寰楃敤鎴风殑鍐欐搷浣滃彧瑕佽繘鍏ュ唴瀛樺氨鍙互绔嬪埢杩斿洖锛屽疄鐜颁簡HBase I/O鐨勯珮鎬ц兘銆?/p>

HBase鏁版嵁璇诲彇娴佺▼

鏁版嵁璇诲彇:鍏堝埌memstore璇伙紙娌℃湁婧㈠啓鍒帮級鍐嶅埌storefile鏈€鍚庡悎骞躲€?/p>

缁嗚妭鎻忚堪

• Client璁块棶Zookeeper锛岃幏鍙?META.琛ㄤ俊鎭紱

• 浠?META.琛ㄦ煡鎵撅紝鑾峰彇瀛樻斁鐩爣鏁版嵁鐨凴egion淇℃伅锛屼粠鑰屾壘鍒板搴旂殑RegionServer锛?/span>

• 閫氳繃RegionServer鑾峰彇闇€瑕佹煡鎵剧殑鏁版嵁锛?/span>

• RegionServer鐨勫唴瀛樺垎涓篗emStore鍜孊lockCache涓ら儴鍒嗭紝MemStore涓昏鐢ㄤ簬鍐欐暟鎹紝BlockCache涓昏鐢ㄤ簬璇绘暟鎹€傝璇锋眰鍏堝埌MemStore涓煡鏁版嵁锛屾煡涓嶅埌灏卞埌BlockCache涓煡锛屽啀鏌ヤ笉鍒板氨浼氬埌StoreFile涓婅锛屽苟鎶婅鐨勭粨鏋滄斁鍏lockCache銆?/span>

瀵诲潃杩囩▼锛歝lient鈥?gt;Zookeeper鈥?gt;.META. 琛ㄢ€?gt;RegionServer鈥?gt;Region鈥?gt;User Table鈥?gt;client

Hbase Region鐞嗚В

Hbase 鏋舵瀯缁嗚妭

杩欏紶鍥炬湁涓€涓己闄峰氨鏄紝Hbase涓竴涓狧log鏄鐞嗙殑涓€涓狧RegionServer鐨勶紝涓嶆槸涓€涓狧Region銆?/p>

Hbase璇诲彇鏁版嵁娴佺▼璇存槑锛?/strong>

瀹㈡埛绔鍙栨暟鎹鍏堣繛鎺K锛屼粬闇€瑕佸幓ZK鎵惧埌杩欎釜琛紙region锛夋墍鍦ㄧ殑浣嶇疆銆?br>杩欓噷Zk瀛樺偍浜嗕笅鍥惧唴瀹癸細

鐒跺悗鍘籱eta琛ㄦ壂鎻忎俊鎭紝姣斿鎵惧埌user鐨剅owkey鐒跺悗鎵惧埌浠栧搴旂殑region锛屽緱鍒扮鐞嗕粬鐨剅egionserver銆?/p>

Master闇€瑕佽繛鎺K鏄洜涓轰粬闇€瑕佺煡閬撳摢浜汻S鏄椿鐫€鐨勩€?/p>

鐗规畩璇存槑锛?/strong>

• 鏈夊嚑涓垪钄熷氨鏈夊嚑涓猄tore锛孲tore閲岄潰鏈変竴涓猰emstore鍜屽緢澶氫釜storeFile銆?/p>

• Hlog瀛樺湪鐨勬剰涔夋槸锛屽綋闆嗙兢绐佺劧鎸傛帀memstore鏁版嵁涓㈠け鍚庯紝褰撻噸鍚満鍣ㄦ椂Hlog閲嶆柊璇诲彇鏂囦欢绯荤粺鐨勬暟鎹紝骞朵笖鎭㈠鍥炴潵銆?/p>

RS->涓€涓猂S瀵瑰簲涓€涓洰褰?/p>

meta-region-server
瀛樺偍meta琛ㄥ搴旂殑region琚摢涓猺egionserver绠＄悊鐨勪俊鎭?/p>

HBase鏂扮増鏈腑锛屾湁浜嗙被浼间簬RDBMS涓璂ataBase鐨勬蹇?/p>

鍛戒护绌洪棿

- 鐢ㄦ埛鑷畾涔夌殑琛紝榛樿鎯呭喌涓嬪懡鍚嶇┖闂?br>

default

- 绯荤粺鑷甫鐨勫厓鏁版嵁琛ㄧ殑鍛藉悕绌洪棿

hbase

鎬荤粨

Hbase杩欎竴鍧楅渶瑕佸涔犵殑鍐呭寰堝寰堝銆傝繖涓€鑺備富瑕佸氨鏄璇碒base鐨勪竴浜涘熀纭€鐗规€э紝shell锛屼互鍙奌base鐨勬灦鏋勫垎鏋愩€傚唴瀹硅櫧鐒剁湅鐫€姣旇緝澶氾紝浣嗘槸鑲畾涔熶笉鍏紝鏆傛椂灏辫褰曡繖涔堝锛屽鏈夐渶瑕佸啀鍘荤湅瀹樻柟鏂囨。馃馃馃銆傛繁鍏ュ涔犲繀涓嶅彲灏戙€?

闄涗笅...鐪嬪畬濂忔姌锛岀偣涓禐鍐嶈蛋鍚э紒

鎺ㄨ崘闃呰

鎶€鏈細

鎶€鏈細

鎶€鏈細

鎶€鏈細

鎶€鏈細

宸ュ叿锛?/p>

宸ュ叿锛?/p>