閲嶆俯澶ф暟鎹?--Hbase閮ㄧ讲浠ュ強鏋舵瀯鍒嗘瀽
Posted 浜掓壇绋嬪簭
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了閲嶆俯澶ф暟鎹?--Hbase閮ㄧ讲浠ュ強鏋舵瀯鍒嗘瀽相关的知识,希望对你有一定的参考价值。
杩欑瘒鏂囩珷涓昏浠嬬粛涓€涓嬶紝Hbase鏄粈涔堬紝鑳藉共浠€涔堬紝骞茬殑鎬庝箞鏍枫€備互鍙奌base鐨勫畨瑁呴厤缃拰璋冭瘯绛夈€傛€荤殑鏉ヨHbase鍦ㄥぇ鏁版嵁鐨勭敓鎬佺郴缁熶腑鍗犳湁寰堥噸瑕佺殑浣嶇疆锛屽浜嶩base鐨勫涔犺繕闇€瑕佹洿鍔犳繁鍏ユ墠琛屻€傚浜庡ぇ鏁版嵁鐨勫唴瀹规垜閮藉缓璁鐪嬬湅瀹樻柟鏂囨。锛屼互鍓嶆垜鎬昏寰楄嚜宸辩湅涓嶆噦鍝簺楂樺ぇ涓婄殑锛屽叾瀹炰笉鐒躲€傪煒戰煒岎煒屾劅瑙夊畼鏂规枃妗e啓鐨勯兘寰堜翰姘戙€?/p>
Hbase
HBase鏄竴涓垎甯冨紡鐨勩€侀潰鍚戝垪鐨勫紑婧愭暟鎹簱锛岃鎶€鏈潵婧愪簬 Fay Chang 鎵€鎾板啓鐨凣oogle璁烘枃鈥淏igtable锛氫竴涓粨鏋勫寲鏁版嵁鐨勫垎甯冨紡瀛樺偍绯荤粺鈥濄€傚氨鍍廈igtable鍒╃敤浜咷oogle鏂囦欢绯荤粺锛團ile System锛夋墍鎻愪緵鐨勫垎甯冨紡鏁版嵁瀛樺偍涓€鏍凤紝HBase鍦℉adoop涔嬩笂鎻愪緵浜嗙被浼间簬Bigtable鐨勮兘鍔涖€侶Base鏄疉pache鐨凥adoop椤圭洰鐨勫瓙椤圭洰銆侶Base涓嶅悓浜庝竴鑸殑鍏崇郴鏁版嵁搴擄紝瀹冩槸涓€涓€傚悎浜庨潪缁撴瀯鍖栨暟鎹瓨鍌ㄧ殑鏁版嵁搴撱€傚彟涓€涓笉鍚岀殑鏄疕Base鍩轰簬鍒楃殑鑰屼笉鏄熀浜庤鐨勬ā寮忋€?- 鐧惧害鐧剧
绠€鍗曠殑璇碒Base灏辨槸涓€涓垎甯冨紡鐨勫彲鎵╁睍鐨勫ぇ鏁版嵁閲忕殑闈炲叧绯诲瀷鏁版嵁搴擄紙NoSQL锛夈€傚畠鍏锋湁涓€鑸殑鍏崇郴鍨嬫暟鎹?Oracle/mysql鐨勫熀纭€鍔熻兘濡傦細
瀛樺偍鏁版嵁
妫€绱㈡暟鎹?/span>
鏌ヨ鏁版嵁
Hbase鑳藉仛鐨勪粈涔?/strong>
Hbase 浣滀负涓€涓潪鍏崇郴鍨嬫暟鎹簱锛岄鍏堝氨鏄鏁版嵁鐨勫熀纭€鐨勫瓨鍌紝鍏舵鏄彲浠ユ彁渚涗紶缁熷叧绯诲瀷鏁版嵁搴撹揪涓嶅埌鐨勪嚎绾ф暟鎹噺瀹炴椂鐨勬煡璇€?/p>
娴烽噺鏁版嵁鐨勫瓨鍌?/span>
娴烽噺鏁版嵁鐨勬煡璇?/span>
Hbase鍦ㄤ笂浜跨殑鏁版嵁琛ㄩ噷鑳藉仛鍒扮绾у埆鏌ヨ锛堝疄鏃舵煡璇級
Hbase浣跨敤鍦烘櫙
浜ら€氭暟鎹?/span>
璐﹀崟鏁版嵁
娓告垙鏁版嵁
鐢靛晢浜ゆ槗鏁版嵁
涓€鍒囧ぇ鏁版嵁閲忕殑锛屽瀹炴椂鏌ヨ渚濊禆楂樼殑鐨勫満鏅兘鑳借В鍐炽€?/p>
Hbase鍦ㄧ敓鎬佸湀涓殑浣嶇疆
鍥句腑鍙锛孒base鏄熀浜嶩DFS鐨勶紝涔熷氨鏄Hbase鐨勬暟鎹槸瀛樺偍鍦℉DFS涓婇潰鐨勶紝瀹冨彲浠ュ緢濂藉湴缁ф壙MapReduce鍜孒ive鏉ヨ繘琛屾暟鎹殑澶勭悊銆?/p>
Hbase鏁版嵁瀛樺偍鏂瑰紡瀛樺偍
Hbase鏄潪鍏崇郴鍨嬫暟鎹簱涓吀鍨嬬殑鍒楀紡瀛樺偍鏁版嵁搴?/p>
Hbase鏁版嵁鐨勭粍鎴愬厓绱?/strong>
Hbase涓暟鎹槸浠ュ瓧鑺傛暟缁勮繘琛屽瓨鍌紝涓嶅瓨鍦ㄦ暟鎹被鍨嬨€傞粯璁ゆ暟鎹笁浠斤紙HDFS锛夊苟涓旀瘡鏉℃暟鎹湁鍞竴鐨勬爣璇嗙銆?/p>
rowkey + columnfamily锛堝垪钄燂級 + column01锛堝垪鍚嶏級 + timestamp锛堟椂闂存埑锛?nbsp;锛?nbsp; value 锛堝€硷級 => cell (鍗曞厓)
Row Key: 琛岄敭锛孴able鐨勪富閿紝Table涓殑璁板綍鎸夌収Row Key鎺掑簭
Timestamp: 鏃堕棿鎴筹紝姣忔鏁版嵁鎿嶄綔瀵瑰簲鐨勬椂闂存埑锛屽彲浠ョ湅浣滄槸鏁版嵁鐨剉ersion number
Column Family锛氬垪绨囷紝Table鍦ㄦ按骞虫柟鍚戞湁涓€涓垨鑰呭涓狢olumn Family缁勬垚锛屼竴涓狢olumn Family涓彲浠ョ敱浠绘剰澶氫釜Column缁勬垚锛屽嵆Column Family鏀寔鍔ㄦ€佹墿灞曪紝鏃犻渶棰勫厛瀹氫箟Column鐨勬暟閲忎互鍙婄被鍨嬶紝鎵€鏈塁olumn鍧囦互浜岃繘鍒舵牸寮忓瓨鍌紝鐢ㄦ埛闇€瑕佽嚜琛岃繘琛岀被鍨嬭浆鎹€?/span>
涓轰粈涔圚base鑳借繖涔堝揩鍛紵
鍏抽敭鐐瑰湪浜庤〃涓璻owkey鐨勮璁°€?/p>
1銆乭base鏄彲鍒掑垎鎴愬涓猺egion銆?br>2銆乺owkey鏄帓濂藉簭浜嗙殑銆?br>3銆佹暟鎹槸鎸夊垪瀛樺偍鐨勩€?/p>
棣栧厛锛岃兘蹇€熸壘鍒拌鎵€鍦ㄧ殑region(鍒嗗尯)锛屽亣璁捐〃鏈?0浜挎潯璁板綍锛屽崰绌洪棿1TB, 鍒嗗垪鎴愪簡500涓猺egion, 1涓猺egion鍗?涓狦. 鏈€澶氳鍙?G鐨勮褰曪紝灏辫兘鎵惧埌瀵瑰簲璁板綍
鍏舵锛屾槸鎸夊垪瀛樺偍鐨勶紝鍏跺疄鏄垪鏃忥紝鍋囪鍒嗕负3涓垪鏃忥紝姣忎釜鍒楁棌灏辨槸666M锛?濡傛灉瑕佹煡璇㈢殑涓滆タ鍦ㄥ叾涓?涓垪鏃忎笂锛?涓垪鏃忓寘鍚?涓垨鑰呭涓狧StoreFile锛屽亣璁句竴涓狧StoreFile鏄?28M锛?璇ュ垪鏃忓寘鍚?涓狧StoreFile鍦ㄧ鐩樹笂. 鍓╀笅鐨勫湪鍐呭瓨涓€?/span>
鍐嶆锛屾槸鎺掑ソ搴忎簡鐨勶紝浣犺鐨勮褰曟湁鍙兘鍦ㄦ渶鍓嶉潰锛屼篃鏈夊彲鑳藉湪鏈€鍚庨潰锛屽亣璁惧湪涓棿锛屾垜浠彧闇€閬嶅巻2.5涓狧StoreFile鍏?00M
鏈€鍚庯紝姣忎釜HStoreFile(HFile鐨勫皝瑁?锛屾槸浠ラ敭鍊煎锛坘ey-value锛夋柟寮忓瓨鍌紝鍙閬嶅巻涓€涓釜鏁版嵁鍧椾腑鐨刱ey鐨勪綅缃紝骞跺垽鏂鍚堟潯浠跺彲浠ヤ簡銆?涓€鑸琸ey鏄湁闄愮殑闀垮害锛屽亣璁捐窡value鏄?:19锛堝拷鐣File涓婂叾瀹冨潡锛夛紝鏈€缁堝彧闇€瑕?5M灏卞彲鑾峰彇鐨勫搴旂殑璁板綍锛屾寜鐓х鐩樼殑璁块棶100M/S锛屽彧闇€0.15绉掋€?鍔犱笂鍧楃紦瀛樻満鍒讹紙LRU鍘熷垯锛夛紝浼氬彇寰楁洿楂樼殑鏁堢巼銆?/span>
Hbase鐨勬暟鎹ā鍨?/strong>
Hbase鏋舵瀯
鍥句腑鍙HDFS锛孼ookeeper銆侶base鏈夊灏戜釜RS锛屽叾绠$悊鐨勫唴瀹归兘瀛樺湪ZK鐨刏Node涓婇潰锛孼K鍙互甯姪鐢ㄦ埛鎵惧埌鏌愬紶琛ㄥ湪鍝釜RS锛屽仛涓€涓绱㈠姛鑳姐€傛灦鏋勭殑璇︾粏锛岃寰€鍚庨潰鐪嬨€?/p>
Hbase瀹夎閮ㄧ讲
瑙e帇瀹夎锛屼笉蹇呭璇?/span>
閰嶇疆hbase-env.sh 閰嶇疆JDK
閰嶇疆hbase-site.xml 璁剧疆HDFS涓奾base鏁版嵁瀛樺偍鐩綍锛?/p>
鍚屾椂鍏抽棴榛樿ZK绠$悊锛岀敱鎴戜滑鍚姩鑷繁瀹夎鐨刏K銆傛敞閲婃帀灏辫銆?/p>
hbase-env.sh
# export HBASE_MANAGES_ZK=true
淇敼 regionservers
* zookeeper
* namenode
* datanode
* HM
璁块棶Web椤甸潰 绔彛鍙?0010
鍚勪釜鐩綍鐨勫惈涔夛紙瀹樻柟鏂囨。锛夛細
1./.tmp 杩欎釜鐩綍鐢ㄦ潵瀛樺偍涓存椂鏂囦欢锛屽綋瀵硅〃杩涜鎿嶄綔鐨勬椂鍊欙紝棣栧厛浼氬皢琛ㄧЩ鍔ㄥ埌璇ョ洰褰曚笅锛岀劧鍚庡啀杩涜鎿嶄綔銆?nbsp;
2./WALs WAL鎰忎负Write ahead log鐢ㄦ潵鍋氱伨闅炬仮澶嶃€傝HLog瀹炰緥绠$悊鐨刉AL鏂囦欢銆?nbsp;
3./data hbase 鐨勬牳蹇冪洰褰曪紝绯荤粺浼氶缃袱涓?nbsp;namespace 鍗筹細hbase鍜?span class="mq-117">default銆?data/hbase 瀛樺偍浜嗗瓨鍌ㄤ簡 HBase 鐨?nbsp;namespace銆乵eta 涓や釜绯荤粺绾ц〃銆?span class="mq-119">namespace 涓瓨鍌ㄤ簡 HBase 涓殑鎵€鏈?nbsp;namespace 淇℃伅锛屽寘鎷缃殑hbase 鍜?nbsp;default銆?data/default/ 瀛樺偍鎵€鏈夌敤鎴锋暟鎹〃/data/default/琛ㄥ悕銆?nbsp;
4./hbase.id 瀛樺偍闆嗙兢鍞竴鐨?nbsp;cluster id 鍙凤紝鏄竴涓?nbsp;uuid銆?nbsp;
5./hbase.version 鍚屾牱涔熸槸涓€涓枃浠讹紝瀛樺偍闆嗙兢鐨勭増鏈彿锛岃矊浼兼槸鍔犲瘑鐨勶紝鐪嬩笉鍒帮紝鍙兘閫氳繃web-ui 鎵嶈兘姝g‘鏄剧ず鍑烘潵銆?nbsp;6./oldWALs 褰?WALs 涓殑HLog鏂囦欢琚寔涔呭寲鍒板瓨鍌ㄦ枃浠朵腑锛屼笉鍐嶉渶瑕佹棩蹇楁枃浠舵椂锛屽畠浠細琚Щ鍔ㄥ埌/oldWALs鐩綍銆?br>
娉ㄦ剰 锛氬鏋滄湁闇€瑕佺殑璇濆氨鎹adoop鐨刯ar锛屾浛鎹adoop鐨勭増鏈€?/p>
Hbase Shell浣跨敤
list 鏌ョ湅琛?/span>
help 鈥樷€欐煡鐪嬪府鍔╃粏鑺?help 鈥榗reate鈥?/span>
鍒涘缓琛?create 鈥榰ser鈥欙紝鈥榠nfo鈥?/span>
鎻忚堪琛?describe 鈥榰ser鈥?/span>
鎻掓暟鎹?put 鈥榰ser鈥?鈥?001鈥?鈥榠nfo:name鈥?鈥榵ianglei鈥?锛堣〃鍚嶏紝rowkey锛屽垪钄?鍒楀悕锛屽€硷級
鏌ョ湅鏁版嵁锛堜笁绉嶆柟寮忥級
渚濇嵁rowkey鏌ヨ锛屾渶蹇殑
浠巑emstore鍐欏埌storefile flush 鈥榰ser鈥?/span>
鎵嬪姩compact compact 鈥榰ser鈥?/span>
鏌ョ湅鏁版嵁
get
鑼冨洿鏌ヨ锛堢敤鐨勬渶澶氾級
scan 鈥榰ser鈥?{STARTROW=>鈥?002鈥檥
鍏ㄨ〃鎵弿
scan
鍒犻櫎鏁版嵁
delete 鈥榰ser鈥?鈥?001鈥?鈥榠nfo:name鈥?br>deleteall 鈥榰ser鈥?鈥?001鈥?/p>
鍒犻櫎鎵€鏈夋暟鎹?/p>
鍒犻櫎琛?/p>
鍒犻櫎鍓嶏紝蹇呴』鍏坉isable
琛ㄧ殑鍚敤绂佺敤
绂佺敤
鍚敤
Hbase 鐗╃悊妯″瀷
璇磋Hbase涓暟鎹瓨鍌ㄧ殑鐗╃悊妯″瀷銆傛暟鎹槸鎬庝箞鍦℉base涓瓨鍌ㄧ殑銆傜畝鍗曟潵璇存槸鎸塺owkey鏉ヨ繘琛屽姩鎬佸垎鍖猴紙region锛夊瓨鍌ㄧ殑銆?/p>
缁嗚妭璇存槑锛?/p>
姣忎釜columnfamily瀛樺偍鍦℉DFS涓婄殑涓€涓崟鐙枃浠朵腑锛汯ey 鍜?Version number鍦ㄦ瘡涓?column family涓潎鐢变竴浠斤紱绌哄€间笉浼氳淇濆瓨銆?/p>
HBase鏁版嵁鍐欏叆娴佺▼
put 鈥?gt; cell - wal(棰勫啓鏃ュ織) -> hdfs- memstore锛堟孩鍐欙級- storefile -> hdfs
缁嗚妭鎻忚堪
Client閫氳繃Zookeeper鐨勮皟搴︼紝鍚慠egionServer鍙戝嚭鍐欐暟鎹姹傦紝鍦≧egion涓啓鏁版嵁锛?/span>
鏁版嵁琚啓鍏egion鐨凪emStore锛岀煡閬揗emStore杈惧埌棰勮闃€鍊?鍗矼emStore婊?锛?/span>
MemStore涓殑鏁版嵁琚獸lush鎴愪竴涓猄toreFile锛?/span>
闅忕潃StoreFile鏂囦欢鐨勪笉鏂澶氾紝褰撳叾鏁伴噺澧為暱鍒颁竴瀹氶榾鍊煎悗锛岃Е鍙慍ompact鍚堝苟鎿嶄綔锛屽皢澶氫釜StoreFile鍚堝苟鎴愪竴涓猄toreFile锛屽悓鏃惰繘琛岀増鏈悎骞跺拰鏁版嵁鍒犻櫎锛?/span>
StoreFiles閫氳繃涓嶆柇鐨凜ompact鍚堝苟鎿嶄綔锛岄€愭褰㈡垚瓒婃潵瓒婂ぇ鐨凷toreFile锛?/span>
鍗曚釜StoreFile澶у皬瓒呰繃涓€瀹氶榾鍊煎悗锛岃Е鍙慡plit鎿嶄綔锛屾妸褰撳墠Region Split鎴?涓柊鐨凴egion銆傜埗Region浼氫笅绾匡紝鏂癝plit鍑虹殑2涓瓙Region浼氳HMaster鍒嗛厤鍒扮浉搴旂殑RegionServer涓婏紝浣垮緱鍘熷厛1涓猂egion鐨勫帇鍔涘緱浠ュ垎娴佸埌2涓猂egion涓娿€?/span>
鍙互鐪嬪嚭HBase鍙湁澧炴坊鏁版嵁锛屾墍鏈夌殑鏇存柊鍜屽垹闄ゆ搷浣滈兘鏄湪鍚庣画鐨凜ompact鍘嗙▼涓妇琛岀殑锛屼娇寰楃敤鎴风殑鍐欐搷浣滃彧瑕佽繘鍏ュ唴瀛樺氨鍙互绔嬪埢杩斿洖锛屽疄鐜颁簡HBase I/O鐨勯珮鎬ц兘銆?/p>
HBase鏁版嵁璇诲彇娴佺▼
鏁版嵁璇诲彇:鍏堝埌memstore璇伙紙娌℃湁婧㈠啓鍒帮級鍐嶅埌storefile鏈€鍚庡悎骞躲€?/p>
缁嗚妭鎻忚堪
Client璁块棶Zookeeper锛岃幏鍙?META.琛ㄤ俊鎭紱
浠?META.琛ㄦ煡鎵撅紝鑾峰彇瀛樻斁鐩爣鏁版嵁鐨凴egion淇℃伅锛屼粠鑰屾壘鍒板搴旂殑RegionServer锛?/span>
閫氳繃RegionServer鑾峰彇闇€瑕佹煡鎵剧殑鏁版嵁锛?/span>
RegionServer鐨勫唴瀛樺垎涓篗emStore鍜孊lockCache涓ら儴鍒嗭紝MemStore涓昏鐢ㄤ簬鍐欐暟鎹紝BlockCache涓昏鐢ㄤ簬璇绘暟鎹€傝璇锋眰鍏堝埌MemStore涓煡鏁版嵁锛屾煡涓嶅埌灏卞埌BlockCache涓煡锛屽啀鏌ヤ笉鍒板氨浼氬埌StoreFile涓婅锛屽苟鎶婅鐨勭粨鏋滄斁鍏lockCache銆?/span>
瀵诲潃杩囩▼锛歝lient鈥?gt;Zookeeper鈥?gt;.META. 琛ㄢ€?gt;RegionServer鈥?gt;Region鈥?gt;User Table鈥?gt;client
Hbase Region鐞嗚В
Hbase 鏋舵瀯缁嗚妭
杩欏紶鍥炬湁涓€涓己闄峰氨鏄紝Hbase涓竴涓狧log鏄鐞嗙殑涓€涓狧RegionServer鐨勶紝涓嶆槸涓€涓狧Region銆?/p>
Hbase璇诲彇鏁版嵁娴佺▼璇存槑锛?/strong>
瀹㈡埛绔鍙栨暟鎹鍏堣繛鎺K锛屼粬闇€瑕佸幓ZK鎵惧埌杩欎釜琛紙region锛夋墍鍦ㄧ殑浣嶇疆銆?br>杩欓噷Zk瀛樺偍浜嗕笅鍥惧唴瀹癸細
鐒跺悗鍘籱eta琛ㄦ壂鎻忎俊鎭紝姣斿鎵惧埌user鐨剅owkey鐒跺悗鎵惧埌浠栧搴旂殑region锛屽緱鍒扮鐞嗕粬鐨剅egionserver銆?/p>
Master闇€瑕佽繛鎺K鏄洜涓轰粬闇€瑕佺煡閬撳摢浜汻S鏄椿鐫€鐨勩€?/p>
鐗规畩璇存槑锛?/strong>
鏈夊嚑涓垪钄熷氨鏈夊嚑涓猄tore锛孲tore閲岄潰鏈変竴涓猰emstore鍜屽緢澶氫釜storeFile銆?/p>
Hlog瀛樺湪鐨勬剰涔夋槸锛屽綋闆嗙兢绐佺劧鎸傛帀memstore鏁版嵁涓㈠け鍚庯紝褰撻噸鍚満鍣ㄦ椂Hlog閲嶆柊璇诲彇鏂囦欢绯荤粺鐨勬暟鎹紝骞朵笖鎭㈠鍥炴潵銆?/p>
RS->涓€涓猂S瀵瑰簲涓€涓洰褰?/p>
meta-region-server
瀛樺偍meta琛ㄥ搴旂殑region琚摢涓猺egionserver绠$悊鐨勪俊鎭?/p>
HBase鏂扮増鏈腑锛屾湁浜嗙被浼间簬RDBMS涓璂ataBase鐨勬蹇?/p>
鍛戒护绌洪棿
- 鐢ㄦ埛鑷畾涔夌殑琛紝榛樿鎯呭喌涓嬪懡鍚嶇┖闂?br>
default
- 绯荤粺鑷甫鐨勫厓鏁版嵁琛ㄧ殑鍛藉悕绌洪棿
hbase
鎬荤粨
Hbase杩欎竴鍧楅渶瑕佸涔犵殑鍐呭寰堝寰堝銆傝繖涓€鑺備富瑕佸氨鏄璇碒base鐨勪竴浜涘熀纭€鐗规€э紝shell锛屼互鍙奌base鐨勬灦鏋勫垎鏋愩€傚唴瀹硅櫧鐒剁湅鐫€姣旇緝澶氾紝浣嗘槸鑲畾涔熶笉鍏紝鏆傛椂灏辫褰曡繖涔堝锛屽鏈夐渶瑕佸啀鍘荤湅瀹樻柟鏂囨。馃馃馃銆傛繁鍏ュ涔犲繀涓嶅彲灏戙€?
闄涗笅...鐪嬪畬濂忔姌锛岀偣涓禐鍐嶈蛋鍚э紒
鎺ㄨ崘闃呰
鎶€鏈細
鎶€鏈細
鎶€鏈細
鎶€鏈細
鎶€鏈細
宸ュ叿锛?/p>
宸ュ叿锛?/p>
以上是关于閲嶆俯澶ф暟鎹?--Hbase閮ㄧ讲浠ュ強鏋舵瀯鍒嗘瀽的主要内容,如果未能解决你的问题,请参考以下文章
Hyperf 鍙戝竷 杞婚噺绾ф湁鍚戞棤鐜浘浠诲姟缂栨帓搴?鍜?閫嗘尝鍏拌〃绀烘硶 瀛靛寲缁勪欢浠ュ強 v2.1.4 鐗堟湰