Posted 鏋侀摼绉戞妧
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了相关的知识，希望对你有一定的参考价值。
鏍忕洰馃摪锛?/span>鐮佸啘鎵嬭 馃捇
鎾版枃鉁嶐煆伙細鏋侀摼鍚庣Team 妗斿瓙
缂栬緫馃摎锛?/span>鏋滄灉
鍏抽敭璇?/span>馃搶锛?/span> 娑堟伅闃熷垪 銆?/span> Kafka鐨勭壒鐐?nbsp;銆?/span> Kafka浣撶郴鏋舵瀯 銆?/span> Topic&Partition 銆?/span> Kafka鐨勫鍒舵満鍒?nbsp;銆?/span> Kafka鐨勫悓姝ユ満鍒?nbsp;銆?/span> 娑堟伅浼犺緭淇濋殰 銆?/span> Kafka浣滀负娑堟伅闃熷垪 銆?/span> Kafka浣滀负鍌ㄥ瓨绯荤粺 銆?/span> Kafka娴佸鐞?nbsp;
閫氬父鍦ㄩ」鐩腑锛屾垜浠細鍥犱负濡備笅闇€姹傝€?span>寮曞叆娑堟伅闃熷垪妯″潡锛?/span>
1 瑙ｈ€?nbsp;
2 缂撳啿
娑堟伅绯荤粺浣滀负涓€涓紦鍐叉睜锛屽簲瀵瑰父瑙佺殑璁块棶閲忎笉鍧囪 鎯呭舰銆?/span>姣斿鐗规畩鑺傚亣鏃ョ殑娴侀噺鍓у鍜屾瘡鏃ヤ笉鍚屾椂娈电殑璁块棶閲忓樊寮傘€備互鍙婂鐞嗕笉鍚屾暟鎹被鍨嬫墍闇€鐨勪笉鍚屽疄鏃舵€с€備娇鏁翠釜涓氬姟澶勭悊鏋舵瀯浠ヨ緝浣庢垚鏈幏寰椾竴瀹氱伒娲绘€с€?/span>
3 寮傛
寰堝鏃跺€欙紝鐢ㄦ埛涓嶆兂涔熶笉闇€瑕佺珛鍗冲鐞嗘秷鎭€?span>娑堟伅闃熷垪鎻愪緵浜嗗紓姝ュ鐞嗘満鍒?/strong>锛?span>鍏佽鐢ㄦ埛鎶婁竴涓秷鎭斁鍏ラ槦鍒楋紝浣嗗苟涓嶇珛鍗冲鐞嗗畠銆傛兂鍚戦槦鍒椾腑鏀惧叆澶氬皯娑堟伅灏辨斁澶氬皯锛岀劧鍚庡湪闇€瑕佺殑鏃跺€欏啀鍘诲鐞嗗畠浠€?/span>
浣滀负涓€绉嶅垎甯冨紡鐨勶紝鍩轰簬鍙戝竷/璁㈤槄鐨勬秷鎭郴缁?/strong>銆侹afka鐨勪富瑕佽璁＄洰鏍囧涓嬶細
5 鏀寔鍦ㄧ嚎姘村钩鎵╁睍銆?/span>
濡備笂鍥炬墍绀猴紝涓€涓?span>鍏稿瀷鐨凨afka浣撶郴鏋舵瀯鍖呮嫭鑻ュ共Producer锛堝彲浠ユ槸鏈嶅姟鍣ㄦ棩蹇楋紝涓氬姟鏁版嵁锛岄〉闈㈠墠绔骇鐢熺殑page view绛夌瓑锛夛紝鑻ュ共broker锛圞afka鏀寔姘村钩鎵╁睍锛屼竴鑸琤roker鏁伴噺瓒婂锛岄泦缇ゅ悶鍚愮巼瓒婇珮锛夛紝鑻ュ共Consumer (Group)锛屼互鍙婁竴涓猌ookeeper闆嗙兢銆?/span>
Kafka閫氳繃Zookeeper绠＄悊闆嗙兢閰嶇疆锛岄€変妇leader锛屼互鍙婂湪consumer group鍙戠敓鍙樺寲鏃惰繘琛宺ebalance銆侾roducer浣跨敤push妯″紡灏嗘秷鎭彂甯冨埌broker锛孋onsumer浣跨敤pull妯″紡浠巄roker璁㈤槄骞舵秷璐规秷鎭€?/span>
涓€涓猼opic鍙互璁や负涓€涓竴绫绘秷鎭紝姣忎釜topic灏嗚鍒嗘垚澶氫釜partition锛屾瘡涓猵artition鍦ㄥ瓨鍌ㄥ眰闈㈡槸append log鏂囦欢銆備换浣曞彂甯冨埌姝artition鐨勬秷鎭兘浼氳杩藉姞鍒發og鏂囦欢鐨勫熬閮紝姣忔潯娑堟伅鍦ㄦ枃浠朵腑鐨勪綅缃О涓簅ffset(鍋忕Щ閲?锛宱ffset涓轰竴涓猯ong鍨嬬殑鏁板瓧锛屽畠鍞竴鏍囪涓€鏉℃秷鎭€?span>姣忔潯娑堟伅閮借append鍒皃artition涓紝椤哄簭鍐欑鐩樺洜姝ゆ晥鐜囬潪甯搁珮銆傝繖鏄疜afka楂樺悶鍚愮巼鐨勯噸瑕佸熀纭€銆?/span>
Producer鍙戦€佹秷鎭埌broker鏃讹紝浼氭牴鎹甈aritition鏈哄埗閫夋嫨灏嗗叾瀛樺偍鍒板摢涓€涓狿artition銆傚鏋淧artition鏈哄埗璁剧疆鍚堢悊锛屾墍鏈夋秷鎭彲浠ュ潎鍖€鍒嗗竷鍒颁笉鍚岀殑Partition閲岋紝杩欐牱灏卞疄鐜颁簡璐熻浇鍧囪 銆?/span>
濡傛灉涓€涓猅opic瀵瑰簲涓€涓枃浠讹紝閭ｈ繖涓枃浠舵墍鍦ㄧ殑鏈哄櫒I/O灏嗕細鎴愪负杩欎釜Topic鐨勬€ц兘鐡堕锛岃€屾湁浜哖artition鍚庯紝涓嶅悓鐨勬秷鎭彲浠ュ苟琛屽啓鍏ヤ笉鍚宐roker鐨勪笉鍚孭artition閲岋紝鏋佸ぇ鐨勬彁楂樹簡鍚炲悙鐜囥€傚彲浠ラ€氳繃閰嶇疆椤筺um.partitions鏉ユ寚瀹氭柊寤篢opic鐨勯粯璁artition鏁伴噺锛屼篃鍙湪鍒涘缓Topic鏃堕€氳繃鍙傛暟鎸囧畾锛屽悓鏃朵篃鍙互鍦═opic鍒涘缓涔嬪悗閫氳繃Kafka鎻愪緵鐨勫伐鍏蜂慨鏀广€?/span>
Kafka 涓殑姣忎釜涓婚鍒嗗尯閮借澶嶅埗浜?n 娆★紝鍏朵腑鐨?n 鏄富棰樼殑澶嶅埗鍥犲瓙锛坮eplication factor锛夈€傝繖鍏佽 Kafka 鍦ㄩ泦缇ゆ湇鍔″櫒鍙戠敓鏁呴殰鏃惰嚜鍔ㄥ垏鎹㈠埌杩欎簺鍓湰锛屼互渚垮湪鍑虹幇鏁呴殰鏃舵秷鎭粛鐒跺彲鐢ㄣ€侹afka 鐨勫鍒舵槸浠ュ垎鍖轰负绮掑害鐨勶紝鍒嗗尯鐨勯鍐欐棩蹇楄澶嶅埗鍒?n 涓湇鍔″櫒銆?nbsp;鍦?n 涓壇鏈腑锛屼竴涓壇鏈綔涓?leader锛屽叾浠栧壇鏈垚涓?followers銆傞【鍚嶆€濅箟锛宲roducer 鍙兘寰€ leader 鍒嗗尯涓婂啓鏁版嵁锛堣涔熷彧鑳戒粠 leader 鍒嗗尯涓婅繘琛岋級锛宖ollowers 鍙寜椤哄簭浠?leader 涓婂鍒舵棩蹇椼€?/span>
鏃ュ織澶嶅埗绠楁硶锛坙og replication algorithm锛夊繀椤绘彁渚涚殑鍩烘湰淇濊瘉鏄紝濡傛灉瀹冨憡璇夊鎴风娑堟伅宸茶鎻愪氦锛岃€屽綋鍓?leader 鍑虹幇鏁呴殰锛屾柊閫夊嚭鐨?leader 涔熷繀椤诲叿鏈夎娑堟伅銆傚湪鍑虹幇鏁呴殰鏃讹紝Kafka 浼氫粠澶卞幓 leader 鐨?ISR 閲岄潰閫夋嫨涓€涓?follower 浣滀负杩欎釜鍒嗗尯鏂扮殑 leader 锛涙崲鍙ヨ瘽璇达紝鏄洜涓鸿繖涓?span> follower 鏄窡涓?leader 鍐欒繘搴︾殑銆?/span>
姣忎釜鍒嗗尯鐨?leader 浼氱淮鎶や竴涓?ISR銆傚綋 producer 寰€ broker 鍙戦€佹秷鎭紝娑堟伅鍏堝啓鍏ュ埌瀵瑰簲 leader 鍒嗗尯涓婏紝鐒跺悗澶嶅埗鍒拌繖涓垎鍖虹殑鎵€鏈夊壇鏈腑銆傚彧鏈夊皢娑堟伅鎴愬姛澶嶅埗鍒版墍鏈夊悓姝ュ壇鏈紙ISR锛夊悗锛岃繖鏉℃秷鎭墠绠楄鎻愪氦銆傜敱浜庢秷鎭鍒跺欢杩熷彈鍒版渶鎱㈠悓姝ュ壇鏈殑闄愬埗锛屽洜姝ゅ揩閫熸娴嬫參鍓湰骞跺皢鍏朵粠 ISR 涓垹闄ら潪甯搁噸瑕併€侹afka 澶嶅埗鍗忚鐨勭粏鑺備細鏈変簺缁嗗井宸埆銆?/span>
Kafka涓嶆槸瀹屽叏鍚屾锛屼篃涓嶆槸瀹屽叏寮傛锛岃€屾槸涓€绉?span>ISR(In-Sync Replicas)鏈哄埗锛?nbsp;
1 leader浼氱淮鎶や竴涓笌鍏跺熀鏈繚鎸佸悓姝ョ殑Replica鍒楄〃锛岃鍒楄〃绉颁负ISR锛屾瘡涓狿artition閮戒細鏈変竴涓狪SR锛岃€屼笖鏄敱leader鍔ㄦ€佺淮鎶?nbsp;銆?/span>
2 濡傛灉涓€涓猣ollower姣斾竴涓猯eader钀藉悗澶锛屾垨鑰呰秴杩囦竴瀹氭椂闂存湭鍙戣捣鏁版嵁澶嶅埗璇锋眰锛屽垯leader灏嗗叾浠嶪SR涓Щ闄ゃ€?/span>
3 褰揑SR涓墍鏈塕eplica閮藉悜Leader鍙戦€丄CK鏃讹紝leader鎵峜ommit锛岃繖鏃跺€檖roducer鎵嶈兘璁や负涓€涓姹備腑鐨勬秷鎭兘commit浜嗐€?/span>
Kafka鎻愪緵浜嗘暟鎹鍒剁畻娉曚繚璇侊紝濡傛灉leader鍙戠敓鏁呴殰鎴栨寕鎺夛紝涓€涓柊leader琚€変妇骞惰鎺ュ彈瀹㈡埛绔殑娑堟伅鎴愬姛鍐欏叆銆侹afka纭繚浠庡悓姝ュ壇鏈垪琛ㄤ腑閫変妇涓€涓壇鏈负leader锛屾垨鑰呰follower杩借刀leader鏁版嵁銆?/span>leader璐熻矗缁存姢鍜岃窡韪狪SR涓墍鏈塮ollower婊炲悗鐨勭姸鎬?/span>銆?/span>
褰損roducer鍙戦€佷竴鏉℃秷鎭埌broker鍚庯紝leader鍐欏叆娑堟伅骞跺鍒跺埌鎵€鏈塮ollower銆傛秷鎭彁浜や箣鍚庢墠琚垚鍔熷鍒跺埌鎵€鏈夌殑鍚屾鍓湰銆傛秷鎭鍒跺欢杩熷彈鏈€鎱㈢殑follower闄愬埗锛岄噸瑕佺殑鏄揩閫熸娴嬫參鍓湰锛屽鏋渇ollower鈥滆惤鍚庘€濆お澶氭垨鑰呭け鏁堬紝leader灏嗕細鎶婂畠浠嶪SR涓垹闄ゃ€?/span>
鍓嶉潰宸茬粡浠嬬粛浜咾afka濡備綍杩涜鏈夋晥鐨勫瓨鍌紝浠ュ強浜嗚В浜唒roducer鍜宑onsumer濡備綍宸ヤ綔銆傛帴涓嬫潵璁ㄨ鐨勬槸Kafka濡備綍纭繚娑堟伅鍦╬roducer鍜宑onsumer涔嬮棿浼犺緭銆傛湁浠ヤ笅涓夌鍙兘鐨勪紶杈撲繚闅滐紙delivery guarantee锛?
At most once: 娑堟伅鍙兘浼氫涪锛屼絾缁濅笉浼氶噸澶嶄紶杈?/span>
At least once锛氭秷鎭粷涓嶄細涓紝浣嗗彲鑳戒細閲嶅浼犺緭
Exactly once锛氭瘡鏉℃秷鎭偗瀹氫細琚紶杈撲竴娆′笖浠呬紶杈撲竴娆?/span>
Kafka鐨勬秷鎭紶杈撲繚闅滄満鍒堕潪甯哥洿瑙傘€傚綋producer鍚慴roker鍙戦€佹秷鎭椂锛屼竴鏃﹁繖鏉℃秷鎭commit锛岀敱浜庡壇鏈満鍒讹紙replication锛夌殑瀛樺湪锛屽畠灏变笉浼氫涪澶便€備絾鏄鏋減roducer鍙戦€佹暟鎹粰broker鍚庯紝閬囧埌鐨勭綉缁滈棶棰樿€岄€犳垚閫氫俊涓柇锛岄偅producer灏辨棤娉曞垽鏂鏉℃秷鎭槸鍚﹀凡缁忔彁浜わ紙commit锛夈€?/span>铏界劧Kafka鏃犳硶纭畾缃戠粶鏁呴殰鏈熼棿鍙戠敓浜嗕粈涔堬紝浣嗘槸producer鍙互retry澶氭锛岀‘淇濇秷鎭凡缁忔纭紶杈撳埌broker涓紝鎵€浠ョ洰鍓岾afka瀹炵幇鐨勬槸at least once銆?/span>
consumer浠巄roker涓鍙栨秷鎭悗锛屽彲浠ラ€夋嫨commit锛岃鎿嶄綔浼氬湪Zookeeper涓瓨涓嬭consumer鍦ㄨpartition涓嬭鍙栫殑娑堟伅鐨刼ffset銆傝consumer涓嬩竴娆″啀璇昏partition鏃朵細浠庝笅涓€鏉″紑濮嬭鍙栥€傚鏈猚ommit锛屼笅涓€娆¤鍙栫殑寮€濮嬩綅缃細璺熶笂涓€娆ommit涔嬪悗鐨勫紑濮嬩綅缃浉鍚屻€傚綋鐒朵篃鍙互灏哻onsumer璁剧疆涓篴utocommit锛屽嵆consumer涓€鏃﹁鍙栧埌鏁版嵁绔嬪嵆鑷姩commit銆傚鏋滃彧璁ㄨ杩欎竴璇诲彇娑堟伅鐨勮繃绋嬶紝閭afka鏄‘淇濅簡exactly once, 浣嗘槸濡傛灉鐢变簬鍓嶉潰producer涓巄roker涔嬮棿鐨勬煇绉嶅師鍥犲鑷存秷鎭殑閲嶅锛岄偅涔堣繖閲屽氨鏄痑t least once銆?/span>
鑰冭檻杩欐牱涓€绉嶆儏鍐碉紝褰揷onsumer璇诲畬娑堟伅涔嬪悗鍏坈ommit鍐嶅鐞嗘秷鎭紝鍦ㄨ繖绉嶆ā寮忎笅锛屽鏋渃onsumer鍦╟ommit鍚庤繕娌℃潵寰楀強澶勭悊娑堟伅灏眂rash浜嗭紝涓嬫閲嶆柊寮€濮嬪伐浣滃悗灏辨棤娉曡鍒板垰鍒氬凡鎻愪氦鑰屾湭澶勭悊鐨勬秷鎭紝杩欏氨瀵瑰簲浜巃t most once浜嗐€傝瀹屾秷鎭厛澶勭悊鍐峜ommit銆傝繖绉嶆ā寮忎笅锛屽鏋滃鐞嗗畬浜嗘秷鎭湪commit涔嬪墠consumer crash浜嗭紝涓嬫閲嶆柊寮€濮嬪伐浣滄椂杩樹細澶勭悊鍒氬垰鏈猚ommit鐨勬秷鎭紝瀹為檯涓婅娑堟伅宸茬粡琚鐞嗚繃浜嗭紝杩欏氨瀵瑰簲浜巃t least once銆?/span>
涓嶅彧鏄疜afka, 绫讳技RabbitMQ浠ュ強RocketMQ杩欑被鍟嗕笟绾т腑闂翠欢涔熷彧淇濋殰at least once, 涓斾篃鏃犳硶浠庤嚜韬幓杩涜娑堟伅鍘婚噸銆傛墍浠ユ垜浠缓璁笟鍔℃柟鏍规嵁鑷韩鐨勪笟鍔＄壒鐐硅繘琛屽幓閲嶏紝姣斿涓氬姟娑堟伅鏈韩鍏峰骞傜瓑鎬э紝鎴栬€呭€熷姪Redis绛夊叾浠栦骇鍝佽繘琛屽幓閲嶅鐞嗐€?/span>
浼犵粺鐨勬秷鎭湁涓ょ妯″紡锛?/strong>闃熷垪鍜屽彂甯冭闃?/strong>銆?nbsp;鍦ㄩ槦鍒楁ā寮忎腑锛屾秷璐硅€呮睜浠庢湇鍔″櫒璇诲彇娑堟伅锛堟瘡涓秷鎭彧琚叾涓竴涓鍙栵級; 鍙戝竷璁㈤槄妯″紡锛氭秷鎭箍鎾粰鎵€鏈夌殑娑堣垂鑰呫€傝繖涓ょ妯″紡閮芥湁浼樼己鐐癸紝闃熷垪鐨勪紭鐐规槸鍏佽澶氫釜娑堣垂鑰呯摐鍒嗗鐞嗘暟鎹?/strong>锛岃繖鏍峰彲浠ユ墿灞曞鐞嗐€?/span>
闃熷垪锛氬厑璁稿悓鍚嶇殑娑堣垂鑰呯粍鎴愬憳鍏卞悓澶勭悊銆?/span>
鍙戝竷璁㈤槄锛氬箍鎾秷鎭粰澶氫釜娑堣垂鑰呯粍銆?/span>
鎵€鏈夊彂甯冩秷鎭埌娑堟伅闃熷垪鍜屾秷璐瑰垎绂荤殑绯荤粺锛屽疄闄呬笂閮藉厖褰撲簡涓€涓?span>涓存椂瀛樺偍绯荤粺銆侹afka杩樻槸涓€涓潪甯搁珮鎬ц兘鐨勫瓨鍌ㄧ郴缁熴€傚啓鍏ュ埌kafka鐨勬暟鎹皢鍐欏埌纾佺洏骞跺鍒跺埌闆嗙兢涓繚璇佸閿欐€с€傚苟鍏佽鐢熶骇鑰呯瓑寰呮秷鎭簲绛旓紝鐩村埌娑堟伅瀹屽叏鍐欏叆銆俴afka鐨勫瓨鍌ㄧ粨鏋勪繚璇佹棤璁烘湇鍔″櫒涓婃湁50KB鎴?0TB鏁版嵁锛屾墽琛屾晥鐜囨槸鐩镐技鐨勶紝鍥犳鍙揪鍒版按骞虫墿灞曠殑鐩爣銆傝繕鍙互璁や负kafka鏄竴绉嶄笓鐢ㄤ簬楂樻€ц兘锛屼綆寤惰繜锛屾彁浜ゆ棩蹇楀瓨鍌紝澶嶅埗锛屽拰浼犳挱鐗规畩鐢ㄩ€旂殑鍒嗗竷寮忔枃浠剁郴缁?/strong>銆?/span>
Kafka鐨勬洿楂樼洰鏍囨槸瀹炴椂娴佸鐞嗐€?span>鍦╧afka涓紝娴佸鐞嗘寔缁幏鍙栬緭鍏opic鐨勬暟鎹紝杩涜澶勭悊鍔犲伐锛岀劧鍚庡啓鍏ヨ緭鍑簍opic銆?/strong>渚嬪锛屼竴涓浂鍞瓵PP锛屾帴鏀堕攢鍞拰鍑鸿揣鐨勮緭鍏ユ祦锛岀粺璁℃暟閲忔垨璋冩暣浠锋牸鍚庤緭鍑恒€?/span>
绠€鍗曠殑闇€姹傚彲浠ョ洿鎺ヤ娇鐢╬roducer鍜宑onsumer API杩涜澶勭悊銆傚浜庡鏉傜殑杞崲锛孠afka鎻愪緵浜嗘洿寮哄ぇ鐨凷treams API銆?span>鍙瀯寤鸿仛鍚堣绠楁垨杩炴帴娴佸埌涓€璧风殑澶嶆潅搴旂敤绋嬪簭銆?/span>
缁间笂鎵€杩帮紝Kafka 鐨勮璁″彲浠ュ府鍔╂垜浠В鍐冲緢澶氭灦鏋勪笂鐨勯棶棰樸€備絾鏄兂瑕佺敤濂?Kafka 鐨?span>楂樻€ц兘銆佷綆鑰﹀悎銆侀珮鍙潬鎬?/strong>绛夌壒鎬э紝鎴戜滑闇€瑕侀潪甯镐簡瑙?Kafka锛屼互鍙婃垜浠嚜韬殑涓氬姟闇€姹傦紝缁煎悎鑰冭檻搴旂敤鍦烘櫙銆?/span>
馃殌
馃憞馃徎鐐瑰嚮銆岄槄璇诲師鏂囥€?/strong>鏌ョ湅鏇村娣卞害鎶€鏈ソ鏂?/span>
以上是关于的主要内容，如果未能解决你的问题，请参考以下文章