Flink 鐏甸瓊涓ょ櫨闂紝杩欒皝椤跺緱浣忥紵

Posted 2021-04-13 zhisheng

涓ょ櫨闂竴鐐归兘涓嶅じ寮狅紝杩欐墠鍏夌煡璇嗘槦鐞冧粠鍒涘缓鍒扮幇鍦ㄧ殑闂鎬绘暟锛屽彟澶栬繕鏈夊涓井淇＄兢鐨勭瓟鐤戣В鎯戙€傛湁鏃跺€欏繊涓嶄綇鎯崇粰鑷繁鐐逛釜璧烇紝鍜嬭兘杩欎箞馃悅鍛紵

鏈」鐩粨鏋?/span>

2019/06/08 鏂板 Flink 鍥涙湰鐢靛瓙涔︾睄鐨?PDF锛屽湪 books 鐩綍涓嬶細

• Introduction_to_Apache_Flink_book.pdf 杩欐湰涔︽瘮杈冭杽锛屽浜庝粙缁嶉樁娈碉紝鍥藉唴鏈夎繖鏈殑缈昏瘧涔︾睄

• Learning Apache Flink.pdf 杩欐湰涔︽瘮杈冨熀纭€锛屽垵瀛︾殑璇濆彲浠ュ鐪嬬湅

• Stream Processing with Apache Flink.pdf 杩欐湰涔︽槸 Flink PMC 鍐欑殑

• Streaming System.pdf 杩欐湰涔﹁瘎浠蜂笉鏄竴鑸殑楂?/p>

2019/06/09 鏂板娴佸鐞嗗紩鎿庣浉鍏崇殑 Paper锛屽湪 paper 鐩綍涓嬶細

• 娴佸鐞嗗紩鎿庣浉鍏崇殑 Paper

鍗氬

Flink 婧愮爜椤圭洰缁撴瀯

鐭ヨ瘑鏄熺悆

鏈変汉瑕侀棶鐭ヨ瘑鏄熺悆閲岄潰鏇存柊浠€涔堝唴瀹癸紵鍊煎緱鍔犲叆鍚楋紵

鐩墠鐭ヨ瘑鏄熺悆鍐呭凡鏇存柊鐨勭郴鍒楁枃绔狅細

闄や簡銆婁粠1鍒?00娣卞叆瀛︿範Flink銆嬫簮鐮佸涔犺繖涓郴鍒楁枃绔狅紝銆婁粠0鍒?瀛︿範Flink銆嬬殑妗堜緥鏂囩珷涔熶細浼樺厛鍦ㄧ煡璇嗘槦鐞冩洿鏂帮紝璁╁ぇ瀹跺厛閫氳繃涓€浜?demo 瀛︿範 Flink锛屽啀鍘绘繁鍏ユ簮鐮佸涔狅紒

濡傛灉瀛︿範 Flink 鐨勮繃绋嬩腑锛岄亣鍒颁粈涔堥棶棰橈紝鍙互鍦ㄩ噷闈㈡彁闂紝鎴戜細浼樺厛瑙ｇ瓟锛岃繖閲屽仛涓姳姝夛紝鑷繁骞虫椂宸ヤ綔涔熸尯蹇欙紝寰俊鐨勯棶棰樹笉鑳藉仛鍏ㄩ儴鍋氫竴浜涜В绛旓紝
浣嗚偗瀹氫細浼樺厛鍥炲缁欑煡璇嗘槦鐞冪殑浠樿垂鐢ㄦ埛鐨勶紝搴嗗垢鐨勬槸鐜板湪鏄熺悆閲岀殑娲昏穬姘涘洿杩樻槸鍙互鐨勶紝鏈変笉灏戦棶棰橀€氳繃鎻愰棶鍜岃В绛旂殑鏂瑰紡娌夋穩浜嗕笅鏉ャ€?/p>

1銆佷负浣曟垜浣跨敤 ValueState 淇濆瓨鐘舵€?Job 鎭㈠鏄姸鎬佹病鎭㈠锛?/span>

2銆乫link涓瓀atermark绌剁珶鏄浣曠敓鎴愮殑锛岀敓鎴愮殑瑙勫垯鏄粈涔堬紝鎬庝箞鐢ㄦ潵澶勭悊涔卞簭鏁版嵁

3銆佹秷璐筴afka鏁版嵁鐨勬椂鍊欙紝濡傛灉閬囧埌浜嗚剰鏁版嵁锛屾垨鑰呮槸涓嶇鍚堣鍒欑殑鏁版嵁绛夌瓑鎬庝箞澶勭悊鍛紵

4銆佸湪Kafka 闆嗙兢涓€庝箞鎸囧畾璇诲彇/鍐欏叆鏁版嵁鍒版寚瀹歜roker鎴栦粠鎸囧畾broker鐨刼ffset寮€濮嬫秷璐癸紵

5銆丗link鑳介€氳繃oozie鎴栬€卆zkaban鎻愪氦鍚楋紵

6銆乯obmanager鎸傛帀鍚庯紝鎻愪氦鐨刯ob鎬庝箞涓嶇粡杩囨墜鍔ㄩ噸鏂版彁浜ゆ墽琛岋紵

7銆佷娇鐢╢link-web-ui鎻愪氦浣滀笟骞舵墽琛?浣嗘槸/opt/flink/log鐩綍涓嬫病鏈夋棩蹇楁枃浠?璇烽棶鍏充簬flink鐨勬棩蹇楋紙鍖呮嫭jobmanager銆乼askmanager銆佹瘡涓猨ob鑷繁鐨勬棩蹇楅粯璁ゅ垎鍒瓨鍦ㄥ摢涓洰褰?锛夐渶瑕佹€庝箞閰嶇疆锛?/span>

8銆侀€氳繃flink 浠〃鐩樻彁浜ょ殑jar 鏄瓨鍌ㄥ湪鍝釜鐩綍涓嬶紵

9銆佷粠Kafka娑堣垂鏁版嵁杩涜etl娓呮礂锛屾妸缁撴灉鍐欏叆hdfs鏄犲皠鎴恏ive琛紝鍘嬬缉鏍煎紡銆乭ive鐩存帴鑳藉璇诲彇flink鍐欏嚭鐨勬枃浠躲€佹寜鐓ф枃浠跺ぇ灏忔垨鑰呮椂闂存粴鍔ㄧ敓鎴愭枃浠?/span>

10銆乫link jar鍖呬笂浼犺嚦闆嗙兢涓婅繍琛岋紝鎸傛帀鍚庯紝鎸傛帀鏈熼棿kafka涓湭琚秷璐圭殑鏁版嵁锛屽湪閲嶆柊鍚姩绋嬪簭鍚庯紝鏄嚜鍔ㄤ粠checkpoint鑾峰彇鎸傛帀涔嬪墠鐨刱afka offset浣嶇疆锛岃嚜鍔ㄦ秷璐逛箣鍓嶇殑鏁版嵁杩涜澶勭悊锛岃繕鏄渶瑕佹煇浜涙墜鍔ㄧ殑鎿嶄綔鍛紵

11銆乫link 鍚姩鏃朵笉鑷姩鍒涘缓 涓婁紶jar鐨勮矾寰勶紝鑳芥寚瀹氫竴涓垱寤哄ソ鐨勭洰褰曞悧

12銆丗link sink to es 闆嗙兢涓婃姤 slot 涓嶅锛屽崟鏈鸿窇鏄ソ鐨勶紝涓轰粈涔堬紵

13銆丗llink to elasticsearch濡備綍鍒涘缓绱㈠紩鏂囨。鏈熸椂闂存埑锛?/span>

14銆乥link鏈夋病鏈塧pi鏂囨。鎴栬€卍emo锛屾槸鍚﹀缓璁産link鐢ㄤ簬鐢熶骇鐜銆?/span>

15銆乫link鐨凱ython api鎬庢牱锛焍ug澶氬悧锛?/span>

16銆丗link VS Spark Streaming VS Storm VS Kafka Stream

17銆佷綘浠仛瀹炴椂澶у睆鐨勬妧鏈灦鏋勬槸浠€涔堟牱瀛愮殑锛焒lume鈫択afka鈫抐link鈫抮edis锛岀劧鍚庡悗绔幓redis閲岄潰鎹炴暟鎹紝閰辩传鍙鍚楋紵

18銆佸仛涓€涓粺璁℃寚鏍囩殑鏃跺€欙紝闇€瑕佸湪Flink鐨勮绠楄繃绋嬩腑澶氭璇诲啓redis锛屾劅瑙夊ソ鎬紝鏄熶富鏈夋病鏈夊ソ鐨勬柟妗堬紵

19銆丗link 浣跨敤鍦烘櫙澶у垎鏋愶紝鍒椾妇浜嗗緢澶氱殑甯哥敤鍦烘櫙锛屽彲浠ュソ濂藉弬鑰冧竴涓?/span>

20銆佸皢kafka涓暟鎹畇ink鍒癿ysql鏃讹紝metadata鐨勬暟鎹负绌猴紝瀵煎叆mysql鏁版嵁涓嶆垚鍔燂紵锛燂紵

21銆佷娇鐢ㄤ簡ValueState鏉ヤ繚瀛樹腑闂寸姸鎬侊紝鍦ㄨ繍琛屾椂涓棿鐘舵€佷繚瀛樻甯革紝浣嗘槸鍦ㄦ墜鍔ㄥ仠姝㈠悗锛屽啀閲嶆柊杩愯锛屽彂鐜颁腑闂寸姸鎬佸€兼病鏈変簡锛屼箣鍓嶅嚭鐜扮殑閿€兼槸浠?寮€濮嬭鏁扮殑锛岃繖鏄负浠€涔堬紵鏄渶瑕佸疄鐜癈heckpointedFunction鍚楋紵

22銆乫link on yarn jobmanager鐨凥A闇€瑕佹€庝箞閰嶇疆銆傝繕鏄yarn缁欑鐞嗕簡

23銆佹湁涓や釜鏁版嵁娴佸氨琛宑onnect锛屽叾涓竴涓槸瀹炴椂鏁版嵁娴侊紙kafka 璇诲彇)锛屽彟涓€涓槸閰嶇疆娴併€傜敱浜庨厤缃祦鏄粠鍏崇郴鍨嬫暟鎹簱涓鍙栵紝閫熷害杈冩參锛屽鑷村疄鏃舵暟鎹祦娴佸叆鏁版嵁鐨勬椂鍊欙紝閰嶇疆淇℃伅杩樻湭鍙戦€侊紝杩欐牱浼氬鑷存湁浜涘疄鏃舵暟鎹鍙栦笉鍒伴厤缃俊鎭€傜洰鍓嶉噰鍙栫殑鎺柦鏄湪connect鏂规硶鍚庣殑flatmap鐨勫疄鐜扮殑鍦╫pen 鏂规硶涓紝鎻愬墠鍔犺浇涓€娆￠厤缃俊鎭紝鎰熻杩欑瀹炵幇鏂瑰紡涓嶅弸濂斤紝璇烽棶杩樻湁鍏朵粬鐨勫疄鐜版柟寮忓悧锛?/span>

24銆丗link鑳介€氳繃oozie鎴栬€卆zkaban鎻愪氦鍚楋紵

25銆佷笉閲囩敤yarm閮ㄧ讲flink锛岃繕鏈夊叾浠栫殑鏂规鍚楋紵涓昏鎯宠В鍐虫湇鍔″櫒閲嶅惎鍚庯紝flink鏈嶅姟鎬庝箞鑷姩鎷夎捣锛焜obmanager鎸傛帀鍚庯紝鎻愪氦鐨刯ob鎬庝箞涓嶇粡杩囨墜鍔ㄩ噸鏂版彁浜ゆ墽琛岋紵

26銆佸湪涓€涓?Job 閲屽皢鍚屼唤鏁版嵁鏄ㄦ櫄娓呮礂鎿嶄綔鍚庯紝sink 鍒板悗绔涓湴鏂癸紙鐪嬩笟鍔￠渶姹傦級锛屽浣曚繚鎸佷竴鑷存€э紵锛堜竴涓猻ink鍑洪敊锛屽彟澶栫殑涔熶繚璇佷笉鑳芥彃鍏ワ級

27銆乫link sql浠诲姟鍦ㄦ煇涓壒瀹氶樁娈典細鍙戠敓tm鍜宩m涓㈠け蹇冭烦锛屾槸涓嶆槸鐢变簬gc鏃堕棿杩囬暱鍛紝

28銆佹湁杩欐牱涓€涓渶姹傦紝缁熻鐢ㄦ埛杩戜袱鍛ㄨ繘鍏ヤ骇鍝佽鎯呴〉鐨勬潵婧愶紙1棣栭〉澶ф悳绱紝2浜у搧棰戦亾鎼滅储锛?鍏朵粬锛夛紝涓簆hp鍚庣鎻愪緵鏁版嵁鏀寔锛岃淇℃伅鍦ㄧ涓婃姤浜嬩欢涓紝php鐩存帴鑾峰彇鏈夌偣鍥伴毦銆傛垜鐜板湪鐨勮В鍐虫柟妗?閫氳繃flink婊氬姩绐楀彛锛堝崐灏忔椂锛夛紝缁熻鐢ㄦ埛鍗婂皬鏃跺唴3涓潵婧恜v锛岀劧鍚庢寜鐓ф棩鏈熷簭鍒楀寲锛岀洿鎺ュ啓mysql銆俻hp浠庢暟鎹簱涓В鏋愬嚭鏉ワ紝鍐嶅幓缁熻杩戜袱鍛ㄥ崰姣斻€傞棶棰?锛岃繖涓渶姹傞€傚悎鐢╢link鍘诲仛鍚楋紵闂2锛屾垜鐨勬柟妗堟€绘劅瑙夋€€殑锛屾湁娌℃湁濂界殑鏂规锛?/span>

29銆佷竴涓猼ask slot 鍙兘鍚屾椂杩愯涓€涓换鍔¤繕鏄涓换鍔″憿锛熷鏋渢ask slot杩愯鐨勪换鍔℃瘮杈冨ぇ锛屼細鍑虹幇OOM鐨勬儏鍐靛悧锛?/span>

30銆佷綘浠€庝箞瀵圭嚎涓奻link鍋氱洃鎺х殑锛屽鏋滄暣涓▼搴忓け璐ヤ簡鎬庝箞鑷姩閲嶅惎绛夌瓑

31銆乫link cep瑙勫垯鍔ㄦ€佽В鏋愭湁鎺ヨЕ鍚楋紵鏈夋病鏈夋垚鍨嬬殑妗嗘灦锛?/span>

32銆佹瘡涓€涓猈indow閮芥湁涓€涓獁atermark鍚楋紵window鏄€庝箞鏍规嵁watermark杩涜瑙﹀彂鎴栬€呴攢姣佺殑锛?/span>

33銆?CheckPoint涓嶴avePoint鐨勫尯鍒槸浠€涔堬紵

34銆乫link鍙互鍦ㄧ畻瀛愪腑鍏变韩鐘舵€佸悧锛熸垨鑰呭ぇ浣綘鏈変粈涔堟柟娉曞彲浠ュ叡浜姸鎬佺殑鍛紵

35銆佽繍琛屽嚑鍒嗛挓灏辨姤浜嗭紝鐪媡askmager鏃ュ織锛屾姤鐨勬槸 failed elasticsearch bulk request null锛屽彲鏄垜浠ｇ爜閲岄潰宸茬粡鍋氳繃绌哄€煎垽鏂簡鍛€ 鑰屼笖涔熻繃婊ゆ帀浜嗭紝flink鐗堟湰1.7.2 es鐗堟湰6.3.1

36銆佽繖绉嶆儏鍐碉紝鎴戜滑璋冨苟琛屽害 杩樻槸閰嶇疆鍙傛暟濂?/span>

37銆佸ぇ瀹堕兘鐢╦dbc鍐欙紝鍚勭鏁版嵁搴撳鍒犳煡鏀规嫾sql鏈夋病鏈夎寰楀緢绱紝ps.set浠ｇ爜涓€澶у爢锛岃繕瑕佽绠楁瘡涓弬鏁扮殑浣嶇疆

38銆佸叧浜巇atasource鐨勯厤缃紝姣忎釜taskmanager瀵瑰簲涓€涓猟atasource?杩樻槸姣忎釜slot? 瀹為檯杩愯涓嬫潵锛屾瘡涓猻lot涓璬atasorce绾跨▼姹犲彧瑕佽缃?灏辫浜嗭紝澶氫簡涔熺敤涓嶅埌?

39銆乲afka鐜板湪姣忓ぉ鍑虹幇鏁版嵁涓㈠け锛岀幇鍦ㄥ皬鎵归噺鏁版嵁锛屼竴澶?00W宸﹀彸, kafka鐗堟湰涓?1.0.0锛岄泦缇ゆ€诲叡7涓妭鐐癸紝TOPIC鏈夊崄鍏釜鍒嗗尯锛屽崟鏉℃姤鏂?.5k宸﹀彸

40銆佹牴鎹甼ey.hash鐨勭粷瀵瑰€?瀵瑰苟鍙戝害姹傛ā锛岃繘琛屽垎缁勶紝鍋囪10鍚勫苟鍙戝害锛屽疄闄呭彧鏈?涓垎鍖烘湁澶勭悊鏁版嵁锛屾湁2涓缁堜笉澶勭悊锛岃繕鏈変竴涓垎鍖哄鐞嗙殑鏁版嵁鏄叾浠栫殑涓夊€嶏紝濡傛埅鍥?/span>

41銆乫link姣?灏忔椂涓嶇煡閬撳湪澶勭悊浠€涔堬紝 CPU 璐熻浇 姣?灏忔椂锛屾湁涓€娆￠珮宄帮紝5鍒嗛挓鍐呭钩鍧囪礋杞借秴杩?.8锛屽鎴浘

42銆佹湁娌℃湁Flink鍐欑殑椤圭洰鎺ㄨ崘锛熸垜鎯崇湅鍒扮敤Flink鍐欑殑鏁翠綋椤圭洰鏄€庝箞缁勭粐鐨勶紝涓嶅崟鍗曟槸涓€涓崟渚嬪瓙

43銆丗link 婧愮爜鐨勭粨鏋勫浘

44銆佹垜鎯虫牴鎹笉鍚屼笟鍔¤〃锛坈ase when锛夎繘琛屼笉鍚岀殑redis sink锛坔ash 锛宻et锛夛紝鎴戣濡備綍鎿嶄綔锛?/span>

45銆佽繖涓渶瑕佹竻鐞嗕粈涔堟暟鎹憖锛屾垜鎶奾dfs閲岄潰鐨勫凡缁忔竻鐞嗕簡 鍚姩杩樻槸鎶ヨ繖涓?/span>

46銆?nbsp;鍦ㄦ祦澶勭悊绯荤粺锛屽湪鏈哄櫒鍙戠敓鏁呴殰鎭㈠涔嬪悗锛屼粈涔堟儏鍐垫秷鎭渶澶氫細琚鐞嗕竴娆★紵浠€涔堟儏鍐垫秷鎭渶灏戜細琚鐞嗕竴娆″憿锛?/span>

47銆佹垜妫€鏌ョ偣閮借皟鍒?鍒嗛挓浜嗭紝杩欐槸浠€涔堥棶棰?/span>

48銆乺educe鏂规硶鍚?閭ｄ釜浜ゆ槗鏃堕棿 鎬庝箞涓嶆槸鏈€鏂扮殑锛屾槸绗竴娆¤繘鍏ョ殑閭ｄ釜鏃堕棿锛?/span>

49銆丗link on Yarn 妯″紡锛岀敤yarn session鑴氭湰鍚姩鐨勬椂鍊欙紝鎴戝湪鍚庡彴娌℃湁鐪嬪埌鍒癑obmanager锛孴askManager锛孉pplicationMaster杩欏嚑涓繘绋嬶紝鎯宠闂竴涓嬭繖鏄粈涔堝師鍥犲憿锛熷洜涓轰箣鍓嶇湅瀹樼綉鐨勬椂鍊欙紝璇碕obmanager灏辨槸涓€涓猨vm杩涚▼锛孴askmanage涔熸槸涓€涓狫VM杩涚▼

50銆丗link on Yarn鐨勬椂鍊欏緱鎸囧畾 澶氬皯涓猅askManager鍜屾瘡涓猅askManager slot鍘昏繍琛屼换鍔★紝杩欐牱鍋氭劅瑙変笉澶悎鐞嗭紝鍥犱负鐢ㄦ埛涔熶笉鐭ラ亾闇€瑕佸灏戜釜TaskManager閫傚悎锛孎link 鏈夊姩鎬佸惎鍔═askManager鐨勬満鍒跺悧銆?/span>

51銆佸弬鑰冭繖涓緥瀛愶紝Flink 闆跺熀纭€瀹炴垬鏁欑▼锛氬浣曡绠楀疄鏃剁儹闂ㄥ晢鍝?| Jark's Blog锛?绐楀彛鑱氬悎鐨勬椂鍊欙紝鐢╧eywindow锛岀敤鐨勬槸timeWindowAll锛岀劧鍚庡湪aggregate鐨勬椂鍊欑敤aggregate(new CustomAggregateFunction(), new CustomWindowFunction())锛屾墦鍗扮粨鏋滃悗锛屽彂鐜扮獥鍙ｄ腑涓€鐩翠娇鐢ㄧ殑閲嶅鐨勬暟鎹紝缁熻鐨勭粨鏋滀篃涓嶅彉锛屽幓鎺塁ustomWindowFunction()灏辨甯镐簡 锛熼潪甯稿鎬?/span>

52銆佺敤鎴疯繘鍏ヤ骇鍝侀瀹氶〉闈紙绔煁鐐逛笂鎶ワ級锛屽苟濉啓浜嗕竴浜涗俊鎭紙绔煁鐐逛笂鎶ワ級锛屼絾鍗婂皬鏃跺唴骞舵病鏈変骇鐢熶换浣曡鍗曪紝鐒跺悗缁欒绫荤敤鎴峰彂閫佷竴涓猵ush銆?. 杩欑闇€姹傞€傚悎鐢╢link鍘诲仛鍚楋紵2. 濡傛灉閫傚悎锛岃涓嬪ぇ姒傜殑鎬濊矾

53銆佷笟鍔″満鏅槸瀹炴椂鑾峰彇鏁版嵁瀛榬edis锛岃闂垜瑕佸浣曟寜澶┿€佹寜鍛ㄣ€佹寜鏈堝垎鍒瓨鍏edis閲岋紵锛堟瘮鏂硅杩囦簡涓€澶╄嚜鍔ㄦ崲涓€涓綅缃瓨redis锛?/span>

54銆佹湁浜?AggregatingState 鐨勪緥瀛愬悧, 鎰熻瀹樻柟鐨勪緥瀛愬拰 瀹樼綉鐨勪笉澶竴鏍?

55銆乫link-jdbc杩欎釜jar鏈夊悧锛熸€庝箞娌℃壘鍒板晩锛?.8.0鐨勬病鎵惧埌锛?.6.2鐨勬湁

56銆佺幇鏈変釜鍏充簬savepoint鐨勯棶棰橈紝鎿嶄綔娴佺▼涓猴紝鍙栨秷浠诲姟鏃惰缃繚瀛樼偣锛屾洿鏂颁换鍔★紝浠庝繚瀛樼偣鍚姩浠诲姟锛涚幇鍦ㄩ亣鍒颁釜闂锛屽亣璁炬垜涓棿鏌愪釜绠楀瓙閲嶅啓锛屽師鍏堥€氳繃state缂栧啓锛屾湁鐢ㄥ畾鏃跺櫒锛岀幇鍦ㄦ洿鏀瑰悗锛岄噰鐢ㄧ獥鍙ｏ紝鍙嶆灏辨槸瀹炵幇鏂瑰紡瀹屽叏涓嶄竴鏍凤紱浠庝繚瀛樼偣鍚姩灏变細涓€鐩存姤閿欙紝閲嶅惎锛屽師鍏堢殑淇濆瓨鐐逛笉鑳借繕鍘燂紝姝ゆ椂灏变細鏈夊緢澶氭暟鎹噸澶嶇瓑鍚勭闂锛屽浣曟墠鑳戒繚璇佹暟鎹笉涓㈠け锛屼笉閲嶅绛夛紝鎭㈠鍒板仠姝㈢殑鏃跺€欙紝鐜板湪鎯冲埌鐨勬槸璁颁笅kafka鐨勫亸绉婚噺锛屽啀鍋氬鐞嗭紝璨屼技涔熶笉鏄緢濂藉紕锛屾湁浠€涔堣В鍐冲姙娉曞悧

57銆侀渶瑕佸湪flink璁＄畻app椤甸潰璁块棶鏃堕暱锛屾秷璐筀afka璁＄畻鍚庤緭鍑哄埌Kafka銆傜涓€鏉og闇€瑕佺瓑寰呯浜屾潯log鐨勬椂闂存埑璁＄畻璁块棶鏃堕暱銆傛垜鎯抽棶鐨勬槸锛宖link鏄垎甯冨紡鐨勶紝閭ｄ箞瀹冭兘鍚︿繚璇佹墽琛岀殑椤哄簭鎬э紵鍚庢潵鐨勬暟鎹湁娌℃湁鍙兘鍏堣鎵ц锛?/span>

58銆佹垜鍏徃鎯冲仛瀹炴椂澶у睆锛岀幇鏈夋妧鏈槸灏嗕笟鍔℃墍闇€鎸囨爣瀹炴椂鐢╯park鎷夊埌redis閲屽瓨鐫€锛岀劧鍚庡啀鐢ㄤ竴鏉park streaming娴佽绠楃畝鍗曚箻闄よ繍绠楋紝鎸囨爣鍖呭惈浜嗗悇鏈堜唤鐨勬瘮杈冦€傝闂垜璇ュ浣曠敤flink绠€鍖栦笂杩版祦绋嬶紵

59銆乫link on yarn 鏂瑰紡锛岃繖鏍风悊瑙ｄ笉鐭ラ亾瀵逛笉瀵癸紝yarn-session杩欎釜鑴氭湰鍏跺疄灏辨槸鍑嗗yarn鐜鐨勶紝鎵цrun浠诲姟鐨勬椂鍊欙紝鏍规嵁yarn-session鍒濆鍖栫殑yarnDescription 鎶?flink 浠诲姟鐨刯obGraph鎻愪氦鍒皔arn涓婂幓鎵ц

60銆佸悓鏍风殑浠ｇ爜閫昏緫鍐欏湪鍗曠嫭鐨刴ain鍑芥暟涓氨鍙互鎴愬姛鐨勬秷璐筴afka 锛屽啓鍦ㄤ竴涓猻pring boot鐨勭▼搴忎腑锛屾帴鍙楀閮ㄨ姹傦紝鐒跺悗鎵ц鐩稿悓鐨勯€昏緫灏变笉鑳芥秷璐筴afka銆備綘閬囧埌杩囧悧锛熻兘缁欎竴浜涙煡闂鐨勫缓璁紝鎴栬€呭湪鍝噷鎵撲釜鏂偣锛岃兘鐪嬪埌涓轰粈涔堟秷璐逛笉鍒発afka鐨勬秷鎭憿锛?/span>

61銆佽闂笅flink鍙互瀹炵幇涓€涓祦涓悓鏃跺瓨鍦ㄨ鍗曡〃鍜岃鍗曞晢鍝佽〃鐨勬暟鎹?涓よ€呮槸涓€瀵瑰鐨勫叧绯?鑳藉疄鐜板緱鍒?浠ヨ鍗曡〃涓轰富 涓€涓鍗曞涓晢鍝?杩欑闇€姹傚槢

62銆佸湪鐢ㄤ腑闂寸姸鎬佺殑鏃跺€欙紝濡傛灉涓棿涓€浜涗俊鎭繚瀛樺湪state涓紝鏈夋病鏈夊繀瑕佸湪redis涓啀淇濆瓨涓€浠斤紝鏉ュ仛绗笁鏂圭殑瀛樺偍銆?/span>

63銆佽兘鍚﹀嚭涓€鏈焒link state鐨勬枃绔犮€備粈涔堝満鏅笅鐢ㄤ粈涔堟牱鐨剆tate锛熷锛屾渶绠€鍗曠殑锛屽疄鏃剁疮鍔爑pdate鍒皊tate銆?/span>

64銆乫link鐨勫弻娴乯oin鍗氫富鏈変娇鐢ㄧ殑缁忛獙鍚楋紵浼氭湁浠€涔堝父瑙佺殑闂鍚?/span>

65銆佺獥鍙ｈЕ鍙戠殑鏉′欢闂

66銆乫link 瀹氭椂浠诲姟鎬庝箞鍋氾紵鏈夌浉鍏崇殑demo涔堬紵

67銆佹祦寮忓鐞嗚繃绋嬩腑鏁版嵁鐨勪竴鑷存€у浣曚繚璇佹垨鑰呭浣曟娴?/span>

68銆侀噸鍚痜link鍗曟満闆嗙兢锛岃繕鎶ob not found 寮傚父銆?/span>

69銆乲afka鐨勬暟鎹槸鐢?org.apache.kafka.common.serialization.ByteArraySerialize搴忓垪鍖栫殑锛宖link杩欒竟娑堣垂鐨勬椂鍊欐€庝箞閫氳繃FlinkKafkaConsumer鍒涘缓DataStream 锛?/span>

70銆佺幇鍦ㄥ叕鍙告湁涓€涓渶姹傦紝涓€浜涚敤鎴风殑鏀粯鏃ュ織锛岄€氳繃sls鏀堕泦锛岃鎶婅繖浜涙棩蹇楀鐞嗗悗锛岀粨鏋滃啓鍏ュ埌MySQL锛屽叧閿繖浜涙棩蹇楀彲鑳借繛鐫€鏉ュソ鍑犳潯鎵嶆槸涓€涓敤鎴风殑锛屽洜涓哄彂璧疯姹傦紝鍝嶅簲绛夋瘡涓幆鑺傞兘鏈夌浉搴旂殑鏃ュ織锛岃繖鍑犳潯鏃ュ織缁煎悎澶勭悊鎵嶈兘寰楀埌鏈€缁堢殑缁撴灉锛岃闂崥涓绘湁浠€涔堝ソ鐨勬柟娉曟病鏈夛紵

71銆乫link 鏀寔hadoop 涓诲涔堬紵hadoop涓昏妭鐐规寕浜?flink 浼氬垏鎹㈠埌hadoop 澶囩敤鑺傜偣锛?/span>

72銆佽鏁欏ぇ瀹? 瀹為檯 flink 寮€鍙戜腑鐢?scala 澶氳繕鏄?java澶氫簺锛熷垰鍏ユ墜 flink 澶ф暟鎹?scala 闇€瑕佹繁鍏ュ涔犱箞锛?/span>

73銆佹垜浣跨敤鐨勬槸flink鏄?.7.2鏈€杩戠敤浜唖plit鐨勬柟寮忓垎娴侊紝浣嗘槸搴曞眰鐨凷plitStream涓婂嵈鏍囨敞涓篋eprecated锛岃闂槸瀹樻柟涓嶆帹鑽愪娇鐢ㄥ垎娴佺殑鏂瑰紡鍚楋紵

74銆並eyBy 鐨勬纭悊瑙ｏ紝鍜屾暟鎹€炬枩闂鐨勮В閲?/span>

75銆佺敤flink鏃讹紝閬囧埌涓棶棰?checkpoint澶ф鏈?G宸﹀彸锛?鏈夎儗鍘嬫椂锛宖link浼氶噸鍚湁閬囧埌杩囪繖涓棶棰樺悧

76銆乫link浣跨敤yarn-session鏂瑰紡閮ㄧ讲锛屽浣曚繚璇亂arn-session鐨勭ǔ瀹氭€э紝濡傛灉yarn-session鎸備簡锛岄渶瑕侀噸鏂伴儴缃蹭竴涓獃arn-session锛屽浣曟仮澶嶄箣鍓峺arn-session涓婄殑job鍛紝涔嬪墠鐨刢heckpoint杩樿兘浣跨敤鍚楋紵

77銆佹垜鎯宠鏁欎竴涓嬪叧浜巗ink鐨勯棶棰樸€傛垜鐜板湪鐨勯渶姹傛槸浠嶬afka娑堣垂Json鏁版嵁锛岃繖涓狫son鏁版嵁瀛楁鍙兘浼氬鍔狅紝鐒跺悗灏嗘嬁鍒扮殑json鏁版嵁浠arquet鐨勬牸寮忓瓨鍏dfs銆傜幇鍦ㄦ垜鍙互鎷垮埌json鏁版嵁鐨剆chema锛屼絾鏄湪淇濆瓨parquet鏂囦欢鐨勬椂鍊欎笉鐭ラ亾鎬庝箞澶勭悊銆備竴鏄痜link娌℃湁涓撻棬鐨刦ormat parquet锛屼簩鏄浜庡彲鍙樺瓧娈电殑Json鎬庝箞澶勭悊鎴恜arquet姣旇緝鍚堥€傦紵

78銆乫link濡備綍鍦ㄨ緝澶х殑鏁版嵁閲忎腑鍋氬幓閲嶈绠椼€?/span>

79銆乫link鑳藉湪娌℃湁鏁版嵁鐨勬椂鍊欎篃瀹氭椂鎵ц绠楀瓙鍚楋紵

80銆佷娇鐢╮ocksdb鐘舵€佸悗绔紝鑷畾涔塸ojo鎬庝箞瀹炵幇搴忓垪鍖栧拰鍙嶅簭鍒楀寲鐨勶紝鏈夌浉鍏砫emo涔堬紵

81銆乧heck point 鑰佹槸澶辫触锛屾槸涓嶆槸鑷畾涔夌殑pojo闂锛熷埌鏈湴鍙互锛屽埌hdfs灏变笉琛岋紝缃戜笂涔熸湁寰堝绫讳技鐨勯棶棰?閮芥病鏈変竴涓緢濂界殑瑙ｉ噴鍜岃В鍐虫柟妗?/span>

82銆乧ep瑙勫垯濡傚浘锛屽綋start浜嬩欢杩涘叆鏃讹紝鏃堕棿00:00:15锛岃€屽悗杩涘叆end浜嬩欢锛屾椂闂?0:00:40銆傛垜鍙戠幇瑙勫垯鏃犳硶鍛戒腑銆傝闂畐ithin 鏄粠start浜嬩欢寮€濮嬭鏃讹紵杩樻槸璺焪indow涓€鏍锋牴鎹郴缁熸椂闂村垝鍒嗙殑锛熷鏋滄槸鍚庤€咃紝璇烽棶鎬庝箞閰嶇疆鎵嶈兘浠巗tart寮€濮嬭鏃讹紵

83銆丗link鑱氬悎缁撴灉鐩存帴鍐橫ysql鐨勫箓绛夋€ц璁￠棶棰?/span>

84銆丗link job鎵撳紑浜哻heckpoint锛岀敤鐨剅ocksdb锛岄€氳繃瑙傚療hdfs涓奵heckpoint鐩綍锛屼负鍟ョ畻鍓湰鎬婚噺浼氭毚澧炵垎鍑?/span>

85銆乕Flink 鎻愪氦浠诲姟鐨?jar鍖呭彲浠ユ寚瀹氳矾寰勪负 HDFS 涓婄殑鍚梋()

86銆佸湪flink web Ui涓婃彁浜ょ殑浠诲姟锛岃缃殑骞惰搴︿负2锛宖link鏄痵tand alone閮ㄧ讲鐨勩€備袱涓换鍔￠兘姝ｅ父鐨勮繍琛屼簡鍑犲ぉ浜嗭紝浠婂ぉ鏈変釜鍦版柟閫昏緫闇€瑕佷慨鏀癸紝浜庢槸灏嗕换鍔ancel鎺?鍦ㄥ懡浠よcancel涔熻瘯浜?锛岀粨鏋渢askmanger鎸傛帀浜嗕竴涓妭鐐广€傚悗鏉ョ敤鍏朵粬浠诲姟璇曚簡锛屼篃鍚屾牱浼氬鑷磋妭鐐规寕鎺?/span>

87銆佷竴涓厤缃姩鎬佹洿鏂扮殑闂鎶樿吘濂戒箙锛堥厤缃敤涓潤鎬佺殑map鍙橀噺瀛樼潃锛屾湁涓嚎绋嬪畾鏃跺幓鏁版嵁搴撴崬鏁版嵁鐒跺悗瀛樺湪杩欎釜map閲岄潰鏇存柊涓€鎶婏級锛屾湰鍦?idea 璋冭瘯娌￠棶棰橈紝闆嗙兢閮ㄧ讲灏变竴鐩存姤 绌烘寚閽堝紓甯搞€備笅娓哥殑绠楀瓙浣跨敤杩欎釜闈欐€佸彉閲弇ap鍘籫et key鍦ㄩ泦缇ゆā寮忎笅浼氬嚭鐜拌繖涓┖鎸囬拡寮傚父锛屼及璁″氨鏄嬁涓嶅埌 map

88銆佹壒閲忓啓鍏ySQL锛屽畬鎴怘Base鎵归噺鍐欏叆

89銆佺敤flink娓呮礂鏁版嵁锛屽叾涓璁块棶redis锛屾牴鎹畆edis鐨勭粨鏋滄潵鍐冲畾鏄惁鎶婃暟鎹紶閫掑埌涓嬫祦锛岃繖鏈夊彲鑳藉疄鐜板悧锛?/span>

90銆佺洃鎺ч〉闈㈡祦澶勭悊鐨勬椂鍊欒繖涓彂閫佸拰鎺ユ敹瀛楄妭涓?銆?/span>

91銆乻ink鍒癕ySQL锛屽鏋滅洿鎺ョ敤idea鐨勮瘽鍙互杩愯锛屽苟涓旀垚鍔燂紝澶уぇ鐨勪唬鐮佷笂闈㈢敤鐨凢linkKafkaConsumer010锛岃€屾垜鐨凢link鐗堟湰涓?.7锛宬afka鐗堟湰涓?.12锛屾墍浠ュ綋鎴戠敤FlinkKafkaConsumer010灏辨湁闂锛屼簬鏄敼涓?FlinkKafkaConsumer灏卞彲浠ョ洿鎺ュ湪idea瀹屾垚sink鍒癕ySQL锛屼絾鏄负浣曞綋鎴戞妸璇ョ▼搴忔墦鎴怞ar鍖咃紝鍘昏繍琛岀殑鏃跺€欙紝灏辨槸鎶linkKafkaConsumer鎵句笉鍒板憿

92銆丼ocketTextStreamWordCount涓緭鍏ヤ腑鏂囩粺璁′笉鍑烘潵锛岃闂繖涓€庝箞瑙ｅ喅锛屾垜鐚滄祴搴旇鏄渶瑕佷慨鏀逛竴涓嬩唬鐮侊紝搴旇鏄繖涓緥瀛愰粯璁ょ粺璁¤嫳鏂?/span>

93銆?Flink 搴旂敤绋嬪簭鏈湴 ide 閲岄潰杩愯鐨勬椂鍊欏苟琛屽害鏄€庝箞绠楃殑锛?/span>

94銆?nbsp;璇烽棶涓媐link涓浜庣獥鍙ｇ殑鍏ㄩ噺鑱氬悎鏈塧pply鍜宲rocess涓ょ 浠栦滑鏈夊暐鍖哄埆鍛?/span>

95銆佷笉鐭ラ亾澶уぇ鐔熸倝Hbase涓嶏紝鎴戞兂鐩存帴鍦℉base涓煡璇㈡煇涓€鍒楁暟鎹紝鍥犱负鏈夐噸澶嶆暟鎹紝鎵€浠ユ兂浣跨敤distinct缁熻瀹為檯鏁版嵁閲忥紝璇烽棶Hbase涓湁娌℃湁绫讳技浜巗ql鐨刣istinct鍏抽敭瀛椼€傚鏋滄病鏈夛紝鎯冲疄鐜拌繖绉嶅彲浠ヤ笉锛?/span>

96銆?nbsp;鏉ュ垎鏋愪竴涓嬬幇鍦‵link,Kafka鏂归潰鐨勫氨涓氬舰鍔匡紝浠ュ強鍑嗗灏变笟璇ュ浣曞噯澶囩殑杩欐柟闈㈠唴瀹瑰憿锛?/span>

97銆?nbsp;澶т浆鐭ラ亾flink鐨刣ataStream鍙互杞崲涓篸ataSet鍚楋紵鍥犱负鏁版嵁闇€瑕?1鍒嗛挓涓€涓壒娆¤绠椾簲鍏釜鎸囨爣锛屽苟涓旀秹鍙婂ソ鍑犳reduce锛岃绠楃殑鎸囨爣涔嬮棿鏈夎仈绯伙紝鐢⊿tream鍗′綇浜嗐€?/span>

98銆?.濡備綍鍦ㄥ悓涓€绐楀彛鍐呭疄鐜板娆＄殑鑱氬悎锛屾瘮濡傚儚spark涓殑杩欐牱2.澶氫釜瀹炴椂娴佺殑jion鍙互鐢╳indow鏉ュ鐞嗕竴鎵规鐨勬暟鎹悧锛?/span>

99銆佸啓鐨勬壒澶勭悊鐨勫姛鑳斤紝鐜板湪鏈満璺戞槸娌￠棶棰樼殑锛屽氨鏄湪linux闆嗙兢涓婂嚭鐜颁簡闂锛屽氨鏄笉鐭ラ亾濡傛灉閫氳繃鏈湴璋冪敤杩滅▼jar鍖呯劧鍚庝紶鍙傛暟鍜屾嬁鍒扮粨鏋滃弬鏁拌繑鍥炴湰鏈?/span>

100銆佹垜鐢╯tandalone寮€鍚竴涓猣link闆嗙兢锛屼笂浼爁link瀹樻柟鐢ㄤ緥Socket Window WordCount鍋氭祴璇曪紝寮€鍚袱涓猵arallelism鑳芥甯歌繍琛岋紝浣嗘槸寮€鍚?涓猵arallelism鍚庡嚭鐜伴敊璇?/span>

101銆?nbsp;鏈変娇鐢ˋssignerWithPunctuatedWatermarks 鐨勬渚婦emo鍚楋紵缃戜笂鎵句簡閮芥槸AssignerWithPeriodicWatermarks鐨勶紝涓嶇煡閬撳叿浣撴€庝箞浣跨敤锛?/span>

102銆?nbsp;鏈変竴涓猟atastream(浠庢枃浠惰鍙栫殑)锛岀劧鍚庢垜鐢╢link sql杩涜璁＄畻锛岃繖涓猻ql鏄竴涓姞鎬荤殑杩愮畻锛岀劧鍚庨€氳繃retractStreamTableSink鍙互鎶婃枃浠跺仛sql鐨勭粨鏋滆緭鍑哄埌鏂囦欢鍚楋紵杩欎釜杈撳嚭鍒版枃浠剁殑鎺ュ彛鏄敤浠€涔堝憿锛?/span>

103銆?nbsp;涓哄暐split杩欎釜娴佽缃负杩囨湡鐨?/span>

104銆?nbsp;闇€瑕佷娇鐢╢link table鐨勬按鍗版満鍒舵帶鍒舵椂闂寸殑涔卞簭闂锛岃繖绉嶅満鏅笅鎴戝氨浣跨敤姘村嵃+绐楀彛浜嗭紝鎴戠幇鍦ㄥ啓鐨刣emo閬囧埌浜嗛棶棰橈紝灏辨槸鍦ㄦ妸瑙﹀彂璁＄畻鐨勭獥鍙able锛圵indowedTable锛夎浆鎹㈡垚table杩涜sql鎿嶄綔鏃跺彂鐜扮獥鍙ｄ腑鐨勬暟鎹繕鏄贡搴忕殑锛屾槸涓嶆槸flink table鐨刉indowedTable涓嶆敮鎸佹按鍗扮獥鍙ｈ浆table-sql鐨勫姛鑳?/span>

105銆?nbsp;Flink 瀵?SQL 鐨勯噸瑙嗘€?/span>

106銆?flink job鎵撳紑浜哻heckpoint锛屼换鍔¤窇浜嗗嚑涓皬鏃跺悗灏卞嚭鐜颁笅闈㈢殑閿欙紝鎴浘鏄墦鍑烘潵鐨勬棩蹇楋紝鏈変釜OOM锛屽張閬囧埌杩囩殑娌★紵

107銆?nbsp;鏈湴娴嬭瘯鏄湁鏁版嵁鐨勶紝涔嬪墠璇ヤ换鍔℃斁鍦ㄩ泦缇や篃鏄湁鏁版嵁鐨勶紝鍙兘鎻愪氦杩囧娆★紝鐜板湪璇讳笉鍒版暟鎹簡 group id 涔熸崲杩囦簡锛?鍙兘閲嶅惎闆嗙兢瑙ｅ喅涔堬紵

108銆佷娇鐢╢link娓呮礂鏁版嵁瀛樺埌es涓紝鐩存帴鍦╢latmap涓澶勭悊鍑烘潵鐨勬暟鎹敤es鑷繁鐨凜lientInterface绫荤洿鎺ュ皢鏁版嵁瀛樺叆es褰撲腑锛屼笉璧皊ink锛岃繖鏍风殑澶勭悊閫昏緫鏄笉鏄細鏈夐棶棰樸€?/span>

108銆?flink浠巏afka鎷挎暟鎹紙鍗冲閲忔暟鎹級涓庡瓨閲忔暟鎹繘琛屽唴瀛樿仛鍚堢殑闇€姹傦紝鐜板湪鏈変竴涓柟妗堝氨鏄▼搴忓惎鍔ㄧ殑鏃跺€欏厛鐢╢link table灏嗗瓨閲忔暟鎹姞杞藉埌鍐呭瓨涓垱寤簍able涓紝鐒跺悗灏唖tream鐨勫閲忔暟鎹笌table鐨勬暟鎹繘琛屽叧鑱旇仛鍚堝悗杈撳嚭缁撴潫锛屼笉鐭ラ亾杩欑鏂规鍙涔堛€傜洰鍓嶄釜浜鸿涓烘湁涓や釜涓昏闂锛?鏄閲忔暟鎹畇tream杞寲鎴恆ppend table鍚庝笉鐭ラ亾鑳戒笌瀛橀噺鐨則able鍏宠仈鑱氬悎涓嶏紝2鏄仛鍚堝悗杈撳嚭鐨勭粨鏋滄暟鎹槸鍚﹁繃浜庨绻侀€犳垚缃戠粶浼犺緭鍘嬪姏杩囧ぇ

109銆?nbsp;璁剧疆鏃堕棿鏃堕棿鐗规€ф湁浠€涔堝尯鍒憿, 鍒嗗埆鍦ㄤ粈涔堝満鏅笅浣跨敤鍛?涓ょ璁剧疆鏃堕棿寤惰繜鏈変粈涔堝尯鍒憿 , 鍒嗗埆鍦ㄤ粈涔堝満鏅笅浣跨敤

110銆?flink浠巖abbitmq涓鍙栨暟鎹紝璁剧疆浜唕abbitmq鐨凜orrelationDataId鍜宑heckpoint涓篍XACTLY_ONCE锛涘鏋渇link瀹屾垚涓€娆heckpoint鍚庯紝鍦ㄨ繖娆heckpoint涔嬪墠娑堣垂鐨勬暟鎹兘浼氫粠mq涓垹闄ゃ€傚鏋滄煇娆link鍋滄満鏇存柊锛岄偅灏变細鍑虹幇mq涓殑涓€浜涙暟鎹秷璐逛絾鏄浜嶶nacked鐘舵€併€傚湪flink鍙堥噸鏂板紑鍚悗杩欐壒鏁版嵁鍙堜細閲嶆柊娑堣垂銆傞偅杩欐牱鏄笉鏄氨涓嶈兘淇濊瘉EXACTLY_ONCE浜?/span>

111銆?. 鍦‵link checkpoint 涓? 鍍?operator鐨勭姸鎬佷俊鎭?鏄湪璁剧疆浜哻heckpoint 涔嬪悗鑷姩鐨勮繘琛屽揩鐓у悧 ?2. 涓婇潰杩欎釜鍜屾垜浠墜鍔ㄥ瓨鍌ㄧ殑 Keyed State 杩涜蹇収(杩欎釜搴旇鏄閲忓揩鐓?

112銆佺幇鍦ㄦ湁涓疄鏃跺晢鍝佹暟锛屼氦鏄撻杩欑缁熻闇€姹傦紝鎵撶畻鐢?flink浠巏afka璇诲彇binglog鏃ュ織杩涜璁＄畻锛屼絾binglog娑夊強鍒癷nsert鍜寀pdate杩欑鎿嶄綔鏃?鎬庝箞澶勭悊鎵嶈兘缁熻鍑嗙‘锛岄伩鍏嶉偅绉嶉噸澶嶈绠楃殑闂锛?/span>

113銆佹垜杩欒竟鐢╢link鍋氬疄鏃剁洃鎺э紝鍔熻兘寰堢畝鍗曪紝灏辨槸姣忔潯娑堟伅鍋歬eyby鐒跺悗涓夊垎閽熺獥鍙ｏ紝鐒跺悗鍋氫簺鍘婚噸鎿嶄綔锛岃Е鍙戦槇鍊煎垯鎶ヨ锛岀幇鍦ㄩ棶棰樻槸鍚屼竴涓椂闂寸獥鍙ｅ悓涓€涓汉鐨勫憡璀︿細瑙﹀彂涓ゆ锛岄泦缇ゆ槸涓夊彴鏈哄櫒锛宻tandalone cluster锛屽垵姝ョ粨鏋滄槸涓変釜绠楀瓙閲屾湁涓や釜鏀跺埌浜嗗悓鏍风殑鏁版嵁

114銆佸湪浣跨敤WaterMark鐨勬椂鍊欙紝榛樿鏄瘡200ms鍘昏缃竴娆atermark锛岄偅涔堟瘡涓猼askmanager涔嬮棿锛岀敱浜庡緱鍒扮殑鏁版嵁涓嶅悓锛屾墍浠ュ線寰€浜х敓鐨勬渶澶х殑watermark涓嶅悓銆傞偅涔堣繖涓椂鍊欙紝鏄悇涓猼askmanager骞挎挱杩欎釜watermark锛屽緱鍒板叏灞€鐨勬渶澶х殑watermark锛岃繕鏄鍚勪釜taskmanager閮藉悇鑷敤鑷繁鐨剋atermark銆備富瑕佹病鐪嬪埌骞挎挱watermark鐨勬簮鐮併€備笉鐭ラ亾鏄嚜宸辫瀵熶笉浠旂粏杩樻槸灏辨槸娌℃湁骞挎挱杩欎釜鍙橀噺銆?/span>

115銆佺幇鍦ㄩ亣鍒颁竴涓渶姹傦紝闇€瑕佸湪job鍐呴儴瀹氭椂鍘昏鍙杛edis鐨勪俊鎭紝鎯宠鏁檉link鑳藉疄鐜板儚鏅€氱▼搴忛偅鏍风殑瀹氭椂浠诲姟鍚楋紵

116銆佹湁涓Е鍙戜簨浠跺紑濮嬭仛鍚堬紝绛夊埌鏁伴噺瓒冲锛屾垨鑰呰秴鏃跺垯sink鎺╩q 鐜 flink 1.6 鐢ㄤ簡mapState 璁板綍瑙﹀彂浜嬩欢 1 鏁版嵁瓒冲杩欎釜OK 2 瓒呮椂state ttl 1.6鏀寔锛屼絾鏄棶棰樻潵浜嗭紝濡備綍鍦ㄨ秴鏃舵椂鍊欏鍔犺嚜瀹氫箟澶勭悊锛?/span>

117銆佽闂甶mpala杩欑mpp鏋舵瀯鐨剆ql寮曟搸锛屼负浠€涔堢ǔ瀹氭€ф瘮杈冨樊鍛紵

118銆亀atermark璺熷苟琛屽害鐩稿叧涓嶆槸锛岃繃浜庡叏灞€浜嗭紝鏈熸湜鏄痥eyby涔嬪悗鍐嶉拡瀵规瘡涓猭eyed stream 鎵搘atermark锛岃繖涓湁浠€涔堝ソ鐨勫疄璺靛憿锛?/span>

119銆佽闂鏋滄妸涓€涓枃浠剁殑鍐呭璇诲彇鎴恉atastream鍜宒ataset锛屾湁浠€涔堝尯鍒悧锛燂紵浠栦滑閮芥槸涓€鏉℃暟鎹竴鏉℃暟鎹殑琚鍙栧悧锛?/span>

120銆佹湁娌℃湁kylin鐩稿叧鐨勮祫鏂欙紝鎴栬€呰皟浼樼殑缁忛獙锛?/span>

121銆乫link鍏堜粠jdbc璇诲彇閰嶇疆琛ㄥ埌娴佷腑锛屽彟澶栦粠kafka涓柊澧炴垨鑰呬慨鏀硅繖涓厤缃紝杩欎釜鍦烘櫙鎬庝箞鎶婁袱涓祦涓€浠介厤缃祦锛熸垜鐢ㄧ殑connect,鎺ョ潃鍙戜笉鎴愬箍鎾彉閲忥紝鍐嶅拰瀹炰綋娴佸悎骞讹紝浣嗗湪鍚堝苟鏃舵姤Exception in thread "main" java.lang.IllegalArgumentException

122銆丗link exactly-once锛宬afka鐗堟湰涓?.11.0 锛宻ink鍩轰簬FlinkKafkaProducer011 姣忎簲鍒嗛挓涓€娆heckpoint锛屼絾鏄痗heckpoint寮€濮嬪悗绯荤粺鐩存帴鍗℃锛宎t-lease-once 涓€鍒嗛挓鑳藉畬鎴愮殑checkpoint锛?鐜板湪鍗佸垎閽熸棤娉曞畬鎴愭病杩涘害杩樻槸0锛?涓嶇煡閬撳摢閲屽崱浣忎簡

123銆乫link鐨勭姸鎬佹槸榛樿瀛樺湪浜庡唴瀛樼殑(涔熷彲浠ヨ缃负rocksdb鎴杊dfs)锛岃€宑heckpoint閲岄潰鏄畾鏃跺瓨鏀炬煇涓椂鍒荤殑鐘舵€佷俊鎭紝鍙互璁剧疆hdfs鎴杛ocksdb鏄繖鏍风悊瑙ｇ殑鍚楋紵

124銆丗link寮傛IO涓紝涓嬪浘杩欎袱绉嶆湁浠€涔堝尯鍒紵涓哄暐瑕佸姞 CompletableFuture.supplyAsync锛屼笉澶槑鐧斤紵

125銆乫link鐨勭姸鎬佹槸榛樿瀛樺湪浜庡唴瀛樼殑(涔熷彲浠ヨ缃负rocksdb鎴杊dfs)锛岃€宑heckpoint閲岄潰鏄畾鏃跺瓨鏀炬煇涓椂鍒荤殑鐘舵€佷俊鎭紝鍙互璁剧疆hdfs鎴杛ocksdb鏄繖鏍风悊瑙ｇ殑鍚楋紵

126銆佹湁涓绠楀満鏅紝浠巏afka娑堣垂涓や釜鏁版嵁婧愶紝涓や釜鏁版嵁缁撴瀯閮芥湁鏃堕棿娈垫蹇碉紝璁＄畻闇€瑕佸仛鐨勬槸鍖归厤涓や釜鏃堕棿娈碉紝鍖归厤鍒颁簡锛屽氨鐢熸垚涓€鏉℃柊鐨勮褰曘€傝闂娇鐢ㄥ摢涓伐鍏锋洿鍚堥€傦紝flink table杩樻槸cep锛熻澶х鎸囩偣涓€涓?鎴戣繖杈逛箣鍓嶇殑鍋氭硶锛屽皢涓や釜鏁版嵁娴佽浆涓簍able.涓や釜table over window鍚巎oin鎴愭柊鐨勮〃銆傜粨鏋渏ob璺戜竴浼氬氨oom.

127銆佷竴涓簰鑱旂綉鍏徃锛屾垨鑰呬竴涓笟鍔＄郴缁燂紝濡傛灉鎯冲仛涓€涓叏闈㈢殑鐩戞帶瑕佹€庝箞鍋氾紵鏈変粈涔堟垚鐔熺殑鏂规鍙互鍙傝€冧氦娴佸悧锛熸湁浠€涔堟湁浠€涔堝害閲忔寚鏍囧悧锛?/span>

128銆佹€庝箞娣卞叆瀛︿範flink,鎴栬€呭叾浠栧ぇ鏁版嵁缁勪欢锛岃兘涓烘湭鏉ョ鎷涙壘涓€浠藉ぇ鏁版嵁鐩稿叧锛堣绠楁柟鍚戯級鐨勫伐浣滃鍔犺嚜宸辩殑绔炰簤鍔涳紵

129銆乷ppo鐨勫疄鏃舵暟浠擄紝鍏朵腑鏄庣粏灞傚拰姹囨€诲眰閮藉湪kafka涓紝浠栦滑鐨勫叧绯诲簱鐨勫疄鏃舵暟鎹篃鎶藉彇鍒発afka鐨刼ds锛岄偅涔堝湪鏋勫缓鏁颁粨鐨勶紝闇€瑕乯oin 涓夊洓涓ぇ涓氬姟琛紝涓氬姟琛ㄤ細鍙樺寲锛岄偅涔堟槸澶х殑涓氬姟琛ㄦ槸浠巏afka鐨刼ds璇诲彇鍚楋紵瀹炴椂鏁颁粨锛屽涓ぇ琛╦oin鍙互鍚?/span>

130銆乀uple绫诲瀷鏈変粈涔堟柟娉曡浆鎹㈡垚json瀛楃涓插悧锛熺幇鍦ㄧ殑鍦烘櫙鏄紝缁撴灉鍦ㄥ瓨鍌ㄥ埌sink涓椂甯屾湜瀛樼殑鏄痡son瀛楃涓诧紝杩欐牱搴旂敤绋嬪簭鑾峰彇鏁版嵁姣旇緝濂借浆鎹竴鐐广€傚鏋淭uple涓嶅ソ杞崲json瀛楃涓诧紝閭ｄ箞搴旇浠ヤ粈涔堟暟鎹牸寮忓瓨鍌ㄥ埌sink涓?/span>

140銆佺鍒扮鐨勬暟鎹繚璇侊紝鏄惁鎰忓懗鐫€涓棿澶勭悊绋嬪簭涓柇锛屼篃涓嶄細閫犳垚璇ユ壒娆″鐞嗗け璐ョ殑娑堟伅涓㈠け锛屽鐞嗙▼搴忛噸鏂板惎鍔ㄤ箣鍚庯紝浼氬啀娆″鐞嗕笂娆℃湭澶勭悊鐨勬秷鎭?/span>

141銆佸叧浜巉link datastream window鐩稿叧鐨勩€傛瘮濡傛垜鐜板湪浣跨敤婊氬姩绐楀彛锛岀粺璁′竴鍛ㄥ唴鍘婚噸鐢ㄦ埛鎸囨爣锛屾寜鐓ф甯竪atermark瑙﹀彂璁＄畻锛岄渶瑕佺瓑鍒板綋鍓嶅懆鐨剋indow鍒拌揪window鐨別ndtime鏃讹紝鎵嶄細瑙﹀彂锛岃繖鏍锋寚鏍囦竴鍛ㄥ悗鎵嶈兘浜у嚭缁撴灉銆傛垜鑳戒笉鑳藉疄鐜颁竴灏忔椂瑙﹀彂涓€娆¤绠楋紝姣忔缁熻鎴鍒板綋鍓嶆椂闂达紝window涓墍鏈夊埌杈惧厓绱犵殑鍘婚噸鏁伴噺銆?/span>

142銆丗LIP-16 Loop Fault Tolerance 鏄鐜板湪鐨刢heckpoint鏈哄埗鏃犳硶鍦╯tream loop鐨勬椂鍊欏閿欏悧锛熺幇鍦ㄨ繖涓棶棰樿В鍐充簡娌℃湁鍛€锛?/span>

143銆佺幇鍦ㄧ殑闇€姹傛槸锛岀粺璁″悇涓猭ey鐨勪粖鏃ョ疮璁″€硷紝涓€鍒嗛挓杈撳嚭涓€娆°€傚锛屽悇涓敤鎴蜂粖鏃ョ疮璁＄偣鍑绘鏁般€傝繖绉嶉渶姹傜敤datastream杩樻槸table API鏂逛究鐐癸紵

144銆佹湰鍦癷dea鍙互璺戠殑宸ョ▼锛屾斁鍦╯tandalone闆嗙兢涓婏紝鎬绘姤閿欙紝鎶ラ敊鎴浘濡備笅锛屽ぇ浣闂繖鏄暐鍘熷洜

145銆佹瘮濡傜幇鍦ㄧ敤k8s璧蜂簡涓€涓猣link闆嗙兢锛岃繖鏃跺€欐暟鎹簮kafka鎴栬€卙dfs浼氬湪鍚屼竴涓泦缇や笂鍚楋紝杩樻槸浼氬崟鐙啀璧蜂竴涓猦dfs/kafka闆嗙兢

146銆乫link kafka sink 鐨凢linkFixedPartitioner 鍒嗛厤绛栫暐锛屽湪骞惰搴﹀皬浜巘opic鐨刾artitions鏃讹紝涓€涓苟琛屽疄渚嬪浐瀹氱殑鍐欐秷鎭埌鍥哄畾鐨勪竴涓猵artition锛岄偅涔堝氨鏈変竴浜沺artition娌℃暟鎹啓杩涘幓锛?/span>

147銆佸熀浜庝簨浠舵椂闂达紝姣忎簲鍒嗛挓涓€涓獥鍙ｏ紝浜旂閽熸粦鍔ㄤ竴娆★紝鍚屾椂watermark鐨勬椂闂村悓鏍锋槸鍩轰簬浜嬩欢浜嬩欢鏃堕棿鐨勶紝寤惰繜璁句负1鍒嗛挓锛屽亣濡傛暟鎹祦浠?2锛?0寮€濮嬶紝濡傛灉12锛?7-12锛?9鏈熼棿娌℃湁浜х敓浠讳綍涓€鏉℃暟鎹紝鍗冲湪12锛?7-12锛?9杩欐闂寸殑鏁版嵁娴佹儏鍐典负路路路路 锛?2锛?7:00锛寈xx锛?(12:09:00,xxx)路路路路路路锛岄偅涔堢獥鍙12:02:05-12:07:05]锛孾12:02:10-12:07:10]绛夊嚑涓獥鍙ｇ殑璁＄畻鏄惁鎰忓懗鐫€鍙湁绛夊埌锛?2锛?9锛?0鐨勬暟鎹埌杈句箣鍚庢墠浼氳Е鍙?/span>

148銆佷娇鐢╢link1.7锛屽綋娑堣垂鍒版煇鏉℃秷鎭?protobuf鏍煎紡)锛屾姤Caused by: org.apache.kafka.common.KafkaException: Record batch for partition Notify-18 at offset 1803009 is invalid, cause: Record is corrupt 杩欎釜寮傚父銆傚浣曡缃烦杩囧凡鎹熷潖鐨勬秷鎭户缁秷璐逛笅涓€鏉℃潵淇濊瘉涓氬姟涓嶇粓鏂紵鎴戠湅浜嗗畼缃慿afka connectors閭ｉ噷锛岃鍦―eserializationSchema.deserialize(鈥?鏂规硶涓繑鍥瀗ull锛宖link灏变細璺宠繃杩欐潯娑堟伅锛岀劧鑰屼緷鏃ф姤杩欎釜寮傚父

149銆佹槸鍚﹀彲浠ユ娊绌烘€荤粨涓€绡嘑link 鐨?watermark 鐨勫師鐞嗘渚嬶紵涓€鐩存病鎼炴槑鐧藉熀浜庝簨浠舵椂闂村鐞嗘椂鐨勬暟鎹贡搴忓拰鏁版嵁杩熷埌搴曞拫鍥炰簨

150銆乫link涓璻pc閫氫俊鐨勫師鐞嗭紝涓庡嚑涓被鐨勮瑙ｏ紝鏈夋病鏈夌郴缁熻缁嗙殑鏂囩珷鏍凤紝濡傛湁姹傚垎浜紝璋㈣阿

151銆丗link涓浣曚娇鐢ㄥ熀浜庝簨浠舵椂闂村鐞嗭紝浣嗘槸鍙堜笉浣跨敤Watermarks? 鎴戝湪浼氳瘽绐楀彛涓娇鐢ㄩ亣鍒颁竴浜涢棶棰橈紝鍥句竴鏄熀浜庡鐞嗘椂闂寸殑锛屾祴璇曠粨鏋渟ession鏄熀浜巏eyby(鐢ㄦ埛)鐨勶紝鍥句簩鏄熀浜庝簨浠舵椂闂寸殑锛屼笉鐭ラ亾鏄垜鐢ㄦ硶涓嶅杩樻槸鎬庝箞鐨勶紝娴嬭瘯缁撴灉鍙戠幇骞朵笉鏄熀浜巏eyby(鐢ㄦ埛鐨?锛岃€屾槸鍏ㄥ眬鐨剆ession銆備笉鐭ラ亾鎬庝箞淇敼锛?/span>

152銆乫link瀹炴椂璁＄畻骞冲彴锛寉arn妯″紡鏃ュ織鏀堕泦鎬庝箞鍋氾紝涓轰粈涔堜細checkpoint澶辫触锛屾姤璀﹀鐞嗭紝鍚庨渶瑕佸仛浠€涔堝悧锛焜ob鐩戞帶鎬庝箞鍋?/span>

153銆佹湁flink涓巎storm鐨勫湪涓嶅悓搴旂敤鍦烘櫙涓? 鎬ц兘姣旇緝鐨勬暟鎹悧? 浠庣綉缁滀笂鑳芥壘澶ч儴鍒嗛兘鏄痜link涓巗torm鐨勬瘮杈? 鍦╦storm瀹樼綉涓婃湁涓€浠芥瘮杈冪殑鍥捐〃, 鎰熻鍙傝€冩剰涔変笉澶? 搴旇鏄瘮杈冩棭鐨刦link鐗堟湰.

154銆佷负浠€涔堜娇鐢⊿essionWindows.withGap绐楀彛鐨勮瘽锛孲tate瀛樹笉浜嗕笢瑗垮憖锛屾瘡娆″姞1 锛屾嬁鍑烘潵閮芥槸null, 鎴戞崲鎴?TimeWindow灏辨病闂銆?/span>

155銆佽闂竴涓嬶紝flink datastream娴佸鐞嗘€庝箞缁熻鍘婚噸鎸囨爣锛熷畼鏂规枃妗ｄ腑鍙湅鍒版壒澶勭悊鏈塪istinct姒傚康銆?/span>

156銆佸ソ鍏ㄧ殑涓€绡囨枃绔狅紝瀵规瘮鍒嗘瀽 Flink锛孲park Streaming锛孲torm 妗嗘灦

157銆佸叧浜?structured_streaming 鐨?paper

158銆亃ookeeper闆嗙兢鍒囨崲棰嗗浜嗭紝flink闆嗙兢椤圭洰閲嶅惎浜嗗氨娌℃湁鏁版嵁鐨勮緭鍏ュ拰杈撳嚭浜嗭紝杩欎釜璇ヤ粠鍝柟闈㈠叆鎵嬭В鍐筹紵

159銆佹垜鎯宠鏁欎笅datastream鎬庝箞鍜岄潤鎬佹暟鎹甹oin鍛?/span>

160銆佹椂閽熼棶棰樺鑷存敹鍒颁簡鏄庡ぉ鐨勬暟鎹紝杩欐椂鍊欐湁浠€涔堟瘮杈冨ソ鐨勫鐞嗘柟娉曪紵鐪嬪埌鏈変汉璁剧疆涓€涓渶澶х殑璺宠穬闃堝€硷紝濡傛灉褰撳墠鏁版嵁鏃堕棿 - 鍘嗗彶鏈€澶ф椂闂?瓒呰繃闃堝€煎氨涓嶆洿鏂般€傚浣曞悎鐞嗙殑璁捐姘村嵃锛屾湁娌℃湁涓€浜涚粡楠屽憿锛?/span>

161銆佸ぇ浣滑flink鎬庝箞瀹氭椂鏌ヨ鏁版嵁搴擄紵

162銆佺幇鍦ㄦ垜浠叕鍙告湁涓兂娉曪紝灏辨槸鎻愪緵涓€涓〉闈紝鍦ㄩ〉闈笂閫夋嫨source sink 濉啓涓妔ql璇彞锛岀劧鍚庡悗鍙扮敓鎴愪竴涓猣link鐨勪綔涓氾紝鐒跺悗鎻愪氦鍒伴泦缇ゃ€傚姛鑳芥湁鐐圭被浼间簬鍗庝负鐨勬暟鎹腑鍙帮紝灏辨槸椤甸潰鍌荤摐寮忔搷浣溿€傚悗鍙拌兘鑷姩鏍规嵁鐩稿簲閰嶇疆寰楀埌缁撴灉銆傝闂嫎浣犵殑浜嗚В锛屽彲浠ュ疄鐜板悧锛熷浣曞疄鐜帮紵鏈変粈涔堝ソ鐨勬€濊矾銆傜幇鍦ㄦ垜鏃犱粠涓嬫墜

163銆佽鏁欎竴涓?flink on yarn 鐨?ha鏈哄埗

164銆佸湪涓€鑸殑娴佸鐞嗕互鍙奵ep, 閮藉彲浠ュ浜巈venttime璁剧疆watermark, 鏈夋椂鍙兘闇€瑕佽缃浉瀵瑰ぇ涓€鐐圭殑鍊? 杩欏唴瀛樺帇鍔涘氨姣旇緝澶? 鏈夋病鏈夊姙娉曚笉搴旂敤jvm涓殑鍐呭瓨, 鑰岀敤鍫嗗鍐呭瓨, 鎴栬€呭叾浠栫紦瀛? 鏈€濂芥湁cache鏈哄埗, 杩欐牱鍙互搴斿澶ф祦閲忕殑宄板€?

165銆佽鏁欎竴涓猣link sql鐨勯棶棰樸€傛垜鏈変袱涓仛鍚堝悗鐨勬祦琛ˋ鍜孊锛孉鍜孊join寰楀埌C琛ㄣ€傚湪璁剧疆state TTL 鐨勬椂鍊欐槸鐩存帴瀵笴琛ㄨ缃繕鏄紝瀵笰琛ㄥ拰B琛ㄨ缃瘮杈冨ソ锛?/span>

166銆乻park鏀瑰啓涓篺link锛屼細涓嶄細寰堝鏉傦紝杩樻湁杩欎袱鑰呭湪SQL鏂归潰鐨勬敮鎸佸樊鍒ぇ鍚楋紵

167銆佽闂甪link allowedLateness瀵艰嚧绐楀彛琚娆ire锛屾渶缁堟暟鎹噸澶嶆秷璐癸紝杩欑闂鎬庝箞澶勭悊锛屾暟鎹槸鍐欏埌es涓?/span>

168銆佽缃畉askmanager.numberOfTaskSlots: 4鐨勬椂鍊欐病鏈夐棶棰橈紝浣嗘槸cpu娌℃湁鍘嬩笂鍘伙紝鍙敤浜?0%宸﹀彸锛屼簬鏄缃簡taskmanager.numberOfTaskSlots: 8锛屼絾鏄氨鎶ラ敊璇壘涓嶅埌鍏朵腑涓€涓嚜瀹氫箟鐨勭被锛岀劧鍚巏afka鏁版嵁灏变笉娑堣垂浜嗐€備负浠€涔堬紵cpu鍒板灏戝悎閫傦紵slot鏄笉鏄拰cpu鏁伴噺涓€鑷存槸鏈€浣抽厤缃紵kafka鍒嗗尯鏁板灏戝悎閫傦紝鏄笉鏄拰slot,parallesim涓€鑷存渶浣筹紵

169銆侀渶姹傛槸鏍规嵁姣忔潯鏃ュ織鍒囧垎鍑洪渶瑕?涓瓧娈碉紝鏈変簲涓寚鏍囧啀鏍规嵁9涓瓧娈电殑涓嶅悓缁勫悎鍘诲仛璁＄畻銆傜涓€涓柟娉曟槸锛氭垜鐩墠鍋氭硶鏄垏鍒嗙殑9涓瓧娈靛紑5鍒嗛挓澶у皬1鍒嗛挓璁＄畻涓€娆＄殑婊戝姩绐楀彛绐楀彛锛岃繘琛屼竴娆educe鍘婚噸锛岀劧鍚庡啀map鍙栧嚭闇€瑕佺殑瀛楁锛岀劧鍚庤繃婊ゅ啀寮€5鍒嗛挓澶у皬1鍒嗛挓璁＄畻涓€娆＄殑婊戝姩绐楀彛绐楀彛杩涜璁＄畻淇濆瓨缁撴灉锛岃繖涓€濊矾閬囧埌鐨勯棶棰樻槸涓婁竴涓粦鍔ㄧ獥鍙ｄ細姣忎竴鍒嗛挓浼氳绠?鍒嗛挓鏁版嵁锛屽埌绗簩涓獥鍙ｅ垝瀹氱殑5鍒嗛挓鑼冨洿鐨勬暟鎹細鏈夊ソ澶氶噸澶嶏紝杩欎釜鎬濊矾浼氶€犳垚鏁版嵁閲嶅銆傜浜屼釜鏂规硶鏄細鍒囧垎鐨?涓瓧娈靛紑5鍒嗛挓澶у皬1鍒嗛挓璁＄畻涓€娆＄殑婊戝姩绐楀彛绐楀彛锛屽啀pross鏂规硶閲屽畬鎴愭墍鏈夌殑杩囨护锛岃仛鍚堣绠楋紝浣嗘槸鍐嶉珮宄版湡姣忓垎閽?00涓囨潯鏁版嵁锛岃繖涓€濊矾鎷呭績鍦ㄩ珮宄版湡flink璁＄畻涓嶈繃鏉?/span>

170銆乤,b,c涓変釜琛紝a鍜宑鏈塭venttime锛宎鍜宑鐩存帴join鍙互锛宎鍜宐 join鍚庡啀鍜宑 join 灏变細鎶ラ敊锛岃繖鏄€庝箞鍥炰簨鍛?/span>

171銆佽嚜瀹氫箟鐨剆ource鏄繖鏍风殑锛堝浘涓€鎵€绀猴級 浣跨敤鐨勬椂鍊欐槸杩欐牱鐨勶紙鍥句簩鎵€绀猴級锛屼负浠€涔堟棤璁?sum.print().setParallelism(2)锛堝浘2鎵€绀猴級鐨勫苟琛屽害璁剧疆鎴愬嚑鏈€鍚庣粨鏋滈兘鏄繖鏍风殑

172銆佸垰鎺ヨЕflink锛屽鏈夐棶鐨勪笉鍚堥€傜殑鍦版柟锛岃瑙佽皡銆?銆佷负浠€涔堣flink鏄湁鐘舵€佺殑璁＄畻锛?銆佽繖涓姸鎬佹槸浠€涔堬紵3銆佺姸鎬佸瓨鍦ㄥ摢閲?/span>

173銆佽繖杈圭敤flink 1.8.1鐨勭増鏈紝閲囩敤flink on yarn锛宧adoop鐗堟湰2.6.0銆備唬鐮佹槸涓€涓畝鍗曠殑婊氬姩绐楀彛缁熻鍑芥暟锛屼絾鍚姩鐨勬椂鍊欐姤閿欙紝濡備笅鍥剧墖銆傦紙2锛夌劧鍚庢垜鎶奻link鐗堟湰鎹㈡垚1.7.1锛岄噸鏂版彁浜ゅ埌2.6.0鐨剏arn骞冲彴锛屽氨鑳芥甯歌繍琛屼簡銆傦紙3锛夋垜浠祴璇曢泦缇adoop鐗堟湰鏄?.0锛屾垜鐢╢link 1.8.1鐗堟湰灏嗚繖涓▼搴忓啀娆℃墦鍖咃紝鎻愪氦鍒?.0鐗堟湰鐨剏arn骞冲彴锛屼篃鑳芥甯歌繍琛屻€傝矊浼兼槸flink 1.8.1鐗堟湰涓巠arn 2.6.0鐗堟湰涓嶅吋瀹归€犳垚鐨勮繖涓棶棰?/span>

174銆丼tateBackend鎴戜娇鐢ㄧ殑鏄疢emoryStateBackend锛?State鏄€庝箞閲婃斁鍐呭瓨鐨勶紝渚嬪鎴戝湪鍑芥暟涓敤ValueState瀛樺偍浜嗗巻鍙茬姸鎬佷俊鎭€備絾鏄巻鍙茬姸鎬佹暟鎹垜娌℃湁鎵嬪姩閲婃斁锛岄偅涔堢▼搴忎細鑷姩閲婃斁涔堬紵杩樻槸涓€鐩撮┗鐣欏湪鍐呭瓨涓?/span>

175銆佽闂€佸笀鏄惁鍙互鎻愪緵涓€浜汚pachebeam鐨勫涔犺祫鏂?璋㈣阿

176銆乫link 鐨?DataSet鎴栬€匘ataStream鏀寔绱㈠紩鏌ヨ浠ュ強鍒犻櫎鍚楋紝鍍弒park rdd锛屽鏋滀笉鏀寔鐨勮瘽锛岃杞崲鎴愪粈涔?/span>

177銆佸叧浜巉link鐨勭姸鎬侊紝鑳藉惁鎶婂畠褰撳仛鏁版嵁搴撲娇鐢紝绫讳技浜庡唴瀛樻暟鎹簱锛屽湪澶勭悊杩囩▼涓瓨涓氬姟鏁版嵁銆傚鏋滄槸鏁版嵁搴撳彲浠ョ畻鏄垎甯冨紡鏁版嵁搴撳悧?鏄笉鏄娇鐢╮ocksdb杩欑瀛樺偍鏂瑰紡鎵嶇畻鏄?鏀寔鐨勫崟搴撳ぇ灏忔槸涓嶆槸鍙槸璺熸湰鍦版満鍣ㄧ殑纾佺洏澶у皬鐩稿叧?濡傛灉浣跨敤纭洏瀛樺偍浼氫笉浼氭晥鐜囨€ц兘鏈夊奖鍝?/span>

178銆佹垜杩欒竟鍋氫簡涓猦ttp sink锛屾兂瑕佹壒閲忓彂閫佹暟鎹紝涓嶈繃鐜板湪鍙兘鐢ㄦ暟閲忔帶鍒跺彂閫侊紝浣嗘渶鍚庣殑鍑犱釜璁板綍娌℃硶瑙﹀彂鍙戦€佸姩浣滐紝鎯抽棶涓嬫湁娌℃湁浠€涔堝姙娉?/span>

179銆佽闂笅濡備綍鍋氬畾鏃跺幓閲嶈鏁帮紝灏辨槸鏍规嵁鏃堕棿鍒嗙獥鍙ｏ紝绐楀彛鍐呮牴鎹甶d鍘婚噸璁℃暟寰楀嚭缁撴灉锛屽璋€傝瘯浜嗕笉灏戝姙娉曪紝娌℃湁绠€鍗曠洿鎺ュ姙娉?/span>

180銆佹垜鏈変釜job浣跨敤浜唀lastic search sink. 璁剧疆浜嗘壒閲?000涓€鍐欏叆锛屼絾鏄湅es鐩戞帶鏄剧ず姣忕鍙兘鎻掑叆500鏉°€傛槸涓嶆槸bulkprocessor鐨刢urrentrequest涓?鏈夊叧

181銆佹湁docker閮ㄧ讲flink鐨勮祫鏂欏悧

182銆佸湪璇存槑KeyBy鐨凷treamGraph鎵ц杩囩▼鏃讹紝keyBy鐨処D涓哄暐鏄?锛熸牴鎹墠闈㈣锛孖D鏄竴涓潤鎬佸彉閲忥紝姣忓彇涓€娆″氨閫掑1锛屾垜瑙夊緱搴旇鏄?鍟婏紝鏄垜鐞嗚В閿欎簡鍚?/span>

183銆佹湁娌¤鍒掑嚭Execution Graph鐨勮繙鐮佽В鏋?/span>

184銆佸彲浠ュ垎浜笅鐗╃悊鎵ц鍥炬€庢牱鍒掑垎task锛屼互鍙妕ask濡備綍鎵ц锛岃繕鏈変粬浠箣闂存暟鎹浣曚紶閫掕繖鍧椾唬鐮佸槢锛?/span>

185銆丗link婧愮爜鍜岃繖涓涔犻」鐩殑缁撴瀯鍥?/span>

186銆佽闂甪link1.8锛屽浣曞仛鍒板姩鎬佸姞杞藉閮╱df-jar鍖呭憿锛?/span>

187銆佸悓涓€涓猅ask Manager涓笉鍚岀殑Slot鏄€庝箞浜や簰鐨勶紝姣斿锛歴ource澶勭悊瀹岃浼犻€掔粰map鐨勬椂鍊欙紝濡傛灉鍦ㄤ笉鍚岀殑Slot涓紝浠栦滑鐨勫唴瀛樻槸鐩镐簰闅旂锛屾槸鎬庝箞浜や簰鐨勫憿锛熸垜鐚滄槸閫氳繃搴忓垪鍖栧拰鍙嶅簭鍒楀寲瀵硅薄锛屽苟涓旈€氳繃缃戠粶鏉ヨ繘琛屼氦浜掔殑

188銆佷綘浠湁娌℃湁杩欑涓氬姟鍦烘櫙銆俧link浠巏afka閲岄潰鍙栨暟鎹紝姣忎竴鏉℃暟鎹噷闈㈡湁mongdb琛ˋ鐨刬d,杩欐椂鎴戜細鍦╩ap鐨勬椂鍊欓噰鐢╢link鐨勫紓姝O杩炴帴A琛紝鐒跺悗鏌ヨ鍑篈琛ㄧ殑瀛楁1锛屽啀鏍规嵁璇ュ瓧娈?鍙堥渶瑕佸紓姝O鍘籅琛ㄦ煡璇㈠瓧娈?锛岀劧鍚庡張鏍规嵁瀛楁2鍘籆琛ㄦ煡璇㈠瓧娈?鈥?.鍍忚繖鏍风殑涓氬姟鍦烘櫙锛屽鏋滃鏉ュ嚑绉嶉€昏緫锛屾垜搴旇鐢ㄤ粈涔堟柟妗堟渶濂藉憿

189銆佷粖澶╂湰鍦拌繍琛宖link绋嬪簭锛屾秷璐箂ocket涓殑鏁版嵁锛岃繛缁彧鑳芥秷璐逛袱鏉★紝绗笁鏉link灏辨秷璐逛笉浜嗕簡

190銆佹簮鏁版嵁缁忚繃杩囨护鍚庡垎鎴愪簡涓ゆ潯娴侊紝鐒跺悗鍐嶅垎鍒彁鍙栦簨浠舵椂闂村拰姘村嵃锛屽仛鏃堕棿绐楀彛锛屾垜娴嬭瘯鏃朵竴鏉℃祦娌℃湁鏁版嵁锛屽彟涓€鏉＄殑鏁版嵁鐪嬫棩蹇楀埌浜嗙獥鍙ｆ搷浣滈偅杈瑰氨娌¤蛋涓嬪幓锛岃矊浼肩獥鍙ｄ竴鐩存病鏈夌瓑鍒拌Е鍙?/span>

191銆佹湁鍋歠link cep鐨勫悧锛屾湁璧勬枡娌★紵

192銆侀夯鐑﹂棶涓€涓?BucketingSink璺ㄩ泦缇ゅ啓锛屽鏋滀换鍔¤繍琛屽湪hadoop A闆嗙兢锛屼粠kafka璇诲彇鏁版嵁澶勭悊鍚庡啓鍒癏adoo B闆嗙兢锛屽嵆浣挎妸core-site.xml鍜宧dfs-site.xml鎷疯礉鍒颁唬鐮乺esources涓嬶紝璺緞浣跨敤hdfs://hadoopB/xxx锛屼細鎻愮ずava.lang.RuntimeException: Error while creating FileSystem when initializing the state of the BucketingSink.锛岃法闆嗙兢鍐欒繖涓棶棰?flink涓嶆敮鎸佸悧锛?/span>

193銆佹兂鍜ㄨ涓嬶紝濡備綍瀵筬link涓殑datastream鍜宒ataset杩涜鏁版嵁閲囨牱

194銆佷竴涓猣link浣滀笟缁忓父鍙戠敓oom锛屽彲鑳芥槸浠€涔堝師鍥犲鑷寸殑銆傚鐞嗘祦绋嬪彧鏈?5+瀛楁鐨勮В鏋愶紝redis鏁版嵁璇诲彇绛夋搷浣滐紝TM閰嶇疆10g銆備笟鍔′細鍦ㄥ闂村埛鏁版嵁锛宷ps鑳芥墦鍒?500宸﹀彸~

195銆佹垜鐪嬪埌flink 1.8鐨勭姸鎬佽繃鏈熶粎鏀寔Processing Time锛岄偅涔堝鏋滄垜浣跨敤鐨勬槸Event time閭ｄ箞鐘舵€佸氨涓嶄細杩囨湡鍚?/span>

196銆佽闂垜鎯虫瘡闅斾竴灏忔椂缁熻涓€涓睘鎬т粠褰撳ぉ闆剁偣鍒板綋鍓嶆椂闂寸殑骞冲潎鍊硷紝杩欐牱鐨勬椂闂寸獥璇ュ浣曞畾涔夛紵

197銆乫link浠诲姟閲岄潰鍙嶅簭鍒楀寲涓€涓被锛屾姤ClassNotFoundException锛屽彲鏄寘閲岄潰鏄湁杩欎釜绫荤殑锛屾湁閬囧埌杩欑鎯呭喌鍚楋紵

198銆佸湪鏋勯€燬treamGraph锛岀被浼糚artitionTransformmation 杩欑绫诲瀷鐨?transform锛屼负浠€涔堣娣诲姞鎴愪竴涓櫄鎷熻妭鐐癸紝鑰屼笉鏄竴涓疄闄呯殑鐗╃悊鑺傜偣鍛紵

199銆乫link娑堣垂kafka鐨勬暟鎹啓鍏ュ埌hdfs涓紝鎴戦噰鐢ㄤ簡BucketingSink 杩欎釜sink灏唎perator鍑烘潵鐨勬暟鎹啓鍏ュ埌hdfs鏂囦欢涓婏紝骞堕€氳繃鍦╤ive涓缓澶栭儴琛ㄦ潵鏌ヨ杩欎釜銆備絾鐜板湪鏈変釜闂锛屽浜巌n-progress鐨勬枃浠讹紝hive鏄棤娉曡瘑鍒嚭鏉ヨ鏂囦欢涓殑鏁版嵁锛屽彲鎴戞兂鑳藉湪hive涓疄鏃舵煡璇㈣繘鏉ョ殑鏁版嵁锛屼笖涓嶆兂浜х敓寰堝鐨勫皬鏂囦欢锛岃繖涓濡備綍澶勭悊鍛?/span>

200銆侀噰鐢‵link鍗曟満闆嗙兢妯″紡涓€涓猨obmanager鍜屼袱涓猼askmanager锛屾満鍣ㄦ槸鍗曟満鏄?4鏍革紝鐜板湪鍋氫釜绠€鍗曠殑鍔熻兘浠巏afka鐨勪竴涓猼opic杞弧瓒虫潯浠剁殑娑堟伅鍒板彟涓€涓猼opic锛宼opic鐨勫垎鍖烘槸30锛屾垜璁剧疆浜嗙▼搴忛粯璁ゅ苟鍙戜负30锛岀幇鍦ㄦ瘡绉掓秷璐?w澶氭暟鎹紝涓嶅蹇紝璇烽棶鍙互鎬庝箞鎻愰珮job鐨勬€ц兘鍛紵

201銆丗link Metric 婧愮爜鍒嗘瀽

绛夌瓑绛夛紝杩樻湁寰堝锛屽鍒剁矘璐寸殑鎴戞墜绱晩 馃槀

鍙﹀閲岄潰杩樹細鍙婃椂鍒嗕韩 Flink 鐨勪竴浜涙渶鏂扮殑璧勬枡锛堝寘鎷暟鎹€佽棰戙€丳PT銆佷紭绉€鍗氬锛屾寔缁洿鏂帮紝淇濊瘉鍏ㄧ綉鏈€鍏紝鍥犱负鎴戠煡閬?Flink 鐩墠鐨勮祫鏂欒繕涓嶅锛?/p>

鍏充簬鑷繁瀵?Flink 瀛︿範鐨勪竴浜涙兂娉曞拰寤鸿

Flink 鍏ㄧ綉鏈€鍏ㄨ祫鏂欒幏鍙栵紝鎸佺画鏇存柊锛岀偣鍑诲彲浠ヨ幏鍙?/p>

鍐嶅氨鏄槦鐞冪敤鎴风粰鎴戞彁鐨勪竴鐐硅姹傦細涓嶅畾鏈熷垎浜竴浜涜嚜宸遍亣鍒扮殑 Flink 椤圭洰鐨勫疄鎴橈紝鐢熶骇椤圭洰閬囧埌鐨勯棶棰橈紝鏄浣曡В鍐崇殑绛夌粡楠屼箣璋堬紒

1銆佸浣曟煡鐪嬭嚜宸辩殑 Job 鎵ц璁″垝骞惰幏鍙栨墽琛岃鍒掑浘

2銆佸綋瀹炴椂鍛婅閬囧埌 Kafka 鍗冧竾鏁版嵁閲忓爢绉鍜嬪姙锛?/span>

3銆佸浣曞湪娴佹暟鎹腑姣斾袱涓暟鎹殑澶у皬锛?/span>澶氱瑙ｅ喅鏂规硶

4銆乲afka 绯诲垪鏂囩珷

5銆丗link鐜閮ㄧ讲銆佸簲鐢ㄩ厤缃強杩愯搴旂敤绋嬪簭

6銆佺洃鎺у钩鍙拌鏈夋灦鏋勬槸闀胯繖鏍峰瓙鐨?/span>

7銆併€婂ぇ鏁版嵁鈥滈噸纾呯偢寮光€濃€斺€斿疄鏃惰绠楁鏋?Flink銆嬩笓鏍忕郴鍒楁枃绔犵洰褰曞ぇ绾?/span>

8銆併€婂ぇ鏁版嵁鈥滈噸纾呯偢寮光€濃€斺€斿疄鏃惰绠楁鏋?Flink銆婥hat 浠樿垂鏂囩珷

9銆丄pache Flink 鏄浣曠鐞嗗ソ鍐呭瓨鐨勶紵

10銆丗link On K8s

11銆丗link-metrics-core

12銆丗link-metrics-datadog

13銆丗link-metrics-dropwizard

14銆丗link-metrics-graphite

15銆丗link-metrics-influxdb

16銆丗link-metrics-jmx

17銆丗link-metrics-slf4j

18銆丗link-metrics-statsd

19銆丗link-metrics-prometheus

褰撶劧锛岄櫎浜嗘洿鏂?Flink 鐩稿叧鐨勪笢瑗垮锛屾垜杩樹細鏇存柊涓€浜涘ぇ鏁版嵁鐩稿叧鐨勪笢瑗匡紝鍥犱负鎴戜釜浜轰箣鍓嶄笉鏄ぇ鏁版嵁寮€鍙戯紝鎵€浠ョ幇鍦ㄤ篃瑕佺媯琛ヤ簺鐭ヨ瘑锛佹€讳箣锛屽笇鏈涜繘鏉ョ殑绔ラ瀷浠竴璧峰叡鍚岃繘姝ワ紒

1銆丣ava 鏍稿績鐭ヨ瘑鐐规暣鐞?pdf

2銆佸亣濡傛垜鏄潰璇曞畼锛屾垜浼氶棶浣犺繖浜涢棶棰?/span>

3銆並afka 绯诲垪鏂囩珷鍜屽涔犺棰?/span>

4銆侀噸鏂板畾涔?Flink 绗簩鏈?pdf

5銆丟itChat Flink 鏂囩珷绛旂枒璁板綍

6銆丣ava 骞跺彂璇剧▼瑕佹帉鎻＄殑鐭ヨ瘑鐐?/span>

7銆丩ightweight Asynchronous Snapshots for Distributed Dataflows

8銆丄pache Flink鈩? Stream and Batch Processing in a Single Engine

9銆丗link鐘舵€佺鐞嗕笌瀹归敊鏈哄埗

10銆丗link 娴佹壒涓€浣撶殑鎶€鏈灦鏋勪互鍙婂湪闃块噷 鐨勫疄璺?/span>

11銆丗link Checkpoint-杞婚噺绾у垎甯冨紡蹇収

12銆丗link 娴佹壒涓€浣撶殑鎶€鏈灦鏋勪互鍙婂湪闃块噷 鐨勫疄璺?/span>

13銆丼tream Processing with Apache Flink pdf

14銆丗link 缁撳悎鏈哄櫒瀛︿範绠楁硶鐨勭洃鎺у钩鍙板疄璺?/span>

15銆併€婂ぇ鏁版嵁閲嶇鐐稿脊-瀹炴椂璁＄畻Flink銆嬮澶囩瘒鈥斺€斿ぇ鏁版嵁瀹炴椂璁＄畻浠嬬粛鍙婂叾甯哥敤浣跨敤鍦烘櫙 pdf 鍜?瑙嗛

16銆併€婂ぇ鏁版嵁閲嶇鐐稿脊-瀹炴椂璁＄畻Flink銆嬪紑绡囪瘝 pdf 鍜?瑙嗛

17銆佸洓鏈?Flink 涔?/span>

18銆佹祦澶勭悊绯荤粺 鐨勭浉鍏?paper

19銆丄pache Flink 1.9 鐗规€цВ璇?/span>

20銆佹墦閫犲熀浜嶧link Table API鐨勬満鍣ㄥ涔犵敓鎬?/span>

21銆佸熀浜嶧link on Kubernetes鐨勫ぇ鏁版嵁骞冲彴

22銆佸熀浜嶢pache Flink鐨勯珮鎬ц兘鏈哄櫒瀛︿範绠楁硶搴?/span>

23銆丄pache Flink鍦ㄥ揩鎵嬬殑搴旂敤涓庡疄璺?/span>

24銆丄pache Flink-1.9涓嶩ive鐨勫吋瀹规€?/span>

25銆佹墦閫犲熀浜嶧link Table API鐨勬満鍣ㄥ涔犵敓鎬?/span>

26銆佹祦澶勭悊绯荤粺 鐨勭浉鍏?paper