鐮佸啘鎵嬭涓⊿PARK鎬ц兘璋冧紭

Posted 2021-04-05 鏋侀摼绉戞妧

鏍忕洰馃摪锛?span class="mq-12">鐮佸啘鎵嬭 馃捇

鎾版枃鉁嶐煆伙細鏋侀摼绉戞妧 楂樼骇鍚庣寮€鍙?nbsp;鍚村畯浼?/span>

缂栬緫馃摎锛氬叚鍏?/span>

鍏抽敭璇嶐煋?/span>锛?/span> SPARK 銆?/span> 寮€鍙戣皟浼?span class="mq-37"> 銆?/span> 璧勬簮鍙傛暟璋冧紭 

鉁嶏笍

 鐮佸啘鎵嬭 

灏嗕細閭€璇?/span>

涓€鐩村湪骞曞悗

鐢ㄤ唬鐮佸拰绠楁硶鏀瑰彉涓栫晫鐨?/span>

鎶€鏈ぇ浣滑

灏嗕細涓嶅畾鏈熸帹閫佷粬浠?/span>

鎵€鍐欑殑鍦ㄦ妧鏈笓涓氫腑鐨?/strong>

鎶€鏈粡楠?鐮旂┒/璁烘枃

涓轰綘鍛堢幇

/ 鏇村墠娌跨殑鎶€鏈€濊€?/

 / 鏇翠笓涓氱殑鎶€鏈共璐?/

#鐮佸啘鎵嬭# 绗?6鏈?/span>

鏂?| 鏋侀摼绉戞妧Video++ 楂樼骇鍚庣寮€鍙?nbsp;鍚村畯浼?/span>

/ spark鍩烘湰杩愯鍘熺悊 /

鍦ㄨ繘琛岃皟浼樹箣鍓嶏紝瑕佸厛浜嗚Вspark浣滀笟鍩烘湰鐨勮繍琛屽師鐞嗭紝鍦ㄦ鍩虹涓婃墠鑳藉浼樺寲鏈夋洿娣卞叆鐨勪簡瑙?/span>

1銆乻park-submit鎻愪氦涓€涓猄park浣滀笟涔嬪悗锛岃繖涓綔涓氬氨浼氬惎鍔ㄤ竴涓搴旂殑Driver杩涚▼ 

2銆佹牴鎹綘浣跨敤鐨勯儴缃叉ā寮忥紙deploy-mode:client/cluster锛変笉鍚岋紝Driver杩涚▼鍙兘鍦ㄦ湰鍦板惎鍔紝涔熷彲鑳藉湪闆嗙兢涓煇涓伐浣滆妭鐐逛笂鍚姩 

3銆丏river杩涚▼鏈韩浼氭牴鎹垜浠缃殑鍙傛暟锛屽崰鏈変竴瀹氭暟閲忕殑鍐呭瓨鍜孋PU Core 

4銆丏river杩涚▼瑕佸仛鐨勭涓€浠朵簨鎯咃紝灏辨槸鍚戦泦缇ょ鐞嗗櫒锛堝彲浠ユ槸Spark Standalone5闆嗙兢锛屼篃鍙互鏄痀ARN锛夌敵璇疯繍琛孲park浣滀笟闇€瑕佷娇鐢ㄧ殑璧勬簮銆?span>璧勬簮鎸囩殑灏辨槸Executor杩涚▼銆?/strong>鍦ㄥ悇涓伐浣滆妭鐐逛笂锛屽惎鍔ㄤ竴瀹氭暟閲忕殑Executor杩涚▼锛屾瘡涓狤xecutor杩涚▼閮藉崰鏈変竴瀹氭暟閲忕殑鍐呭瓨鍜孋PU Core

5銆佸湪鐢宠鍒颁簡浣滀笟鎵ц鎵€闇€鐨勮祫婧愪箣鍚庯紝Driver杩涚▼灏变細寮€濮嬭皟搴﹀拰鎵ц鎴戜滑缂栧啓鐨勪綔涓氫唬鐮佷簡 

6銆丏river杩涚▼浼氬皢鎴戜滑缂栧啓鐨凷park浣滀笟浠ｇ爜鍒嗘媶涓哄涓猄tage锛屾瘡涓猄tage鎵ц涓€閮ㄥ垎浠ｇ爜鐗囨锛屽苟涓烘瘡涓猄tage鍒涘缓涓€鎵筎ask锛岀劧鍚庡皢杩欎簺Task鍒嗛厤鍒板悇涓狤xecutor杩涚▼涓墽琛?nbsp;

7銆?span>Task鏄渶灏忕殑璁＄畻鍗曞厓锛岃礋璐ｆ墽琛屼竴妯′竴鏍风殑璁＄畻閫昏緫锛堜篃灏辨槸鎴戜滑缂栧啓鐨勬煇涓唬鐮佺墖娈碉級锛屽彧鏄瘡涓猅ask澶勭悊鐨勬暟鎹笉鍚岃€屽凡銆備竴涓猄tage鐨勬墍鏈塗ask閮芥墽琛屽畬姣曚箣鍚庯紝浼氬湪鍚勪釜鑺傜偣鏈湴鐨勭鐩樻枃浠朵腑鍐欏叆璁＄畻涓棿缁撴灉锛岀劧鍚嶥river灏变細璋冨害杩愯涓嬩竴涓猄tage銆備笅涓€涓猄tage鐨凾ask鐨勮緭鍏ユ暟鎹氨鏄笂涓€涓猄tage杈撳嚭鐨勪腑闂寸粨鏋?nbsp;

8銆丼park鏄牴鎹甋huffle绫荤畻瀛愭潵杩涜Stage鐨勫垝鍒嗭紝Shuffle绠楀瓙鎵ц涔嬪墠鐨勪唬鐮佷細琚垝鍒嗕负涓€涓猄tage

涓嬮潰灏嗕粠涓や釜鏂归潰瀵箂park璋冧紭杩涜浠嬬粛

/ 01  寮€鍙戣皟浼?/span> /

涓昏鍖呮嫭锛?span>RDD lineage璁捐銆佺畻瀛愮殑鍚堢悊浣跨敤銆佺壒娈婃搷浣滅殑浼樺寲绛?/strong>銆傚湪寮€鍙戣繃绋嬩腑锛屾椂鏃跺埢鍒婚兘搴旇娉ㄦ剰浠ヤ笂鍘熷垯锛屽苟灏嗚繖浜涘師鍒欐牴鎹叿浣撶殑涓氬姟浠ュ強瀹為檯鐨勫簲鐢ㄥ満鏅紝鐏垫椿鍦拌繍鐢ㄥ埌鑷繁鐨凷park浣滀笟涓€?/span>

 1    閬垮厤鍒涘缓閲嶅鐨凴DD  

鎴戜滑鍦ㄥ紑鍙戜竴涓猄park浣滀笟鏃讹紝棣栧厛鏄熀浜庢煇涓暟鎹簮锛堟瘮濡侶ive琛ㄦ垨HDFS鏂囦欢锛夊垱寤轰竴涓垵濮嬬殑RDD锛?/span> 鎺ョ潃瀵硅繖涓猂DD鎵ц鏌愪釜绠楀瓙鎿嶄綔锛岀劧鍚庡緱鍒颁笅涓€涓猂DD锛?/span> 浠ユ绫绘帹锛屽惊鐜線澶嶏紝鐩村埌璁＄畻鍑烘渶缁堟垜浠渶瑕佺殑缁撴灉銆?/span> 鍦ㄨ繖涓繃绋嬩腑锛屽涓猂DD浼氶€氳繃涓嶅悓鐨勭畻瀛愭搷浣滐紙姣斿map銆乺educe绛夛級涓茶捣鏉ワ紝杩欎釜鈥淩DD涓测€濓紝灏辨槸RDD lineage锛屼篃灏辨槸鈥淩DD鐨勮缂樺叧绯婚摼鈥?/strong>銆?/span>

鎴戜滑鍦ㄥ紑鍙戣繃绋嬩腑瑕佹敞鎰忥細瀵逛簬鍚屼竴浠芥暟鎹紝鍙簲璇ュ垱寤轰竴涓猂DD锛屼笉鑳藉垱寤哄涓猂DD鏉ヤ唬琛ㄥ悓涓€浠芥暟鎹€?/span>

 2    灏藉彲鑳藉鐢ㄥ悓涓€涓猂DD  

闄や簡瑕侀伩鍏嶅湪寮€鍙戣繃绋嬩腑瀵逛竴浠藉畬鍏ㄧ浉鍚岀殑鏁版嵁鍒涘缓澶氫釜RDD涔嬪锛?/span> 鍦ㄥ涓嶅悓鐨勬暟鎹墽琛岀畻瀛愭搷浣滄椂杩樿灏藉彲鑳藉湴澶嶇敤涓€涓猂DD 銆?/span> 姣斿璇达紝鏈変竴涓猂DD鐨勬暟鎹牸寮忔槸key-value绫诲瀷鐨勶紝鍙︿竴涓槸鍗晇alue绫诲瀷鐨勶紝杩欎袱涓猂DD鐨剉alue鏁版嵁鏄畬鍏ㄤ竴鏍风殑銆?/span> 閭ｄ箞姝ゆ椂鎴戜滑鍙互鍙娇鐢╧ey-value绫诲瀷鐨勯偅涓猂DD锛屽洜涓哄叾涓凡缁忓寘鍚簡鍙︿竴涓殑鏁版嵁銆?/span> 瀵逛簬绫讳技杩欑澶氫釜RDD鐨勬暟鎹湁閲嶅彔鎴栬€呭寘鍚殑鎯呭喌锛屾垜浠簲璇ュ敖閲忓鐢ㄤ竴涓猂DD锛岃繖鏍?span>鍙互灏藉彲鑳藉湴鍑忓皯RDD鐨勬暟閲忥紝浠庤€屽敖鍙兘鍑忓皯绠楀瓙鎵ц鐨勬鏁般€?/strong>

 3    灏介噺閬垮厤浣跨敤shuffle绫荤畻瀛?/span>  

濡傛灉鏈夊彲鑳界殑璇濓紝瑕佸敖閲忛伩鍏嶄娇鐢╯huffle绫荤畻瀛愩€?/strong>鍥犱负Spark浣滀笟杩愯杩囩▼涓紝鏈€娑堣€楁€ц兘鐨勫湴鏂瑰氨鏄痵huffle杩囩▼銆俿huffle杩囩▼锛岀畝鍗曟潵璇达紝灏辨槸灏嗗垎甯冨湪闆嗙兢涓涓妭鐐逛笂鐨勫悓涓€涓猭ey锛屾媺鍙栧埌鍚屼竴涓妭鐐逛笂锛岃繘琛岃仛鍚堟垨join绛夋搷浣溿€傛瘮濡俽educeByKey銆乯oin绛夌畻瀛愶紝閮戒細瑙﹀彂shuffle鎿嶄綔銆?/span>

shuffle杩囩▼涓紝鍚勪釜鑺傜偣涓婄殑鐩稿悓key閮戒細鍏堝啓鍏ユ湰鍦扮鐩樻枃浠朵腑锛岀劧鍚庡叾浠栬妭鐐归渶瑕侀€氳繃缃戠粶浼犺緭鎷夊彇鍚勪釜鑺傜偣涓婄殑纾佺洏鏂囦欢涓殑鐩稿悓key銆傝€屼笖鐩稿悓key閮芥媺鍙栧埌鍚屼竴涓妭鐐硅繘琛岃仛鍚堟搷浣滄椂锛岃繕鏈夊彲鑳戒細鍥犱负涓€涓妭鐐逛笂澶勭悊鐨刱ey杩囧锛屽鑷村唴瀛樹笉澶熷瓨鏀撅紝杩涜€屾孩鍐欏埌纾佺洏鏂囦欢涓€?/strong>鍥犳鍦╯huffle杩囩▼涓紝鍙兘浼氬彂鐢熷ぇ閲忕殑纾佺洏鏂囦欢璇诲啓鐨処O鎿嶄綔锛屼互鍙婃暟鎹殑缃戠粶浼犺緭鎿嶄綔銆傜鐩業O鍜岀綉缁滄暟鎹紶杈撲篃鏄痵huffle鎬ц兘杈冨樊鐨勪富瑕佸師鍥犮€?/span>

鍥犳鍦ㄦ垜浠殑寮€鍙戣繃绋嬩腑锛?span>鑳介伩鍏嶅垯灏藉彲鑳介伩鍏嶄娇鐢╮educeByKey銆乯oin銆乨istinct銆乺epartition绛変細杩涜shuffle鐨勭畻瀛愶紝灏介噺浣跨敤map绫荤殑闈瀞huffle绠楀瓙銆?/strong>杩欐牱鐨勮瘽锛屾病鏈塻huffle鎿嶄綔鎴栬€呬粎鏈夎緝灏憇huffle鎿嶄綔鐨凷park浣滀笟锛屽彲浠ュぇ澶у噺灏戞€ц兘寮€閿€銆?/span>

 4  浣跨敤楂樻€ц兘鐨勭畻瀛?/span> 

路 浣跨敤reduceByKey/aggregateByKey鏇夸唬groupByKey

groupByKey娌℃湁杩涜浠讳綍鏈湴鑱氬悎锛屾墍鏈夋暟鎹兘浼氬湪闆嗙兢鑺傜偣涔嬮棿浼犺緭reduceByKey姣忎釜鑺傜偣鏈湴鐨勭浉鍚宬ey鏁版嵁锛岄兘杩涜浜嗛鑱氬悎锛岀劧鍚庢墠浼犺緭鍒板叾浠栬妭鐐逛笂杩涜鍏ㄥ眬鑱氬悎銆?/span>

路 浣跨敤mapPartitions鏇夸唬鏅€歮ap

mapPartitions绫荤殑绠楀瓙锛屼竴娆″嚱鏁拌皟鐢ㄤ細澶勭悊涓€涓猵artition鎵€鏈夌殑鏁版嵁锛岃€屼笉鏄竴娆″嚱鏁拌皟鐢ㄥ鐞嗕竴鏉★紝鎬ц兘鐩稿鏉ヨ浼氶珮涓€浜涖€備絾鏄湁鐨勬椂鍊欙紝浣跨敤mapPartitions浼氬嚭鐜癘OM锛堝唴瀛樻孩鍑猴級鐨勯棶棰樸€?span>鍥犱负鍗曟鍑芥暟璋冪敤灏辫澶勭悊鎺変竴涓猵artition鎵€鏈夌殑鏁版嵁锛屽鏋滃唴瀛樹笉澶燂紝鍨冨溇鍥炴敹鏃舵槸鏃犳硶鍥炴敹鎺夊お澶氬璞＄殑锛屽緢鍙兘鍑虹幇OOM寮傚父銆傛墍浠ヤ娇鐢ㄨ繖绫绘搷浣滄椂瑕佹厧閲嶏紒

路 浣跨敤foreachPartitions鏇夸唬foreach

鍘熺悊绫讳技浜庘€滀娇鐢╩apPartitions鏇夸唬map鈥濓紝涔熸槸涓€娆″嚱鏁拌皟鐢ㄥ鐞嗕竴涓猵artition鐨勬墍鏈夋暟鎹紝鑰屼笉鏄竴娆″嚱鏁拌皟鐢ㄥ鐞嗕竴鏉℃暟鎹€傚湪瀹炶返涓彂鐜帮紝foreachPartitions绫荤殑绠楀瓙锛屽鎬ц兘鐨勬彁鍗囪繕鏄緢鏈夊府鍔╃殑銆傛瘮濡傚湪foreach鍑芥暟涓紝灏哛DD涓墍鏈夋暟鎹啓MySQL锛岄偅涔堝鏋滄槸鏅€氱殑foreach绠楀瓙锛屽氨浼氫竴鏉℃暟鎹竴鏉℃暟鎹湴鍐欙紝姣忔鍑芥暟璋冪敤鍙兘灏变細鍒涘缓涓€涓暟鎹簱杩炴帴锛屾鏃跺氨鍔垮繀浼氶绻佸湴鍒涘缓鍜岄攢姣佹暟鎹簱杩炴帴锛屾€ц兘鏄潪甯镐綆涓嬶紱浣嗘槸濡傛灉鐢╢oreachPartitions绠楀瓙涓€娆℃€у鐞嗕竴涓猵artition鐨勬暟鎹紝閭ｄ箞瀵逛簬姣忎釜partition锛屽彧瑕佸垱寤轰竴涓暟鎹簱杩炴帴鍗冲彲锛岀劧鍚庢墽琛屾壒閲忔彃鍏ユ搷浣滐紝姝ゆ椂鎬ц兘鏄瘮杈冮珮鐨勩€?span>瀹炶返涓彂鐜帮紝瀵逛簬1涓囨潯宸﹀彸鐨勬暟鎹噺鍐橫ySQL锛屾€ц兘鍙互鎻愬崌30%浠ヤ笂銆?/strong>

路 浣跨敤filter涔嬪悗杩涜coalesce鎿嶄綔

閫氬父瀵逛竴涓猂DD鎵цfilter绠楀瓙杩囨护鎺塕DD涓緝澶氭暟鎹悗锛堟瘮濡?0%浠ヤ笂鐨勬暟鎹級锛?span>寤鸿浣跨敤coalesce绠楀瓙锛屾墜鍔ㄥ噺灏慠DD鐨刾artition鏁伴噺锛屽皢RDD涓殑鏁版嵁鍘嬬缉鍒版洿灏戠殑partition涓幓銆傚洜涓篺ilter涔嬪悗锛孯DD鐨勬瘡涓猵artition涓兘浼氭湁寰堝鏁版嵁琚繃婊ゆ帀锛屾鏃跺鏋滅収甯歌繘琛屽悗缁殑璁＄畻锛屽叾瀹炴瘡涓猼ask澶勭悊鐨刾artition涓殑鏁版嵁閲忓苟涓嶆槸寰堝锛屾湁涓€鐐硅祫婧愭氮璐癸紝鑰屼笖姝ゆ椂澶勭悊鐨則ask瓒婂锛屽彲鑳介€熷害鍙嶈€岃秺鎱€傚洜姝ょ敤coalesce鍑忓皯partition鏁伴噺锛屽皢RDD涓殑鏁版嵁鍘嬬缉鍒版洿灏戠殑partition涔嬪悗锛屽彧瑕佷娇鐢ㄦ洿灏戠殑task鍗冲彲澶勭悊瀹屾墍鏈夌殑partition銆傚湪鏌愪簺鍦烘櫙涓嬶紝瀵逛簬鎬ц兘鐨勬彁鍗囦細鏈変竴瀹氱殑甯姪銆?/span>

路 浣跨敤repartitionAndSortWithinPartitions鏇夸唬repartition涓巗ort绫绘搷浣?/span>

repartitionAndSortWithinPartitions鏄疭park瀹樼綉鎺ㄨ崘鐨勪竴涓畻瀛愶紝瀹樻柟寤鸿锛屽鏋滈渶瑕佸湪repartition閲嶅垎鍖轰箣鍚庯紝杩樿杩涜鎺掑簭锛屽缓璁洿鎺ヤ娇鐢╮epartitionAndSortWithinPartitions绠楀瓙銆傚洜涓鸿绠楀瓙鍙互涓€杈硅繘琛岄噸鍒嗗尯鐨剆huffle鎿嶄綔锛屼竴杈硅繘琛屾帓搴忋€俿huffle涓巗ort涓や釜鎿嶄綔鍚屾椂杩涜锛屾瘮鍏坰huffle鍐峴ort鏉ヨ锛屾€ц兘鍙兘鏄楂樼殑銆?/span>

/ 02 璧勬簮鍙傛暟璋冧紭 /

 1  num-executors 

璁剧疆Spark浣滀笟鎬诲叡瑕佺敤澶氬皯涓狤xecutor杩涚▼鏉ユ墽琛孌river鍦ㄥ悜YARN闆嗙兢绠＄悊鍣ㄧ敵璇疯祫婧愭椂锛孻ARN闆嗙兢绠＄悊鍣ㄤ細灏藉彲鑳芥寜鐓т綘鐨勮缃潵鍦ㄩ泦缇ょ殑鍚勪釜宸ヤ綔鑺傜偣涓婏紝鍚姩鐩稿簲鏁伴噺鐨凟xecutor杩涚▼

姣忎釜Spark浣滀笟鐨勮繍琛屼竴鑸嵁闆嗙兢鐨勮妯¤Executor杩涚▼姣旇緝鍚堥€傦紝璁剧疆澶皯鎴栧お澶氱殑Executor杩涚▼閮戒笉濂?/strong>銆傝缃殑澶皯锛屾棤娉曞厖鍒嗗埄鐢ㄩ泦缇よ祫婧愶紱璁剧疆鐨勫お澶氱殑璇濓紝澶ч儴鍒嗛槦鍒楀彲鑳芥棤娉曠粰浜堝厖鍒嗙殑璧勬簮

 2    executor-memory  

璁剧疆姣忎釜Executor杩涚▼鐨勫唴瀛楨xecutor鍐呭瓨鐨勫ぇ灏忥紝寰堝鏃跺€欑洿鎺ュ喅瀹氫簡Spark浣滀笟鐨勬€ц兘锛岃€屼笖璺熷父瑙佺殑JVM OOM寮傚父锛屼篃鏈夌洿鎺ョ殑鍏宠仈

姣忎釜Executor杩涚▼鐨勫唴瀛樿缃?G~8G杈冧负鍚堥€?/span>锛宯um-executors涔樹互executor-memory锛屾槸涓嶈兘瓒呰繃闃熷垪鐨勬渶澶у唴瀛橀噺鐨勶紝Spark闆嗙兢鍙互璁剧疆姣忎釜executor鏈€澶氫娇鐢ㄧ殑鍐呭瓨澶у皬銆傚鏋滀綘鏄窡鍥㈤槦閲屽叾浠栦汉鍏变韩杩欎釜璧勬簮闃熷垪锛岄偅涔堢敵璇风殑鍐呭瓨閲忔渶濂戒笉瑕佽秴杩囪祫婧愰槦鍒楁渶澶ф€诲唴瀛樼殑1/3锝?/2

 3  executor-cores 

璁剧疆姣忎釜Executor杩涚▼鐨凜PU core鏁伴噺鍐冲畾浜嗘瘡涓狤xecutor杩涚▼骞惰鎵цtask绾跨▼鐨勮兘鍔?/span>

鏁伴噺璁剧疆涓?~4涓緝涓哄悎閫傦紝渚濇嵁璧勬簮闃熷垪鐨勬渶澶PU Core闄愬埗鏄灏戯紝鍐嶄緷鎹缃殑Executor鏁伴噺锛屾潵鍐冲畾姣忎釜Executor杩涚▼鍙互鍒嗛厤鍒板嚑涓狢PU Core

 4    driver-memory  

璁剧疆Driver杩涚▼鐨勫唴瀛楧river鐨勫唴瀛橀€氬父鏉ヨ涓嶈缃紝鎴栬€呰缃?G宸﹀彸搴旇灏卞浜?/span>

濡傛灉闇€瑕佷娇鐢?collect 绠楀瓙灏哛DD鐨勬暟鎹叏閮ㄦ媺鍙栧埌Driver涓婅繘琛屽鐞嗭紝閭ｄ箞蹇呴』纭繚Driver鐨勫唴瀛樿冻澶熷ぇ锛屽惁鍒欎細鍑虹幇OOM鍐呭瓨婧㈠嚭鐨勯棶棰?/strong>

 5    spark.default.parallelism  

璁剧疆姣忎釜stage鐨勯粯璁ask鏁伴噺涓嶅幓璁剧疆杩欎釜鍙傛暟锛岄偅涔圫park鏍规嵁搴曞眰HDFS鐨刡lock鏁伴噺鏉ヨ缃畉ask鐨勬暟閲忥紝榛樿鏄竴涓狧DFS block瀵瑰簲涓€涓猼ask锛岄€氬父鏉ヨ锛孲park榛樿璁剧疆鐨勬暟閲忔槸鍋忓皯鐨?/span>

璁剧疆璇ュ弬鏁颁负num-executors * executor-cores鐨?~3鍊嶈緝涓哄悎閫傚鏋渢ask鏁伴噺鍋忓皯鐨勮瘽锛孍xecutor杩涚▼鍙兘鏍规湰灏辨病鏈塼ask鎵ц锛屼篃灏辨槸鐧界櫧娴垂浜嗚祫婧?/span>

 6    spark.storage.memoryFraction  

璁剧疆RDD鎸佷箙鍖栨暟鎹湪Executor鍐呭瓨涓兘鍗犵殑姣斾緥锛岄粯璁ゆ槸0.6鏍规嵁浣犻€夋嫨鐨勪笉鍚岀殑鎸佷箙鍖栫瓥鐣ワ紝濡傛灉鍐呭瓨涓嶅鏃讹紝鍙兘鏁版嵁灏变笉浼氭寔涔呭寲锛屾垨鑰呮暟鎹細鍐欏叆纾佺洏

濡傛灉Spark浣滀笟涓紝鏈夎緝澶氱殑RDD鎸佷箙鍖栨搷浣滐紝璇ュ弬鏁扮殑鍊煎彲浠ラ€傚綋鎻愰珮涓€浜涘鏋淪park浣滀笟涓殑Shuffle绫绘搷浣滄瘮杈冨锛岃€屾寔涔呭寲鎿嶄綔姣旇緝灏戯紝閭ｄ箞杩欎釜鍙傛暟鐨勫€奸€傚綋闄嶄綆涓€浜涙瘮杈冨悎閫傚鏋滃彂鐜颁綔涓氱敱浜庨绻佺殑GC瀵艰嚧杩愯缂撴參锛堥€氳繃Spark WebUI鍙互瑙傚療鍒颁綔涓氱殑GC鑰楁椂锛夛紝鎰忓懗鐫€Task鎵ц鐢ㄦ埛浠ｇ爜鐨勫唴瀛樹笉澶熺敤锛岄偅涔堝悓鏍峰缓璁皟浣庤繖涓弬鏁扮殑鍊?/span>

 7    spark.shuffle.memoryFraction  

璁剧疆Shuffle杩囩▼涓竴涓猼ask鎷夊彇鍒颁笂涓猄tage鐨凾ask鐨勮緭鍑哄悗锛岃繘琛岃仛鍚堟搷浣滄椂鑳藉浣跨敤鐨凟xecutor鍐呭瓨鐨勬瘮渚嬶紝榛樿鏄?.2Shuffle鎿嶄綔鍦ㄨ繘琛岃仛鍚堟椂锛?span>濡傛灉鍙戠幇浣跨敤鐨勫唴瀛樿秴鍑轰簡杩欎釜20%鐨勯檺鍒讹紝閭ｄ箞澶氫綑鐨勬暟鎹氨浼氭孩鍐欏埌纾佺洏鏂囦欢涓幓锛屾鏃跺氨浼氭瀬澶у湴闄嶄綆鎬ц兘

濡傛灉Spark浣滀笟涓殑RDD鎸佷箙鍖栨搷浣滆緝灏戯紝Shuffle鎿嶄綔杈冨鏃讹紝寤鸿闄嶄綆鎸佷箙鍖栨搷浣滅殑鍐呭瓨鍗犳瘮锛屾彁楂楽huffle鎿嶄綔鐨勫唴瀛樺崰姣旀瘮渚嬪鏋滃彂鐜颁綔涓氱敱浜庨绻佺殑GC瀵艰嚧杩愯缂撴參锛屾剰鍛崇潃Task鎵ц鐢ㄦ埛浠ｇ爜鐨勫唴瀛樹笉澶熺敤锛岄偅涔?span>鍚屾牱寤鸿璋冧綆杩欎釜鍙傛暟鐨勫€笺€?/strong>

馃殌

 鏋侀摼绉戞妧Video++ 

 鑱氱劍浜嶢I娑堣垂鐨勭鎶€鍏徃 

馃憞馃徎鐐瑰嚮銆岄槄璇诲師鏂囥€?/strong>鏌ョ湅鏇村娣卞害鎶€鏈ソ鏂?/span>