Hive Join浼樺寲

Posted 2020-12-21

鏍囩锛?a href='http://www.mamicode.com/so/1/%e5%85%ac%e4%bc%97%e5%8f%b7' title='鍏紬鍙?>鍏紬鍙?/a>   璋冧紭   杈撳嚭   rgba   瀹归噺   amp   鍊炬枩   杩炴帴   鍙傛暟   

鍦ㄩ槓杩?a href="https://mp.weixin.qq.com/s/Rte0_EKAkBnK2tvfZMtpXw" target="_blank">Hive Join鍏蜂綋鐨勪紭鍖栨柟娉?/a>涔嬪墠锛岄鍏堢湅涓€涓婬ive Join鐨勫嚑涓噸瑕佺壒鐐癸紝鍦ㄥ疄闄呬娇鐢ㄦ椂涔熷彲浠ュ埄鐢ㄤ笅鍒楃壒鐐瑰仛鐩稿簲浼樺寲锛?/span>

1. 鍙敮鎸佺瓑鍊艰繛鎺?/span>

2. 搴曞眰浼氬皢鍐欑殑HQL璇彞杞崲涓篗apReduce锛屽苟涓攔educe浼氬皢join璇彞涓櫎鏈€鍚庝竴涓〃澶栭兘缂撳瓨璧锋潵

3. 褰撲笁涓垨澶氫釜浠ヤ笂鐨勮〃杩涜join鎿嶄綔鏃讹紝濡傛灉姣忎釜on浣跨敤鐩稿悓鐨勫瓧娈佃繛鎺ユ椂鍙細浜х敓涓€涓猰apreduce

 

鍏蜂綋鐨勪紭鍖栧缓璁細

 

1. 鍚堢悊鐨勮缃甿ap鍜宺educe鏁伴噺

jvm閲嶇敤銆傚彲鍦╤adoop鐨刴apred-site.xml涓缃甹vm琚噸鐢ㄧ殑娆℃暟锛屽弬鏁癿apred.job.reuse.jvm.num.tasks

 

2. 瀵逛簬浠诲姟閲嶆病鏈変緷璧栧叧绯荤殑闃舵寮€鍚苟鍙戞墽琛岋紝璁剧疆灞炴€э細set hive.exec.parallel=true

 

3. 鏌ヨ鍒嗗尯琛ㄦ椂锛屽湪鏌ヨ鏉′欢涓寚瀹氬垎鍖?/span>

 

4. 灏介噺浣跨敤left semi join 鏇夸唬in銆乶ot in銆乪xists

鍥犱负left semi join鍦ㄦ墽琛屾椂锛屽浜庡乏琛ㄤ腑鎸囧畾鐨勪竴鏉¤褰曪紝涓€鏃﹀湪鍙宠〃涓壘鍒扮珛鍗冲仠姝㈡壂鎻忥紝鏁堢巼鏇撮珮

 

5. 褰撳涓〃杩涜鏌ヨ鏃讹紝浠庡乏鍒板彸琛ㄧ殑澶у皬椤哄簭搴旇鏄粠灏忓埌澶с€傚師鍥狅細hive鍦ㄥ姣忚璁板綍鎿嶄綔鏃朵細鎶婂叾浠栬〃鍏堢紦瀛樿捣鏉ワ紝鐩村埌鎵弿鏈€鍚庣殑琛ㄨ繘琛岃绠?/span>

 

6. 瀵逛簬缁忓父join鐨勮〃锛岄拡瀵筳oin瀛楁杩涜鍒嗘《锛岃繖鏍峰湪join鏃朵笉蹇呭叏琛ㄦ壂鎻?/span>

 

7. 灏忚〃杩涜mapjoin

濡傛灉鍦╦oin鐨勮〃涓紝鏈変竴寮犺〃鏁版嵁閲忚緝灏忥紝鍙互瀛樹簬鍐呭瓨涓紝杩欐牱璇ヨ〃鍦ㄥ拰鍏朵粬琛╦oin鏃跺彲浠ョ洿鎺ュ湪map绔繘琛岋紝鐪佹帀reduce杩囩▼锛屾晥鐜囬珮銆傝缃柟寮忎富瑕佸垎涓ょ锛?/span>

1锛夎嚜鍔ㄦ柟寮?/span>

set hive.auto.convert.join=true;hive.mapjoin.smalltable.filesize锛岃缃彲浠apjoin鐨勮〃鐨勫ぇ灏忥紝榛樿鍊兼槸25Mb

2锛夋墜鍔ㄦ柟寮?/span>

select  /*+ mapjoin(A)*/  x.a,  y.b from t_x x join t_y y on x.id=y.id;

 

8. 鍚屼竴绉嶆暟鎹殑澶氱澶勭悊锛氫粠涓€涓暟鎹簮浜х敓鐨勫涓暟鎹仛鍚堬紝鏃犻渶姣忔鑱氬悎閮介渶瑕侀噸鏂版壂鎻忎竴娆°€?/span>

渚嬪锛氫换鍔￠噸闇€瑕佹墽琛宨nsert overwrite table t_y select * from t_x;鍜?/span>

insert overwrite table t_z select * from t_x;

鍙互浼樺寲鎴愶細from t_x insert overwrite table t_y select * insert overwrite table t_z select *

 

9. join涓殑鏁版嵁鍊炬枩澶勭悊

set hive.optimize.skewjoin=true;

set hive.skewjoin.key=100000;

褰撳崟涓猺educe鑺傜偣澶勭悊鏁版嵁闃堝€硷紝浼氳繘琛宻kewjoin锛屽缓璁缃负骞冲潎鏁版嵁閲忕殑2-4鍊嶃€?/span>

鍘熺悊锛氫細浜х敓涓や釜job锛岀涓€涓猨ob浼氬皢瓒呰繃hive.skewjoin.key璁剧疆鍊肩殑璁板綍鐨刱ey鍔犱笂涓€浜涢殢鏈烘暟锛屽皢杩欎簺鐩稿悓鐨刱ey鎵撲贡锛岀劧鍚庡垎閰嶅埌涓嶅悓鐨勮妭鐐逛笂闈㈣繘琛岃绠椼€傛渶鍚庡啀鍚姩涓€涓猨ob锛屽湪绗竴涓猨ob澶勭悊鐨勫熀纭€涓婏紙鍗崇涓€涓猨ob鐨剅educe杈撳嚭缁撴灉锛夊啀杩涜澶勭悊锛屽皢鐩稿悓鐨刱ey鍒嗗彂鍒扮浉鍚岀殑鑺傜偣涓婂鐞嗐€傚洜涓轰細浜х敓涓や釜job杩涜澶勭悊锛屽湪瀹為檯浣跨敤涓繕鏄娉ㄦ剰浠ュ強闃堝€肩殑璁剧疆銆?/span>

 

10. limit璋冧紭

limit璇彞閫氬父鏄墽琛屾暣涓鍙ュ悗杩斿洖閮ㄥ垎缁撴灉銆備絾閫氳繃璁剧疆鍙傛暟set hive.limit.optimize.enable=true锛屽皢閽堝鏌ヨ瀵瑰厓鏁版嵁杩涜鎶芥牱銆傚悓鏃跺彲鑳借繕闇€瑕佽缃互涓嬩袱涓弬鏁帮細

set hive.limit.row.max.size=10000;璁剧疆鏈€灏忕殑閲囨牱瀹归噺

set hive.limit.optimize.limit.file=20;璁剧疆鏈€澶х殑閲囨牱鏍锋湰鏁?/span>

杩欑浼樺寲鏂瑰紡瀛樺湪涓€涓己鐐癸細鏈夊彲鑳介儴鍒嗘暟鎹案杩滀笉浼氳澶勭悊鍒?/span>

---

 

鍏虫敞寰俊鍏紬鍙凤細澶ф暟鎹涔犱笌鍒嗕韩锛岃幏鍙栨洿瀵规妧鏈共璐?/span>