寰蒋鍜岃胺姝屽垎鍒紑婧愬垎甯冨紡娣卞害瀛︿範妗嗘灦锛屽悇鑷帀瀹冲湪鍝紵

Posted 2021-04-19 澶ф暟鎹枃鎽?/a> 寰

澶ф暟鎹枃鎽樺嚭鍝?/span>

鏉ユ簮锛歁edium

缂栬瘧锛歁iggy

寰蒋鍜岃胺姝屼竴鐩村湪绉瀬鐮旂┒鐢ㄤ簬璁粌娣卞害绁炵粡缃戠粶鐨勬柊妗嗘灦锛屽苟涓斿湪鏈€杩戝皢鍚勮嚜鐨勬垚鏋滃紑婧愨€斺€?span>寰蒋鐨凱ipeDream鍜岃胺姝岀殑GPipe銆?/strong>

 

鍘熷垯涓婄湅锛屼粬浠兘閬靛惊浜嗙被浼肩殑鍘熷垯鏉ヨ缁冩繁搴﹀涔犳ā鍨嬨€傝繖涓や釜椤圭洰宸插湪鍚勮嚜鐨勭爺绌惰鏂囷紙PipeDream锛孏Pipe锛変腑杩涜浜嗚缁嗕粙缁嶏紝杩欑瘒鏂囩珷灏嗗姝よ繘琛屾€荤粨銆?/span>

 

鍏堟斁涓奊itHub寮€婧愬湴鍧€馃憞

寰蒋锛?/span>

https://github.com/msr-fiddle/pipedream

璋锋瓕锛?/span>

https://github.com/tensorflow/lingvo/blob/master/lingvo/core/gpipe.py

 

浼楁墍鍛ㄧ煡锛屽湪瀹為獙杩囩▼涓紝铏界劧璁粌鍩烘湰妯″瀷姣旇緝鐞愮锛屼絾澶嶆潅搴﹀嵈闅忔ā鍨嬬殑璐ㄩ噺鍜屽ぇ灏忕嚎鎬у鍔犮€?/strong>渚嬪锛?014骞碔mageNet瑙嗚璇嗗埆鎸戞垬璧涚殑鍐犲啗鏄疓oogleNet锛屽畠閫氳繃400涓囦釜鍙傛暟鑾峰緱浜?4.8锛呯殑top1鍑嗙‘鎬э紝鑰屼粎浠呬笁骞翠箣鍚庯紝2017骞碔mageNet鎸戞垬璧涚殑鍐犲啗灏变娇鐢?.458浜夸釜鍙傛暟锛堝浜?6鍊嶏級鐨勬渶鏂扮缁忕綉缁滃疄鐜颁簡top1鍑嗙‘鐜団€斺€?2.7锛呫€備絾鏄紝鍦ㄥ悓涓€鏃舵湡锛孏PU鍐呭瓨浠呭鍔犱簡绾?鍊嶃€?/span>

 

闅忕潃妯″瀷缂╂斁浠ヨ揪鍒版洿楂樼殑鍑嗙‘鎬э紝瀵硅繖浜涙ā鍨嬬殑璁粌鍙樺緱瓒婃潵瓒婂叿鏈夋寫鎴樻€с€傚墠闈㈢殑鏍锋湰涔熸樉绀轰簡锛屼緷闈燝PU鍩虹缁撴瀯鐨勬敼杩涙潵瀹炵幇鏇村ソ鐨勮缁冩槸涓嶅彲鎸佺画鐨勩€傛垜浠渶瑕佸垎甯冨紡璁＄畻鏂规硶锛岃繖浜涙柟娉曞彲浠ュ苟琛屽寲璺ㄤ笉鍚岃妭鐐圭殑璁粌宸ヤ綔閲忥紝浠ユ墿灞曡缁冭妯°€?span>鍒嗗竷寮忚缁冪殑姒傚康鍚捣鏉ュ緢鐞愮锛屼絾瀹為檯涓婂嵈鏋佸叾澶嶆潅銆?/strong>

 

璋锋瓕鐨凣Pipe

 

GPipe涓撴敞浜庢墿灞曟繁搴﹀涔犺鍒掔殑璁粌宸ヤ綔閲忋€?/span> 浠庡熀纭€鏋舵瀯鐨勮搴︽潵鐪嬶紝璁粌杩囩▼鐨勫鏉傛€ф槸娣卞害瀛︿範妯″瀷缁忓父琚拷瑙嗙殑涓€涓柟闈€傝缁冩暟鎹泦瓒婃潵瓒婂ぇ锛岃秺鏉ヨ秺澶嶆潅銆備緥濡傦紝鍦ㄥ尰鐤椾繚鍋ラ鍩燂紝闇€瑕佷娇鐢ㄦ暟鐧句竾涓珮鍒嗚鲸鐜囧浘鍍忚繘琛岃缁冪殑妯″瀷骞朵笉缃曡銆傜粨鏋滐紝璁粌杩囩▼閫氬父瑕佽姳璐瑰緢闀挎椂闂存墠鑳藉畬鎴愶紝骞朵笖鍐呭瓨鍜孋PU娑堣€楅潪甯稿ぇ銆?/span>

 

鎬濊€冩繁搴﹀涔犳ā鍨嬬殑鍒嗗竷寮忕殑鏈夋晥鏂规硶鏄皢鍏跺垝鍒嗕负鏁版嵁鍒嗗竷寮忓拰妯″瀷鍒嗗竷寮忋€傛暟鎹垎甯冨紡鏂规硶閲囩敤澶у瀷鏈哄櫒闆嗙兢锛屽皢杈撳叆鏁版嵁鎷嗗垎鍒板畠浠箣闂淬€傛ā鍨嬪垎甯冨紡灏濊瘯灏嗘ā鍨嬬Щ鑷冲叿鏈夌壒瀹氱‖浠剁殑鍔犻€熷櫒锛屼緥濡侴PU鎴朤PU锛屼互鍔犻€熸ā鍨嬭缁冦€?/span>

 

姒傚康涓婄湅锛屽嚑涔庢墍鏈夎缁冩暟鎹泦閮藉彲浠ユ寜鐓т竴瀹氱殑閫昏緫杩涜鍒嗗竷寮忚缁冿紝浣嗘槸鍏充簬妯″瀷鐨勮娉曞嵈涓嶅敖鐩稿悓銆備緥濡傦紝涓€浜涙繁搴﹀涔犳ā鍨嬬敱鍙互鐙珛璁粌鐨勫苟琛屽垎鏀粍鎴愩€傚湪閭ｇ鎯呭喌涓嬶紝缁忓吀绛栫暐鏄皢璁＄畻鍒掑垎涓哄涓垎鍖猴紝骞跺皢涓嶅悓鐨勫垎鍖哄垎閰嶇粰涓嶅悓鐨勫垎鏀€備絾鏄紝杩欑绛栫暐鍦ㄦ寜椤哄簭鍫嗗彔鍚勫眰鐨勬繁搴﹀涔犳ā鍨嬩腑鏄笉瓒崇殑锛?/span>

 

GPipe閫氳繃鍒╃敤涓€绉嶇О涓烘祦姘寸嚎鐨勬妧鏈皢鏁版嵁鍜屾ā鍨嬪垎甯冨紡缁撳悎鍦ㄤ竴璧枫€備粠姒傚康涓婅锛孏Pipe鏄竴涓垎甯冨紡鏈哄櫒瀛︿範搴擄紝瀹冧娇鐢ㄥ悓姝ラ殢鏈烘搴︿笅闄嶅拰娴佹按绾垮垎甯冨紡杩涜璁粌锛岄€傜敤浜庣敱澶氫釜杩炵画灞傜粍鎴愮殑浠讳綍DNN銆?/span>

 

GPipe鍦ㄤ笉鍚岀殑鍔犻€熷櫒涔嬮棿鍒掑垎妯″瀷锛屽苟鑷姩灏嗕竴灏忔壒璁粌鏍锋湰鎷嗗垎涓鸿緝灏忕殑寰壒銆傝妯″瀷鍏佽GPipe鐨勫姞閫熷櫒骞惰杩愯锛屼粠鑰屾渶澶ч檺搴﹀湴鎻愰珮浜嗚缁冭繃绋嬬殑鍙墿灞曟€с€?/span>

 

涓嬪浘璇存槑浜嗗叿鏈夎繛缁眰鐨勭缁忕綉缁滅殑GPipe妯″瀷鍦ㄥ洓涓姞閫熷櫒涔嬮棿鍒嗛厤銆侳k鏄k涓垎鍖虹殑澶嶅悎姝ｅ悜璁＄畻鍑芥暟銆侭k鏄浉搴旂殑鍙嶅悜浼犳挱鍑芥暟銆侭k鍙栧喅浜庝笂灞傜殑Bk + 1鍜孎k鐨勪腑闂存縺娲汇€傚湪椤剁骇妯″瀷涓紝鎴戜滑鍙互鐪嬪埌缃戠粶鐨勯『搴忔€ц川濡備綍瀵艰嚧璧勬簮鍒╃敤涓嶈冻銆備笅鍥炬樉绀轰簡GPipe鏂规硶锛屽叾涓皢杈撳叆鐨勮糠浣犳壒澶勭悊鍒嗕负杈冨皬鐨勫畯鎵瑰鐞嗭紝杩欎簺瀹忔壒澶勭悊鍙敱鍔犻€熷櫒鍚屾椂澶勭悊銆?/span>

鍥剧墖鏉ユ簮锛?/span>

https://arxiv.org/pdf/1811.06965.pdf

 

寰蒋鐨凱ipeDream

 

鍑犱釜鏈堝墠锛屽井杞爺绌堕櫌瀹ｅ竷鍒涘缓Project Fiddle锛岃繖鏄竴绯诲垪鏃ㄥ湪绠€鍖栧垎甯冨紡娣卞害瀛︿範鐨勭爺绌堕」鐩€侾ipeDreams鏄疐iddle椤圭洰棣栨鍙戝竷鐨勭増鏈箣涓€锛?span>涓撴敞浜庢繁搴﹀涔犳ā鍨嬭缁冪殑骞惰鍖栥€?/strong>

 

PipeDream閲囩敤涓庡叾浠栨柟娉曚笉鍚岀殑鏂规硶鏉ュ埄鐢ㄧО涓虹閬撳垎甯冨紡鐨勬妧鏈潵鎵╁睍娣卞害瀛︿範妯″瀷鐨勮缁冦€傝繖绉嶆柟娉曡瘯鍥捐В鍐虫暟鎹拰妯″瀷骞惰鎶€鏈殑涓€浜涙寫鎴橈紝渚嬪GPipe涓娇鐢ㄧ殑鎶€鏈€?/span>

 

閫氬父锛屽湪浜戝熀纭€鏋舵瀯涓婅繘琛岃缁冩椂锛屾暟鎹苟琛屾柟娉曞湪瑙勬ā涓婁細鎵垮彈杈冮珮鐨勯€氫俊鎴愭湰锛屽苟涓旈殢鐫€鏃堕棿鐨勬帹绉讳細鎻愰珮GPU璁＄畻閫熷害銆傜被浼煎湴锛屾ā鍨嬪垎甯冨紡鎶€鏈€氬父鍦ㄥ埄鐢ㄧ‖浠惰祫婧愪笂鏇村姞鏁堢巼浣庝笅锛岀▼搴忓憳闇€瑕佸喅瀹氬浣曞湪缁欏畾纭欢閮ㄧ讲鐨勬儏鍐典笅鎷嗗垎鍏剁壒瀹氭ā鍨嬶紝缁欎粬浠甫鏉ヤ簡涓嶅繀瑕佺殑璐熸媴銆?/span>

 

鍥剧墖鏉ユ簮锛?/span>

http://www.microsoft.com/zh-cn/research/uploads/prod/2019/08/fiddle_pipedream_sosp19.pdf

 

PipeDream灏濊瘯閫氳繃浣跨敤绉颁负绠￠亾鍒嗗竷寮忕殑鎶€鏈潵鍏嬫湇鏁版嵁妯″瀷鍒嗗竷寮忔柟娉曠殑涓€浜涙寫鎴樸€?/span>

 

浠庢蹇典笂璁诧紝绠￠亾鍒嗗竷璁＄畻娑夊強灏咲NN妯″瀷鐨勫悇灞傚垝鍒嗕负澶氫釜闃舵锛屽叾涓瘡涓樁娈靛潎鐢辨ā鍨嬩腑鐨勪竴缁勮繛缁眰缁勬垚銆傛瘡涓樁娈甸兘鏄犲皠鍒颁竴涓崟鐙殑GPU锛岃GPU瀵硅闃舵涓殑鎵€鏈夊眰鎵ц姝ｅ悜浼犻€掞紙鍜屽弽鍚戜紶閫掞級銆?/span>

 

缁欏畾涓€涓壒瀹氱殑娣卞害绁炵粡缃戠粶锛孭ipeDream浼氬熀浜庡湪鍗曚釜GPU涓婃墽琛岀殑绠€鐭瑕佸垎鏋愶紝鑷姩纭畾濡備綍瀵笵NN鐨勮繍绠楃杩涜鍒嗗尯锛屽湪涓嶅悓闃舵涔嬮棿骞宠　璁＄畻璐熻浇锛屽悓鏃舵渶澶х▼搴﹀湴鍑忓皯涓庣洰鏍囧钩鍙扮殑閫氫俊銆傚嵆浣垮瓨鍦ㄦā鍨嬪鏍锋€э紙璁＄畻鍜岄€氫俊锛夊拰骞冲彴澶氭牱鎬э紙浜掕繛鎷撴墤鍜屽垎灞傚甫瀹斤級锛孭ipeDream涔熶細鏈夋晥鍦板疄鐜拌礋杞藉钩琛°€侾ipeDream璁粌鍒嗗竷寮忕殑鏂规硶鐨勫師鐞嗘瘮鏁版嵁妯″瀷鍒嗗竷寮忔柟娉曞叿鏈夊涓紭鐐广€?/span>

 

瀵逛簬鍒濆鑰呰€岃█锛孭ipeDream闇€瑕佸湪宸ヤ綔绋嬪簭鑺傜偣涔嬮棿杩涜杈冨皯鐨勯€氫俊锛屽洜涓虹閬撴墽琛屼腑鐨勬瘡涓伐浣滅▼搴忎粎闇€瑕佸皢娓愬彉鐨勫瓙闆嗗拰杈撳嚭婵€娲讳俊鎭紶杈剧粰鍗曚釜鍏朵粬宸ヤ綔绋嬪簭銆?/span>

 

鍥剧墖鏉ユ簮锛?/span>

https ://www.microsoft.com/zh-cn/research/uploads/prod/2019/08/fiddle_pipedream_sosp19.pdf

 

璁粌鍒嗗竷寮忔槸鏋勫缓鏇村ぇ銆佹洿鍑嗙‘鐨勬繁搴﹀涔犳ā鍨嬬殑鍏抽敭鎸戞垬涔嬩竴銆傚垎甯冨紡璁粌鏂规硶鏄繁搴﹀涔犵ぞ鍖轰腑涓€涓椿璺冪殑鐮旂┒棰嗗煙锛岄渶瑕佸皢鏈夋晥鐨勫苟鍙戠紪绋嬫妧鏈笌娣卞害瀛︿範妯″瀷鐨勬湰璐ㄧ浉缁撳悎銆傚敖绠′粛澶勪簬鏃╂湡闃舵锛屼絾Google鐨凣Pipe鍜孧icrosoft鐨凱ipeDream鏈韩宸茬粡鏄緢浼樼鐨勪骇鍝侊紝瀹冩槸娣卞害瀛︿範寮€鍙戜汉鍛樺彲鐢ㄧ殑涓ょ鏈€鍏峰垱閫犳€х殑鍒嗗竷寮忚缁冩柟娉曘€?/span>

 

绱犳潗鏉ユ簮锛?/span>

https://medium.com/dataseries/microsoft-and-google-open-sourced-these-frameworks-based-on-their-work-scaling-deep-learning-c0510e907038

瀹炰範/鍏ㄨ亴缂栬緫璁拌€呮嫑鑱榠ng

鐐广€屽湪鐪嬨€嶇殑浜洪兘鍙樺ソ鐪嬩簡鍝︼紒