多处理池的python保存数据在linux上没有给出预期的结果

Posted 2023-02-16

【中文标题】多处理池的python保存数据在linux上没有给出预期的结果

【英文标题】：python save data of multiprocessing pool does not give expected result on linux

【发布时间】：2021-03-24 15:39:14

【问题描述】：

我正在尝试使用 python 中的多处理池同时运行许多计算。每次计算完成时，我都想将结果保存在文件中以备后用。在下面的代码中，我试图测试这个想法。在 Windows 上，我得到每个计算的不同输出的预期结果。 Linux 上的相同代码存在问题，因为它为所有计算输出相同的结果。我不明白如何在 Linux 上解决这个问题。此外，我很高兴知道如何在每次计算完成时保存数据，而不是等待所有计算完成然后同时保存所有输出。我尝试使用 this site 上的示例使用 python 实现不同类型的并行计算，但无法在 Linux 上解决此问题。过去两天我也在搜索解决方案，但找不到或不明白如何解决这个问题。任何帮助将不胜感激。

这是我的代码：

import multiprocessing
from functools import partial
import numpy as np
rng=np.random.default_rng()
import time

def sim(data_in_1, data_in_2, data_in_3, batch_num, run_number):

    file_str='run_'+str(run_number)+'.npy'
    temp=rng.choice(1000,5);
    time.sleep(temp[0]/1000)
    
    for i in range(temp[0]):
        a=np.sqrt(temp[1])
        
    return run_number, a, temp

def main():
    processes_num=8
    
    batch_num=1;
    num_of_calculations=8*2
    iterable=range(num_of_calculations);
    data_in_1=20;
    data_in_2=10;
    data_in_3=1;
    pool = multiprocessing.Pool(processes=processes_num)
    func = partial(sim, data_in_1, data_in_2, data_in_3, batch_num)
    results=pool.map(func, iterable)
    for r in results:
        out1=r
        file_str='run_'+str(out1[0])+'.npy'
        with open(file_str, 'wb') as f:
            np.save(f,out1[1])
            np.save(f,out1[2])
        print('saved run '+ str(out1[0]))
    pool.close()
    pool.join()
    
    print('Batch no. '+str(batch_num)+' is finished.')
    
    for run_number in range(num_of_calculations):
        file_str='run_'+str(run_number)+'.npy'
        with open(file_str, 'rb') as f:
            temp=np.load(f)
            temp=np.load(f)
        print('result of run ' + str(run_number) +' is: ' + str(temp))

if __name__ == "__main__":
    main()

窗口输出：

saved run 0
saved run 1
saved run 2
saved run 3
saved run 4
saved run 5
saved run 6
saved run 7
saved run 8
saved run 9
saved run 10
saved run 11
saved run 12
saved run 13
saved run 14
saved run 15
Batch no. 1 is finished.
result of run 0 is: [173 600 438 195 877]
result of run 1 is: [925 710 727 604 759]
result of run 2 is: [883 645 558 875 205]
result of run 3 is: [843 541 597 605 513]
result of run 4 is: [342 439 406 101 192]
result of run 5 is: [472 279 796  99 774]
result of run 6 is: [443 982  49 314 854]
result of run 7 is: [383  45 923 356 156]
result of run 8 is: [344 597 675 615 297]
result of run 9 is: [605 241 523 241 570]
result of run 10 is: [330 457 172 670 130]
result of run 11 is: [ 38 926 902 659 782]
result of run 12 is: [573 150 435 216 765]
result of run 13 is: [178 851 878 155 431]
result of run 14 is: [929 749 730 368 504]
result of run 15 is: [235 310 836 940 701]

Linux 上相同代码的输出：

saved run 0
saved run 1
saved run 2
saved run 3
saved run 4
saved run 5
saved run 6
saved run 7
saved run 8
saved run 9
saved run 10
saved run 11
saved run 12
saved run 13
saved run 14
saved run 15
Batch no. 1 is finished.
result of run 0 is: [  9 218 388 265 856]
result of run 1 is: [  9 218 388 265 856]
result of run 2 is: [  9 218 388 265 856]
result of run 3 is: [  9 218 388 265 856]
result of run 4 is: [  9 218 388 265 856]
result of run 5 is: [  9 218 388 265 856]
result of run 6 is: [  9 218 388 265 856]
result of run 7 is: [  9 218 388 265 856]
result of run 8 is: [715 532 364 775 437]
result of run 9 is: [715 532 364 775 437]
result of run 10 is: [715 532 364 775 437]
result of run 11 is: [715 532 364 775 437]
result of run 12 is: [715 532 364 775 437]
result of run 13 is: [715 532 364 775 437]
result of run 14 is: [715 532 364 775 437]
result of run 15 is: [715 532 364 775 437]

【参考方案1】：

事实证明，随机数生成器种子在 Linux 中是从父级继承的，但在 Windows 上，它会针对每个进程进行更改。这就是为什么我在 Linux 上一直得到相同结果的原因。所以我移动了这条线：

rng=np.random.default_rng()

从全局到函数“sim”内部，这就成功了。