多处理：池：直接迭代器或使用变量来存储迭代器

Posted 2023-02-16

【中文标题】多处理：池：直接迭代器或使用变量来存储迭代器

【英文标题】：Multiprocessing : Pool : Direct iterator OR use of variable to store the iterator

【发布时间】：2021-08-15 06:00:21

【问题描述】：

有没有区别：

  with Pool(...) as pool :
      res = pool.imap_unordered(fun, iterator)
      for val in res :
           ........

和..

  with Pool(...) as pool :
      for val in pool.imap_unordered(fun, iterator) :
           ........

是第二个变种更快！

【参考方案1】：

以下是两种情况的反汇编代码：

from multiprocessing import Pool

def fun(x):
    pass


def method1(it):
    with Pool() as pool:
        res = pool.imap_unordered(fun, it)
        for val in res:
            pass

def method2(it):
    with Pool() as pool:
        for val in pool.imap_unordered(fun, it):
            pass

import dis
dis.dis(method1)
print()
print('-' * 80)
print()
dis.dis(method2)

打印：

  8           0 LOAD_GLOBAL              0 (Pool)
              2 CALL_FUNCTION            0
              4 SETUP_WITH              28 (to 34)
              6 STORE_FAST               1 (pool)

  9           8 LOAD_FAST                1 (pool)
             10 LOAD_METHOD              1 (imap_unordered)
             12 LOAD_GLOBAL              2 (fun)
             14 LOAD_FAST                0 (it)
             16 CALL_METHOD              2
             18 STORE_FAST               2 (res)

 10          20 LOAD_FAST                2 (res)
             22 GET_ITER
        >>   24 FOR_ITER                 4 (to 30)
             26 STORE_FAST               3 (val)

 11          28 JUMP_ABSOLUTE           24
        >>   30 POP_BLOCK
             32 BEGIN_FINALLY
        >>   34 WITH_CLEANUP_START
             36 WITH_CLEANUP_FINISH
             38 END_FINALLY
             40 LOAD_CONST               0 (None)
             42 RETURN_VALUE

--------------------------------------------------------------------------------

 14           0 LOAD_GLOBAL              0 (Pool)
              2 CALL_FUNCTION            0
              4 SETUP_WITH              24 (to 30)
              6 STORE_FAST               1 (pool)

 15           8 LOAD_FAST                1 (pool)
             10 LOAD_METHOD              1 (imap_unordered)
             12 LOAD_GLOBAL              2 (fun)
             14 LOAD_FAST                0 (it)
             16 CALL_METHOD              2
             18 GET_ITER
        >>   20 FOR_ITER                 4 (to 26)
             22 STORE_FAST               2 (val)

 16          24 JUMP_ABSOLUTE           20
        >>   26 POP_BLOCK
             28 BEGIN_FINALLY
        >>   30 WITH_CLEANUP_START
             32 WITH_CLEANUP_FINISH
             34 END_FINALLY
             36 LOAD_CONST               0 (None)
             38 RETURN_VALUE

结果

无论迭代器的长度或fun 完成的处理量或处理@ 返回的结果如何，method1 似乎都会导致执行另外两个字节码指令（在位置 18 和 20） 987654325@。换句话说，如此微不足道的差异值得关注。

【讨论】：

哇，谢谢...我很困惑，因为似乎方法 1 首先等待获得结果然后执行循环，即阻塞。我不知道这两种情况 imap 和循环交错/异步