Python多进程

Posted 2021-02-02 zaorunzi

import os

print(‘Process (%s) start...‘ % os.getpid())
# Only works on Unix/Linux/Mac:
pid = os.fork()
if pid == 0:
    print(‘I am child process (%s) and my parent is %s.‘ % (os.getpid(), os.getppid()))
else:
    print(‘I (%s) just created a child process (%s).‘ % (os.getpid(), pid))

Unix/Linux/Mac操作系统提供fork()系统调用

Process (876) start...
I (876) just created a child process (877).
I am child process (877) and my parent is 876.

Windows下 multiprocessing 

from multiprocessing import Process
import os
#子进程执行的代码
def run_proc(name):
    print(‘Run child process %s (%s)...‘ %(name,os.getpid()))


if __name__ == ‘__main__‘:
    print(‘Parent process %s.‘ % os.getpid())
    p = Process(target=run_proc, args=(‘test‘,)) #创建子进程Process实例
    print(‘Child process will start.‘)
    p.start() #start()方法启动
    p.join() #等待子进程结束后继续往下进行 通常用于进程间的同步
    print(‘Child process end.‘)

Parent process 928.
Child process will start.
Run child process test (929)...
Child process end.

 

Pool 进程池

from multiprocessing import Pool
import os, time, random


def long_time_task(name):
    print(‘Run task %s (%s)...‘ % (name, os.getpid()))
    start = time.time()
    time.sleep(random.random() * 3)
    end = time.time()
    print(‘Task %s runs %0.2f seconds.‘ % (name, (end - start)))


if __name__ == ‘__main__‘:
    print(‘Parent process %s.‘ % os.getpid())
    p = Pool(4)
    # Pool的默认大小为为CPU的核数
    for i in range(5):
        p.apply_async(long_time_task, args=(i,))
    print(‘Waiting for all subprocesses done...‘)
    p.close()
    p.join() 
    # join()会等待所有子进程执行完毕 调用join()前必须执行close()
    # 调用close()后就不能添加新的Process
　　 
    print(‘All subprocesses done.‘)

 

Parent process 1784.
Waiting for all subprocesses done...
Run task 0 (8128)...
Run task 1 (9688)...
Run task 2 (7180)...
Run task 3 (6308)...
Task 2 runs 0.23 seconds.
Run task 4 (7180)...
Task 3 runs 0.62 seconds.
Task 1 runs 1.44 seconds.
Task 0 runs 2.14 seconds.
Task 4 runs 2.32 seconds.
All subprocesses done.

注意输出的结果，task 0，1，2，3是立刻执行的，而task 4要等待前面某个task完成后才执行，
这是因为Pool的默认大小在我的电脑上是4，因此，最多同时执行4个进程。这是Pool有意设计的限制，并不是操作系统的限制。如果改成：
p = Pool(5)
就可以同时跑5个进程

进程间通信

Process之间肯定是需要通信的，操作系统提供了很多机制来实现进程间的通信。Python的multiprocessing模块包装了底层的机制，提供了Queue、Pipes等多种方式来交换数据。

以Queue为例，在父进程中创建两个子进程，一个往Queue里写数据，一个从Queue里读数据：

from multiprocessing import Process, Queue
import os, time, random

# 写数据进程执行的代码:
def write(q):
    print(‘Process to write: %s‘ % os.getpid())
    for value in [‘A‘, ‘B‘, ‘C‘]:
        print(‘Put %s to queue...‘ % value)
        q.put(value)
        time.sleep(random.random())

# 读数据进程执行的代码:
def read(q):
    print(‘Process to read: %s‘ % os.getpid())
    while True:
        value = q.get(True)
        print(‘Get %s from queue.‘ % value)

if __name__==‘__main__‘:
    # 父进程创建Queue，并传给各个子进程：
    q = Queue()
    pw = Process(target=write, args=(q,))
    pr = Process(target=read, args=(q,))
    # 启动子进程pw，写入:
    pw.start()
    # 启动子进程pr，读取:
    pr.start()
    # 等待pw结束:
    pw.join()
    # pr进程里是死循环，无法等待其结束，只能强行终止:
    pr.terminate()

 

Process to write: 50563
Put A to queue...
Process to read: 50564
Get A from queue.
Put B to queue...
Get B from queue.
Put C to queue...
Get C from queue.

 

 

参考 https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001431927781401bb47ccf187b24c3b955157bb12c5882d000sdsd