python基础-线程创建线程池进线程异步回调(add_done_callback)进线程数据共享ftp线程池
Posted 易辰_
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了python基础-线程创建线程池进线程异步回调(add_done_callback)进线程数据共享ftp线程池相关的知识，希望对你有一定的参考价值。
线程创建
进程只是用来把资源集中到一起（进程只是一个资源单位，或者说资源集合），而线程才是cpu上的执行单位。
每个进程有一个地址空间，而且默认就有一个控制线程
线程就是一条流水线工作的过程，一条流水线必须属于一个车间，一个车间的工作过程是一个进程
多线程（即多个控制线程）的概念是，在一个进程中存在多个控制线程，多个控制线程共享该进程的地址空间，相当于一个车间内有多条流水线，都共用一个车间的资源
我们之前了解过进程的2种创建方式
下面的代码是2种创建线程的方式
from threading import Thread
from multiprocessing import Process
import time,os
def task():
print('%s is running' %os.getpid())
time.sleep(2)
print('%s is done' %os.getpid())
class Mythread(Thread):
def __init__(self,name):
super().__init__()
self.name=name
def run(self):
print('%s is running' % os.getpid())
time.sleep(5)
print('%s is done' % os.getpid())
if __name__ == '__main__':
t=Thread(target=task)
# t=Mythread('xxxxx')
t.start()
print('主')输出如下：
E:\\python\\python_sdk\\python.exe "E:/python/py_pro/1 开启线程的两种方式.py"
10336 is running
主
10336 is done
Process finished with exit code 0
线程进程pid
part1:在主进程下开启多个线程,每个线程都跟主进程的pid一样
from threading import Thread
from multiprocessing import Process
import time,os
def task():
print('partent：%s self：%s' %(os.getppid(),os.getpid()))
time.sleep(5)
if __name__ == '__main__':
t=Thread(target=task,)
# t=Process(target=task,)
t.start()
print('主',os.getppid(),os.getpid())
输出如下：
partent：9052 self：10120
主 9052 10120开多个进程,每个进程都有不同的pid
from threading import Thread
from multiprocessing import Process
import time,os
def task():
print('partent：%s self：%s' %(os.getppid(),os.getpid()))
time.sleep(5)
if __name__ == '__main__':
t=Process(target=task,)
t.start()
print('主',os.getppid(),os.getpid())输出如下：
主 9052 2668
partent：2668 self：8744线程进程数据共享
进程之间数据不共享，但是进程之间可以通过ipc进行数据通讯
from threading import Thread
from multiprocessing import Process
import time,os
n=100
def task():
global n
n=0
if __name__ == '__main__':
t=Process(target=task,)
t.start()
t.join()
print('主',n)输出如下：
主 100线程之间内存空间共享
from threading import Thread
import time,os
n=100
def task():
global n
n=0
if __name__ == '__main__':
t=Thread(target=task,)
t.start()
t.join()
print('主',n)输出如下：
主 0线程ftp
服务端：
import multiprocessing
import threading
import socket
s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.bind(('127.0.0.1',8081))
s.listen(5)
def action(conn):
while True:
data=conn.recv(1024)
print(data)
conn.send(data.upper())
if __name__ == '__main__':
while True:
conn,addr=s.accept()
p=threading.Thread(target=action,args=(conn,))
p.start()客户端：
from socket import *
client=socket(AF_INET,SOCK_STREAM)
client.connect(('127.0.0.1',8081))
while True:
msg=input('>>: ').strip()
if not msg:continue
client.send(msg.encode('utf-8'))
msg=client.recv(1024)
print(msg.decode('utf-8'))线程池
from concurrent.futures import ProcessPoolExecutor,ThreadPoolExecutor
from threading import current_thread
import time,random
def task(n):
print('%s is running' %current_thread().getName())
time.sleep(random.randint(1,3))
return n**2
if __name__ == '__main__':
t=ThreadPoolExecutor(3) #默认是cpu的核数*5
objs=[]
for i in range(5):
obj=t.submit(task,i)
objs.append(obj)
t.shutdown(wait=True)
for obj in objs:
print(obj.result())
print('主',current_thread().getName())输出如下：
E:\\python\\python_sdk\\python.exe "E:/python/py_pro/4 线程池.py"
ThreadPoolExecutor-0_0 is running
ThreadPoolExecutor-0_1 is running
ThreadPoolExecutor-0_2 is running
ThreadPoolExecutor-0_0 is running
ThreadPoolExecutor-0_1 is running
0
1
4
9
16
主 MainThread
线程池ftp
服务端：
from socket import *
from concurrent.futures import ThreadPoolExecutor
import os
server=socket(AF_INET,SOCK_STREAM)
server.setsockopt(SOL_SOCKET,SO_REUSEADDR,1)
server.bind(('127.0.0.1',8080))
server.listen(5)
def talk(conn,client_addr):
print('进程pid: %s' %os.getpid())
while True:
try:
msg=conn.recv(1024)
if not msg:break
conn.send(msg.upper())
except Exception:
break
if __name__ == '__main__':
p=ThreadPoolExecutor(5)
while True:
conn,client_addr=server.accept()
p.submit(talk,conn,client_addr)
客户端：
from socket import *
client=socket(AF_INET,SOCK_STREAM)
client.connect(('127.0.0.1',8081))
while True:
msg=input('>>: ').strip()
if not msg:continue
client.send(msg.encode('utf-8'))
msg=client.recv(1024)
print(msg.decode('utf-8'))线程的一些其他方法
from threading import Thread,current_thread,enumerate,active_count
import time,os
def task():
print('%s is running' %current_thread().getName())
time.sleep(5)
print('%s is done' %current_thread().getName())
if __name__ == '__main__':
t=Thread(target=task,name='xxxx')
t.start()
print(t.name)
#查看当前活着的线程
print(enumerate()[0].getName())
print(active_count())
print('主',current_thread().getName())
print()
输出如下：
E:\\python\\python_sdk\\python.exe "E:/python/py_pro/3 线程对象的其他属性或方法.py"
xxxx is running
xxxx
MainThread
2
主 MainThread
xxxx is done
异步-回调函数
ProcessPoolExecutor方式
我们之前总结的异步返回结果没有用到调用函数，接下来的是利用了回调函数
#pip install requests
import requests
from concurrent.futures import ThreadPoolExecutor,ProcessPoolExecutor
from threading import current_thread
import time,os
def get(url):
print('%s GET %s' %(os.getpid(),url))
response=requests.get(url)
time.sleep(3)
if response.status_code == 200:
return 'url':url,'text':response.text
def parse(obj):
res=obj.result()
print('[%s] <%s> (%s)' % (os.getpid(), res['url'],len(res['text'])))
if __name__ == '__main__':
urls = [
'https://www.python.org',
'https://www.baidu.com',
'https://www.jd.com',
'https://www.tmall.com',
]
t=ProcessPoolExecutor(2)
for url in urls:
t.submit(get,url).add_done_callback(parse)
t.shutdown(wait=True)
print('主',os.getpid())代码思路是：
t=ProcessPoolExecutor(2)开一个进程池，然后去并发下载网络数据，下载完毕后，
在主进程中add_done_callback去解析
这里由于主进程、子进程不是同一个进程空间，所以在解析数据时候，在主进程
输出如下：
E:\\python\\python_sdk\\python.exe "E:/python/py_pro/5 补充异步的概念.py"
5628 GET https://www.python.org
4816 GET https://www.baidu.com
4816 GET https://www.jd.com
[3204] <https://www.baidu.com> (2443)
[3204] <https://www.python.org> (48856)
5628 GET https://www.tmall.com
[3204] <https://www.jd.com> (124541)
[3204] <https://www.tmall.com> (212080)
主 3204
Process finished with exit code 0
ThreadPoolExecutor方式
import requests
from concurrent.futures import ThreadPoolExecutor,ProcessPoolExecutor
from threading import current_thread
import time
import os
def get(url):
print('%s GET %s,%s' %(current_thread().getName(),os.getpid(),url))
response=requests.get(url)
time.sleep(3)
if response.status_code == 200:
return 'url':url,'text':response.text
def parse(obj):
res=obj.result()
print('[%s] <%s> (%s)' % (current_thread().getName(), res['url'],len(res['text'])))
if __name__ == '__main__':
urls = [
'https://www.python.org',
'https://www.baidu.com',
'https://www.jd.com',
'https://www.tmall.com',
]
t=ThreadPoolExecutor(2)
for url in urls:
t.submit(get,url).add_done_callback(parse)
t.shutdown(wait=True)
print('主',current_thread().getName(),os.getpid())代码思路是：
t=ThreadPoolExecutor(2)开一个线程池，然后去并发下载网络数据，下载完毕后，
在主线程程中add_done_callback去解析
这里由于主线程、子线程是同一个进程空间，所以在解析数据时候，可能主线程、子线程都会解析
输出如下：
E:\\python\\python_sdk\\python.exe "E:/python/py_pro/5 补充异步的概念.py"
ThreadPoolExecutor-0_0 GET 12956,https://www.python.org
ThreadPoolExecutor-0_1 GET 12956,https://www.baidu.com
[ThreadPoolExecutor-0_1] <https://www.baidu.com> (2443)
ThreadPoolExecutor-0_1 GET 12956,https://www.jd.com
[ThreadPoolExecutor-0_0] <https://www.python.org> (48856)
ThreadPoolExecutor-0_0 GET 12956,https://www.tmall.com
[ThreadPoolExecutor-0_1] <https://www.jd.com> (124541)
[ThreadPoolExecutor-0_0] <https://www.tmall.com> (212079)
主 MainThread 12956
Process finished with exit code 0
以上是关于python基础-线程创建线程池进线程异步回调(add_done_callback)进线程数据共享ftp线程池的主要内容，如果未能解决你的问题，请参考以下文章