Python笔记

Posted 2020-10-16 哈哈丶Stupid

最近几天在研究爬虫爬取音频网站如何高效的访问并下载，说实话，我一开始还不知道有协程这个东东~~~并且之前一直觉得爬网站用啥方法都一样，能爬就行，自从发现了协程，爱不释手~~~~~

 

好了废话少说~进入正题：

首先，我们先说一下什么是多线程，在网上的一些教程中都给出这个例子（涉及类和队列）

 

 1 #Example.py
 2 \'\'\'
 3 Standard Producer/Consumer Threading Pattern
 4 \'\'\'
 5 
 6 import time 
 7 import threading 
 8 import Queue 
 9 
10 class Consumer(threading.Thread): 
11     def __init__(self, queue): 
12         threading.Thread.__init__(self)
13         self._queue = queue 
14 
15     def run(self):
16         while True: 
17             # queue.get() blocks the current thread until 
18             # an item is retrieved. 
19             msg = self._queue.get() 
20             # Checks if the current message is 
21             # the "Poison Pill"
22             if isinstance(msg, str) and msg == \'quit\':
23                 # if so, exists the loop
24                 break
25             # "Processes" (or in our case, prints) the queue item   
26             print "I\'m a thread, and I received %s!!" % msg
27         # Always be friendly! 
28         print \'Bye byes!\'
29 
30 
31 def Producer():
32     # Queue is used to share items between
33     # the threads.
34     queue = Queue.Queue()
35 
36     # Create an instance of the worker
37     worker = Consumer(queue)
38     # start calls the internal run() method to 
39     # kick off the thread
40     worker.start() 
41 
42     # variable to keep track of when we started
43     start_time = time.time() 
44     # While under 5 seconds.. 
45     while time.time() - start_time < 5: 
46         # "Produce" a piece of work and stick it in 
47         # the queue for the Consumer to process
48         queue.put(\'something at %s\' % time.time())
49         # Sleep a bit just to avoid an absurd number of messages
50         time.sleep(1)
51 
52     # This the "poison pill" method of killing a thread. 
53     queue.put(\'quit\')
54     # wait for the thread to close down
55     worker.join()
56 
57 
58 if __name__ == \'__main__\':
59     Producer()

 

这个栗子让我想起了大三的课设虽然用的是java实现生产者-消费者模式~~~万恶的java

第一点你需要一个model类，其次需要一个队列来传递对象，是的如果你需要双向通信的话则需要再加入一个一个队列。。

 

下面，我们需要一个worker线程池，在网页检索中通过多线程加速~~~

 1 #Example2.py
 2 \'\'\'
 3 A more realistic thread pool example 
 4 \'\'\'
 5 
 6 import time 
 7 import threading 
 8 import Queue 
 9 import urllib2 
10 
11 class Consumer(threading.Thread): 
12     def __init__(self, queue): 
13         threading.Thread.__init__(self)
14         self._queue = queue 
15 
16     def run(self):
17         while True: 
18             content = self._queue.get() 
19             if isinstance(content, str) and content == \'quit\':
20                 break
21             response = urllib2.urlopen(content)
22         print \'Bye byes!\'
23 
24 
25 def Producer():
26     urls = [
27         \'http://www.python.org\', \'http://www.yahoo.com\'
28         \'http://www.scala.org\', \'http://www.google.com\'
29         # etc.. 
30     ]
31     queue = Queue.Queue()
32     worker_threads = build_worker_pool(queue, 4)
33     start_time = time.time()
34 
35     # Add the urls to process
36     for url in urls: 
37         queue.put(url)  
38     # Add the poison pillv
39     for worker in worker_threads:
40         queue.put(\'quit\')
41     for worker in worker_threads:
42         worker.join()
43 
44     print \'Done! Time taken: {}\'.format(time.time() - start_time)
45 
46 def build_worker_pool(queue, size):
47     workers = []
48     for _ in range(size):
49         worker = Consumer(queue)
50         worker.start() 
51         workers.append(worker)
52     return workers
53 
54 if __name__ == \'__main__\':
55     Producer()

可怕，我写不下去了，太复杂了。

我们换一个方法~~~map

1 urls = [\'http://www.yahoo.com\', \'http://www.reddit.com\']

 2 results = map(urllib2.urlopen, urls) 

对，你没有看错，直接就可以了，result直接遍历urls然后返回的结果存在里面类型是list。

map直接可以实现并行操作了

 

 

 线程过多，切换时所消耗的时间甚至超过实际工作时间，对于实际工作需求，通过尝试来找到合适线程池大小最合适不过了

 1 import urllib2 
 2 from multiprocessing.dummy import Pool as ThreadPool 
 3 
 4 urls = [
 5     \'http://www.python.org\', 
 6     \'http://www.python.org/about/\',
 7     \'http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html\',
 8     \'http://www.python.org/doc/\',
 9     \'http://www.python.org/download/\',
10     \'http://www.python.org/getit/\',
11     \'http://www.python.org/community/\',
12     \'https://wiki.python.org/moin/\',
13     \'http://planet.python.org/\',
14     \'https://wiki.python.org/moin/LocalUserGroups\',
15     \'http://www.python.org/psf/\',
16     \'http://docs.python.org/devguide/\',
17     \'http://www.python.org/community/awards/\'
18     # etc.. 
19     ]
20 
21 # Make the Pool of workers
22 pool = ThreadPool(4) 
23 # Open the urls in their own threads
24 # and return the results
25 results = pool.map(urllib2.urlopen, urls)
26 #close the pool and wait for the work to finish 
27 pool.close() 
28 pool.join() 

其中最关键只有一行，map函数轻易取代前文40多行复杂的例子。

 说了这么多，我要讲的东西还没说~~~无语

好了，我要开始了~~~

我用了半天时间研究了一下喜马拉雅听书的网站，并成功写了一个单进程的小爬虫，下载所需要的类型的全部音频资源~~~~听起来很酷，其实吧爬的时候真的慢，于是乎我就加入了协程这个东西，发现速度上去了一大截，其实就是缩短了访问的时间我主要加了两个协程

 1 #第一个协程
 2 def get_url():
 3     try:
 4         start_urls = [\'http://www.ximalaya.com/dq/book-%E6%82%AC%E7%96%91/{}/\'.format(pn) for pn in range(1,85)]
 5         print(start_urls)
 6         urls_list = []
 7         for start_url in start_urls:
 8             urls_list.append(start_url)
 9             # response = fetch_url_text(start_url)
10             # soup  = BeautifulSoup(response,\'lxml\')
11             # print(soup)
12             # break
13             # for item in soup.find_all(\'div\',class_=\'albumfaceOutter\'):
14             #     print(item)
15             #     href = item.a[\'href\']
16             #     title = item.img[\'alt\']
17             #     img_url = item.img[\'src\']
18             #     # print(title)
19             #     content = {
20             #         \'href\':href,
21             #         \'title\':title,
22             #         \'img_url\':img_url
23             #     }
24             #     get_mp3(href,title)
25         #协程模块1
26         jobs = [gevent.spawn(fetch_url_text,url) for url in urls_list]
27         gevent.joinall(jobs)
28         [job.value for job in jobs]
29         for response in  [job.value for job in jobs]:
30             soup = BeautifulSoup(response,\'lxml\')
31             for item in soup.find_all(\'div\',class_=\'albumfaceOutter\'):
32                 print(item)
33                 href = item.a[\'href\']
34                 title = item.img[\'alt\']
35                 img_url = item.img[\'src\']
36                 # print(title)
37                 content = {
38                     \'href\':href,
39                     \'title\':title,
40                     \'img_url\':img_url
41                 }
42                 get_mp3(href,title)
43     except Exception as e:
44        print(e)
45     return \'\'

第二个协程

 1 def get_mp3(url,title):
 2     response = fetch_url_text(url)
 3     num_list = etree.HTML(response).xpath(\'//div[@class="personal_body"]/@sound_ids\')[0].split(\',\')
 4     print(num_list)
 5 
 6     mkdir(title)
 7     os.chdir(r\'F:\\xmly\\\\\'+title)
 8     ii=1
 9     list_ids = []
10     for id in num_list:
11         list_ids.append(id)
12         # print(id)
13         # # json_url = \'http://www.ximalaya.com/tracks/{}.json\'.format(id)
14         # html = fetch_json(id)
15         # # print(html)
16         # mp3_url = html.get(\'play_path\')
17         # # print(mp3_url)
18         # # download(mp3_url)
19         # content = requests.get(mp3_url, headers=headers).content
20         # name = title + \'_%i.m4a\'%ii
21         # with open(name, \'wb\') as file:
22         #     file.write(content)
23         # print("{} download is ok".format(mp3_url))
24         # ii+=1
25     #协程模块2
26     jobs = [gevent.spawn(fetch_json,id) for id in list_ids]
27     gevent.joinall(jobs)
28     [job.value for job in jobs]
29     for html in [job.value for job in jobs]:
30         mp3_url = html.get(\'play_path\')
31         content = requests.get(mp3_url, headers=headers).content
32         name = title + \'_%i.m4a\' % ii
33         with open(name, \'wb\') as file:
34             file.write(content)
35         print("{} download is ok".format(mp3_url))
36         ii += 1

代码很冗余啊~~~看看就好了，毕竟刚学，现学现用嘛~~~~

其实吧，我感觉速度还是不够

可以再添加多进程模块进去，速度可能又会上去，如果再分布式呢？岂不是分分钟爬完整个站的内容，哈哈

好了，不做白日梦了，我只有一个笔记本，而且实验室服务器就两个，并且都在跑训练模型，就不给他们添加负担了~~~~~

还没说协程是什么啊~~~~~

协程可以理解为绿色的线程，或者微线程，起作用是在执行function_a()时可以随时中断去执行function_b(),然后中断继续function_a()，可自由切换。整个过程看似多线程，然而只有一个线程执行，也就是gevent中的monkeyAPI实现了异步的过程~~~~好神奇是不是

 1 #! -*- coding:utf-8 -*-
 2 #version：2.7
 3 import gevent
 4 from gevent import monkey;monkey.patch_all()
 5 import urllib2
 6 def get_body(i):
 7     print "start",i
 8     urllib2.urlopen("http://cn.bing.com")
 9     print "end",i
10 tasks=[gevent.spawn(get_body,i) for i in range(3)]
11 gevent.joinall(tasks)

这个简单栗子应该很容易明白~~~~~

我总结了一下，只要是有for而且其中每次都需要大量时间做请求的程序都可以调用协程来完成~~~~~

我的理解可能有偏差，欢迎指正~~~~

总结一下，并发和并行吧~~~~~

并发，就相当于有两个队伍，一个caffe机，轮流用这个caffe机

并行，就相当于有两个队伍，两个coffe机，一个队伍用一个，且同时进行。~~~~~

是不是很浅显~~~~~

拜拜~~~~~~~~~~