爬虫神器xpath的用法
Posted 北京流浪儿
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了爬虫神器xpath的用法相关的知识,希望对你有一定的参考价值。
xpath的多线程爬虫
#encoding=utf-8 ‘‘‘ pool = Pool(4) cpu的核数为4核 results = pool.map(爬取函数,网址列表) ‘‘‘ from multiprocessing.dummy import Pool as ThreadPool import requests import time def getsource(url): html = requests.get(url) urls = [] for i in range(1,21): newpage = ‘http://tieba.baidu.com/p/3522395718?pn=‘ + str(i) urls.append(newpage) time1 = time.time() for i in urls: print i getsource(i) time2 = time.time() print u‘单线程耗时:‘ + str(time2-time1) pool = ThreadPool(4) time3 = time.time() results = pool.map(getsource, urls) pool.close() pool.join() time4 = time.time() print u‘并行耗时:‘ + str(time4-time3)
输出:
单线程耗时:12.0818030834
并行耗时:3.58480286598
以上是关于爬虫神器xpath的用法的主要内容,如果未能解决你的问题,请参考以下文章