爬虫神器xpath的用法

Posted 北京流浪儿

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了爬虫神器xpath的用法相关的知识,希望对你有一定的参考价值。

xpath的多线程爬虫

#encoding=utf-8
‘‘‘
pool = Pool(4) cpu的核数为4核
results = pool.map(爬取函数,网址列表)
‘‘‘
from multiprocessing.dummy import Pool as ThreadPool
import requests
import time

def getsource(url):
    html = requests.get(url)

urls = []

for i in range(1,21):
    newpage = http://tieba.baidu.com/p/3522395718?pn= + str(i)
    urls.append(newpage)

time1 = time.time()
for i in urls:
    print i
    getsource(i)
time2 = time.time()
print u单线程耗时: + str(time2-time1)

pool = ThreadPool(4)
time3 = time.time()
results = pool.map(getsource, urls)
pool.close()
pool.join()
time4 = time.time()
print u并行耗时: + str(time4-time3)

输出:

单线程耗时:12.0818030834
并行耗时:3.58480286598

以上是关于爬虫神器xpath的用法的主要内容,如果未能解决你的问题,请参考以下文章

Python爬虫系列之 xpath:html解析神器

爬虫Xpath高级用法

爬虫神器XPath,程序员带你免费获取周星驰等明星热门电影

Python爬虫杂记 - Xpath高级用法

Python爬虫利器三之Xpath语法与lxml库的用法

Python爬虫利器三之Xpath语法与lxml库的用法