Python简单的爬虫

Posted 2020-10-07

Python3 的requests的requests 库

1 安装：

　　在 配好python的基础上，在dos命令框中，使用 pip install requests 就行了

2 演示：

　　python 用requests的get的方法爬取链接是很简单的，代码如下:

　　

import requests  
url=‘https://item.jd.com/5188000.html‘
try:
    r=requests.get(url)
    r.raise_for_status()
    r.encoding=r.apparent_encoding
    print(r.text[:1000])
except:
    print(‘爬取失败！‘)

这只是网页链接的一个简单爬取，但是有的网页有很强的反爬防护，用python的requests库进行爬取的时候会被识别出来，从而报错，爬取失败，在这里我们就需要改下程序所用的ua，改完后，便可继续爬取

import requests
url=‘https://www.amazon.cn/dp/B01JFUFPLY?_encoding=UTF8&ref_=pc_cxrd_658390051_recTab_658390051_t_4&pf_rd_p=7e00fee6-4e12-48f0-b4af-b99068b52067&pf_rd_s=merchandised-search-4&pf_rd_t=101&pf_rd_i=658390051&pf_rd_m=A1AJ19PSB66TGU&pf_rd_r=9C3VDEY9YXT1C6242V9H&pf_rd_r=9C3VDEY9YXT1C6242V9H&pf_rd_p=7e00fee6-4e12-48f0-b4af-b99068b52067‘
try:
    kv={‘user-agent‘:‘Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-US)‘}
    r=requests.get(url,headers=kv)
    r.raise_for_status
    r.encoding=r.apparent_encoding
    print(r.text[1000:2000])
except:
    print(‘爬取失败！！‘)

一些主要浏览器的UA

safari 5.1 – MAC
User-Agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50
safari 5.1 – Windows
User-Agent:Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50
IE 9.0
User-Agent:Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;
IE 8.0
User-Agent:Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)
IE 7.0
User-Agent:Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)
IE 6.0
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Firefox 4.0.1 – MAC
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
Firefox 4.0.1 – Windows
User-Agent:Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
Opera 11.11 – MAC
User-Agent:Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11
Opera 11.11 – Windows
User-Agent:Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11
Chrome 17.0 – MAC
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11