Python爬虫，爬取网站图片，详细解释（看完就会）

Posted 2021-06-22 乔治先生

Xpath

 

 解析图片项目

 

 

# 指定url
url = \'http://pic.netbian.com/4kyingshi/\'
# UA伪装
headers = {
    \'User-Agent\': \'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (Khtml, like Gecko) \'
                  \'Chrome/91.0.4472.101 Safari/537.36 Edg/91.0.864.48 \'
}
# 发送请求
response = requests.get(url=url, headers=headers)
# 获取页面文本信息
page_text = response.text
# 获取编码格式
print(response.encoding)
# 解析返回来的内容
tree = etree.HTML(page_text)
# 编写匹配规则
# Xpath规则:获取所有节点下的div class属性值为"slist" 该div下的/ul便签下的/ul标签
li_list = tree.xpath(\'//div[@class="slist"]/ul/li\')
print(li_list)
# 开始遍历获取src属性值
for li in li_list:
    # 拼接图片url
    # Xpath规则:获取当前目录下的/a标签下的img标签选择src属性
    img_url = \'http://pic.netbian.com\'+li.xpath(\'./a/img/@src\')[0]
    # Xpath规则:原理同上
    # 生成文件名
    img_name = li.xpath(\'./a/img/@alt\')[0]
    # 编码
    img_name_list = img_name.encode(\'iso-8859-1\').decode(\'gbk\')
    print(img_url, img_name_list)
    # 发送二次请求，请求图片的src值
    response_content = requests.get(url= img_url, headers=headers).content
    # 持久化处理
    with open(img_name_list+\'.jpg\',\'wb\') as fp:
        fp.write(response_content)