爬虫（scrapy中的ImagesPipeline）

Posted 2020-10-01 Sip

在使用ImagesPipeline对妹子图网站图片进行下载时，遇到302错误，页面被强制跳转。

解决办法如下：

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don‘t forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
from scrapy.pipelines.images import ImagesPipeline
import scrapy
class MeizituPipeline(object):
    def process_item(self, item, spider):
        return item

class download_pic_pipeline(ImagesPipeline):
    header = {
        ‘User-Agent‘: ‘Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWeb            Kit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.109 Safari/537.36‘,
        ‘Referer‘: ‘http://img.mmjpg.com/‘
    }
    def get_media_requests(self, item, info):
        for i in item[‘image_urls‘]:
            yield scrapy.Request(url= i,headers=self.header)

down_load_pic_pipeline类继承了ImagesPipeline类，我们重写get_meidia_requests方法，在请求图片时，加上一个hearders，多次尝试发现参数Referer是该网站用来反爬的关键点，我们添加这个参数，并设置为http://img.mmjpg.com/   ，运行后状态码为200，问题得到解决！