自定义 scrapy 爬虫的 requests

Posted 2020-08-24 To be a better Programer...

之前使用 scrapy 抓取数据的时候 ，默认是在逻辑中判断是否执行下一次请求

def parse(self):
    # 获取所有的url，例如获取到urls中
    for url in urls:
        yield Request(url)

比如：

def parse(self,response):
    item = MovieItem()
    selector = Selector(response)
    Movies = selector.xpath(‘//div[@class="info"]‘)
    for eachMoive in Movies:
        title = eachMoive.xpath(‘div[@class="hd"]/a/span/text()‘).extract()
        star = eachMoive.xpath(‘div[@class="bd"]/div[@class="star"]/span/em/text()‘).extract()[0]
        quote = eachMoive.xpath(‘div[@class="bd"]/p[@class="quote"]/span/text()‘).extract()
        nextLink = selector.xpath(‘//span[@class="next"]/link/@href‘).extract()
    #下一页
    if nextLink:
        nextLink = nextLink[0]
        yield Request(self.url + nextLink,callback=self.parse)

今天无意查看了 scrapy 的官方文档，可以使用 start_requests() 这个方法循环生成要爬取的网址

  def start_requests(self):
        urls=[]
        for i in range(1,10):
            url=‘http://www.test.com/?page=%s‘%i
            page=scrapy.Request(url)
            urls.append(page)
        return urls

使用 python 一定要简单粗暴，于是把我把之前代码换了如下方式

    # 开始URL
    start_urls = [
      "http://q.stock.sohu.com"
    ]

    #定义爬取的URL
    def start_requests(self):
        # 按日
        return [Request(("http://q.stock.sohu.com/hisHq?code=cn_{0}"+"&start=" + self.begin_date + "&end=" + self.end_date + "&stat=1&order=D&period=d&rt=json&r=0.6618998353094041&0.8423532517054869").format(x[‘code‘])) for x in self.stock_basics]

注意：要注意的是重写 start_requests 这个方法，则不需要设置 start_urls 了 ，并且写了 start_urls 也没有用

This method must return an iterable with the first Requests to crawl for this spider.
This is the method called by Scrapy when the spider is opened for scraping when no particular URLs are specified. If particular URLs are specified, the make_requests_from_url() is used instead to create the Requests. This method is also called only once from Scrapy, so it’s safe to implement it as a generator.
The default implementation uses make_requests_from_url() to generate Requests for each url in start_urls.

REFER:
http://doc.scrapy.org/en/latest/topics/spiders.html#scrapy.spiders.Spider.start_requests
python爬虫----（scrapy框架提高（1），自定义Request爬取）
https://my.oschina.net/lpe234/blog/342741