scrapy

Posted 2020-12-01 xupanfeng

__author__ = ‘Administrator‘
# -*- encoding:utf-8 -*-
import scrapy
class QuoteSpider(scrapy.Spider):
    name = ‘poxiao‘
    start_urls=[‘https://www.poxiao.com/type/movie/‘]
    def parse(self, response):#固定的
        quotes=response.xpath(‘//li/h3‘)#内容
        for quote in quotes:
            yield {
                ‘name‘:quote.xpath(‘./a/text()‘).extract_first(),
                ‘author‘:‘https://www.poxiao.com‘+quote.xpath(‘./a/@href‘).extract_first()
            }
            next_page=response.xpath(‘//div[@class="list-pager"]/a[last()-1]/@href‘).extract_first()
            if next_page:
                yield response.follow(next_page,self.parse)

用SCRAPY爬取某网页链接地址

scrapy runspider ***.py  运行此工程

SCRAPY runspider ***.py -o aa.json      保存成JSON文件

scrap runspider ***.py -o aa.csv -t csv    保存成EXCEL