scrapy学习笔记

Posted 2020-10-28 Talk is cheap

scrapy学习笔记

下面以爬取1919网站为例子，完成对一整个网站数据爬取的scrapy项目创建。

创建一个scrapy文件

在任意目录下输入命令

scrapy startproject OneNine (文件名)

将会得到如下目录的文件

OneNine/
    scrapy.cfg            # 部署配置文件

    OneNine/           # Python模块，你所有的代码都放这里面
        __init__.py

        items.py          # Item定义文件

        pipelines.py      # pipelines定义文件

        settings.py       # 配置文件

        spiders/          # 所有爬虫spider都放这个文件夹下面
            __init__.py
            ...

接着创建一个spider文件用来编写爬取规则

cd OneNine
scrape genspider onenine onenine.com 

此时在spiders文件夹下就会生成一个onenine.py文件，我们将在这个文件中编写爬虫规则

定义Item

在items.py文件中需要编写我们要爬取的字段内容。

import scrapy

class OnenineItem(scrapy.Item):
    url = scrapy.Field()
    good_name = scrapy.Field()
    actual_price = scrapy.Field()
    details = scrapy.Field()
    year = scrapy.Field()
    month = scrapy.Field()
    plateform = scrapy.Field()
    cat_lv_one = scrapy.Field()
    cat_lv_two = scrapy.Field()
    shop_id = scrapy.Field()
    shop_name = scrapy.Field()
    shop_area = scrapy.Field()
    shop_province = scrapy.Field()
    shop_city = scrapy.Field()
    good_id = scrapy.Field()
    brand = scrapy.Field()
    size = scrapy.Field()
    percent = scrapy.Field()
    country = scrapy.Field()
    area = scrapy.Field()
    type = scrapy.Field()
    grape_type = scrapy.Field()
    num = scrapy.Field()
    name_price = scrapy.Field()
    bottle_price = scrapy.Field()
    comments = scrapy.Field()
    accumulate_sales = scrapy.Field()
    month_sales = scrapy.Field()
    month_bottle_sales = scrapy.Field()
    month_sale_amounts = scrapy.Field()

scrapy.Field的属性的字段可以直接在后期直接生成你要的文件格式。

spider文件

在spider文件中我们编写了对于网站爬取规则的编写