scrapy 的一个例子

Posted 2020-10-02 蒋乐兴的技术随笔

 1、目标：

　　scrapy 是一个爬虫构架，现用一个简单的例子来讲解，scrapy 的使用步骤

 

2、创建一个scrapy的项目：

　　创建一个叫firstSpider的项目，命令如下：

scrapy startproject firstSpider 

[[email protected] ~]$ scrapy startproject firstSpider 
New Scrapy project ‘firstSpider‘, using template directory ‘/usr/local/python-3.6.2/lib/python3.6/site-packages/scrapy/templates/project‘, created in:
    /home/jianglexing/firstSpider

You can start your first spider with:
    cd firstSpider
    scrapy genspider example example.com

　　

3、创建一个项目时scrapy 命令干了一些什么：

　　创建一个项目时scrapy 会创建一个目录，并向目录中增加若干文件

[[email protected] ~]$ tree firstSpider/
firstSpider/
├── firstSpider
│   ├── __init__.py
│   ├── items.py
│   ├── middlewares.py
│   ├── pipelines.py
│   ├── __pycache__
│   ├── settings.py
│   └── spiders
│       ├── __init__.py
│       └── __pycache__
└── scrapy.cfg

4 directories, 7 files

 

4、进入项目所在的目录并创建爬虫：

[[email protected] ~]$ cd firstSpider/
[[email protected] firstSpider]$ scrapy genspider financeSpider www.financedatas.com
Created spider ‘financeSpider‘ using template ‘basic‘ in module:
  firstSpider.spiders.financeSpider

 

5、一只爬虫在scrapy 项目中对应一个文件：

[[email protected] firstSpider]$ tree ./
./
├── firstSpider
│   ├── __init__.py
│   ├── items.py
│   ├── middlewares.py
│   ├── pipelines.py
│   ├── __pycache__
│   │   ├── __init__.cpython-36.pyc
│   │   └── settings.cpython-36.pyc
│   ├── settings.py
│   └── spiders
│       ├── financeSpider.py    # 这个就是刚才创建的爬虫文件
│       ├── __init__.py
│       └── __pycache__
│           └── __init__.cpython-36.pyc
└── scrapy.cfg

 

6、编写爬虫的处理逻辑：

　　以爬取 http://www.financedatas.com 网站首页的title为例

# -*- coding: utf-8 -*-
import scrapy


class FinancespiderSpider(scrapy.Spider):
    name = ‘financeSpider‘
    allowed_domains = [‘www.financedatas.com‘]
    start_urls = [‘http://www.financedatas.com/‘]

    def parse(self, response):
        """在parse方法中编写处理逻辑"""
        print(‘*‘*64)
        title=response.xpath(‘//title/text()‘).extract() #xpath 语法抽取数据
        print(title)
        print(‘*‘*64)

 

7、运行爬虫，查看效果：

[[email protected] spiders]$ scrapy crawl financeSpider
2017-08-10 16:11:38 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: firstSpider)
2017-08-10 16:11:38 [scrapy.utils.log] INFO: Overridden settings: {‘BOT_NAME‘: ‘firstSpider‘, ‘NEWSPIDER_MODULE‘: ‘firstSpider.spiders‘, ‘ROBOTSTXT_OBEY‘: True, ‘SPIDER_MODULES‘: [‘firstSpider.spiders‘]}
.... ....
2017-08-10 16:11:39 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://www.financedatas.com/robots.txt> (referer: None)
2017-08-10 16:11:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.financedatas.com/> (referer: None)
****************************************************************
[‘欢迎来到 www.financedatas.com‘]   # 这里就抽取到的数据
****************************************************************2017-08-10 16:11:39 [scrapy.core.engine] INFO: Spider closed (finished)

 

 

 

----