scrapy Data flow

Posted 无花1

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了scrapy Data flow相关的知识,希望对你有一定的参考价值。

The data flow in Scrapy is controlled by the execution engine, and goes like this:
1. The Engine gets the initial Requests to crawl from the Spider.
2. The Engine schedules the Requests in the Scheduler and asks for the next Requests to crawl.
3. The Scheduler returns the next Requests to the Engine.
4. The Engine sends the Requests to the Downloader, passing through the Downloader Middlewares (see
process_request()).
5. Once the page finishes downloading the Downloader generates a Response (with that page) and sends it to the
Engine, passing through the Downloader Middlewares (see process_response()).
6. The Engine receives the Response from the Downloader and sends it to the Spider for processing, passing
through the Spider Middleware (see process_spider_input()).
7. The Spider processes the Response and returns scraped items and new Requests (to follow) to the Engine,
passing through the Spider Middleware (see process_spider_output()).
8. The Engine sends processed items to Item Pipelines, then send processed Requests to the Scheduler and asks
for possible next Requests to crawl.
9. The process repeats (from step 1) until there are no more requests from the Scheduler.

以上是关于scrapy Data flow的主要内容,如果未能解决你的问题,请参考以下文章

[Project] Simulate HTTP Post Request to obtain data from Web Page by using Python Scrapy Framework(代

scrapy 配置文件指定如何导出数据

scrapy相关整理

Scrapy 非常基本的例子

爬虫2.2-scrapy框架-文件写入

scrapy_redis 相关