scrapy错误汇总

Posted qiaoer1993

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了scrapy错误汇总相关的知识,希望对你有一定的参考价值。

爬虫开始
<pymysql.connections.Connection object at 0x0000012D63A1EFD0>
2019-05-02 09:56:21 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-05-02 09:56:21 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-05-02 09:56:24 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://m.kxzhijia.com/en/pg.jsp?entry=mallNav&pgs=1> from <GET http://m.kxzhijia.com/pg.jsp?entry=mallNav&pgs=1>
2019-05-02 09:56:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://m.kxzhijia.com/en/pg.jsp?entry=mallNav&pgs=1> (referer: None)
finished
2019-05-02 09:56:30 [scrapy.core.engine] INFO: Closing spider (finished)
2019-05-02 09:56:30 [chyp_tj] INFO: 结束(正常):chyp_tj

  没有请求头headers时就会这样报错,例如:

‘DEFAULT_REQUEST_HEADERS‘ : {
‘User-Agent‘: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (Khtml, like Gecko) Chrome/73.0.3683.103 Safari/537.36‘,
}

以上是关于scrapy错误汇总的主要内容,如果未能解决你的问题,请参考以下文章

scrapy主动退出爬虫的代码片段(python3)

scrapy按顺序启动多个爬虫代码片段(python3)

Scrapy Spider没有返回所有元素

Scrapy输出文件格式问题汇总

windows 安装Scrapy常见错误总结

Scrapy FormRequest返回400错误代码