celery 调用scrapy

Posted 2022-11-14 walkonmars

　　我的环境： celery 3.1.25 python 3.6.9 window10

 

celery tasks 代码如下，其中 QuotesSpider 是我的scrapy项目爬虫类名称

from celery_app import app
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
from tutorial.spiders.quotes import QuotesSpider

def crawl_run():
    scope = ‘all‘
    process = CrawlerProcess(settings=get_project_settings())
    process.crawl(QuotesSpider, scope)
    process.start()
    process.join()

@app.task(queue=‘default‘)
def execute_task():
    return crawl_run()