ScrapingHub部署失败

Posted 2023-03-01

我试图部署到ScrapingHub，这里是我得到的错误......

Deploy log last 30 lines:
  File "/app/python/lib/python3.8/site-packages/scrapy/cmdline.py", line 142, in execute
    cmd.crawler_process = CrawlerProcess(settings)
  File "/app/python/lib/python3.8/site-packages/scrapy/crawler.py", line 280, in __init__
    super(CrawlerProcess, self).__init__(settings)
  File "/app/python/lib/python3.8/site-packages/scrapy/crawler.py", line 152, in __init__
    self.spider_loader = self._get_spider_loader(settings)
  File "/app/python/lib/python3.8/site-packages/scrapy/crawler.py", line 146, in _get_spider_loader
    return loader_cls.from_settings(settings.frozencopy())
  File "/app/python/lib/python3.8/site-packages/scrapy/spiderloader.py", line 60, in from_settings
    return cls(settings)
  File "/app/python/lib/python3.8/site-packages/scrapy/spiderloader.py", line 24, in __init__
    self._load_all_spiders()
  File "/app/python/lib/python3.8/site-packages/scrapy/spiderloader.py", line 46, in _load_all_spiders
    for module in walk_modules(name):
  File "/app/python/lib/python3.8/site-packages/scrapy/utils/misc.py", line 77, in walk_modules
    submod = import_module(fullpath)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/app/__main__.egg/sstkscrape/spiders/sstkscrape_spider.py", line 4, in <module>
  File "/app/__main__.egg/sstkscrape/spiders/sstkscrape_spider.py", line 7, in QuoteSpider
FileNotFoundError: [Errno 2] No such file or directory: 'urls-test.csv'
"message": "shub-image-info exit code: 1", "details": null, "error": "image_info_error"

"status": "error", "message": "Internal error"
Deploy log location: /var/folders/p7/nwmq6_4138n6t3w2spdnpzfm0000gn/T/shub_deploy_n9zmxfto.log
Error: Deploy failed: b'"status": "error", "message": "Internal error"'

我有一个本地的csv文件，我试图包括在部署和做打开的csv文件，想知道如果这是导致的问题？

答案

你需要使用 package_data setup.py文件的部分。部署非代码文件