pyspider 启动错误遇到的一些坑

Posted 2021-02-12 yuyu666

https://blog.csdn.net/SiHann/article/details/88239892

突然接到一个项目是关于pyspider，遇到了一些小坑，百度一下发现并没有很好的解决所以研究了一下。
我所使用的python版本是3.6.8，因为pyspider是对pip有版本要求的，所以自动升级一下。

升级pip
•# pip install –upgrade pip

一切配置好之后，就在CMD中运行 pyspider all 来看能否跑起来。
这是成功的样子，说明可以了。

不过路上谁能不遇坑，人家各种成功，到了我这里就是各种错误，我就把这些错误整合一下。

首先第一个错误：python3.7不兼容pyspider问题(出现占用关键字的问题)
Python 3.5中引入了async和await，它们在Python 3.7中成为关键字。
所以需要替换一下关键字。

第二个问题也就是我遇到的问题：[[email protected] python]# pyspider all
[W 180629 07:08:26 run:413] phantomjs not found, continue running without it.
[I 180629 07:08:29 result_worker:49] result_worker starting…
[I 180629 07:08:31 processor:211] processor starting…
[I 180629 07:08:31 tornado_fetcher:638] fetcher starting…
[I 180629 07:08:31 scheduler:675] scheduler starting…
[I 180629 07:08:31 scheduler:614] in 5m: new:0,success:0,retry:0,failed:0
[I 180629 07:08:31 scheduler:810] scheduler.xmlrpc listening on 127.0.0.1:23333
[I 180629 07:08:32 app:84] webui exiting…
Traceback (most recent call last):
File “/root/.pyenv/versions/3.6.5/bin/pyspider”, line 11, in
load_entry_point(‘pyspider==0.3.10’, ‘console_scripts’, ‘pyspider’)()
File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pyspider-0.3.10-py3.6.egg/pyspider/run.py”, line 754, in main
cli()
File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py”, line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pyspider-0.3.10-py3.6.egg/pyspider/run.py”, line 497, in all
ctx.invoke(webui, **webui_config)
File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pyspider-0.3.10-py3.6.egg/pyspider/run.py”, line 384, in webui
app.run(host=host, port=port)
File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pyspider-0.3.10-py3.6.egg/pyspider/webui/app.py”, line 59, in run
from .webdav import dav_app
File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pyspider-0.3.10-py3.6.egg/pyspider/webui/webdav.py”, line 216, in
dav_app = WsgiDAVApp(config)
File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/WsgiDAV-3.0.0a2-py3.6.egg/wsgidav/wsgidav_app.py”, line 122, in init
_check_config(config)
File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/WsgiDAV-3.0.0a2-py3.6.egg/wsgidav/wsgidav_app.py”, line 104, in _check_config
raise ValueError("Invalid configuration: - " + " - ".join(errors))
ValueError: Invalid configuration:

Deprecated option ‘domaincontroller’: use ‘domain_controller’ instead.

这是我遇到的，然后网上居然没有特别明显的解决，费劲千辛万苦才查到，这是WsgiDAV发布了版本 pre-release 3.x导致的，所以只要把版本降下来就好了。
将wsgidav替换为2.4.1
•# python -m pip install wsgidav==2.4.1

然后运行 pyspider all 发现居然可以了。上一张成功的图片。

然后很欢快的打开浏览器输入：localhost：5000