CentOS 6.5安装pyspider过程记录
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了CentOS 6.5安装pyspider过程记录相关的知识，希望对你有一定的参考价值。
原文地址：http://blog.sina.com.cn/s/blog_48c95a190102wczx.html
1、根据pyspider官方推荐的安装方法，使用pip命令直接安装pyspider
# sudo pip install pyspider
Installing collected packages: click, itsdangerous, Werkzeug, Flask, chardet, cssselect, lxml, pyquery, ordereddict, backports.ssl-match-hostname, singledispatch, certifi, backports-abc, tornado, Flask-Login, u-msgpack-python, wsgidav, pyspider
Running setup.py install for click
Running setup.py install for itsdangerous
Running setup.py install for chardet
Running setup.py install for lxml
Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__=‘/tmp/pip-build-rpK3h8/lxml/setup.py‘;exec(compile(getattr(tokenize, ‘open‘, open)(__file__).read().replace(‘\r\n‘, ‘\n‘), __file__, ‘exec‘))" install --record /tmp/pip-mmscE4-record/install-record.txt --single-version-externally-managed --compile:
Building lxml version 3.6.0.
Building without Cython.
ERROR: /bin/sh: xslt-config: command not found
** make sure the development packages of libxml2 and libxslt are installed **
Using build configuration of libxslt
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.6
creating build/lib.linux-x86_64-2.6/lxml
copying src/lxml/_elementpath.py -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/doctestcompare.py -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/cssselect.py -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/ElementInclude.py -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/__init__.py -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/builder.py -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/pyclasslookup.py -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/usedoctest.py -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/sax.py -> build/lib.linux-x86_64-2.6/lxml
creating build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/__init__.py -> build/lib.linux-x86_64-2.6/lxml/includes
creating build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/defs.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/html5parser.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/_html5builder.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/_setmixin.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/_diffcommand.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/__init__.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/builder.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/clean.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/ElementSoup.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/usedoctest.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/soupparser.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/formfill.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/diff.py -> build/lib.linux-x86_64-2.6/lxml/html
creating build/lib.linux-x86_64-2.6/lxml/isoschematron
copying src/lxml/isoschematron/__init__.py -> build/lib.linux-x86_64-2.6/lxml/isoschematron
copying src/lxml/lxml.etree.h -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/lxml.etree_api.h -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/includes/config.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/schematron.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/xmlschema.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/relaxng.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/c14n.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/xslt.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/xpath.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/uri.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/etreepublic.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/tree.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/xmlerror.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/htmlparser.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/dtdvalid.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/xmlparser.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/xinclude.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/etree_defs.h -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-2.6/lxml/includes
creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources
creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/rng
copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/rng
creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl
copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl
copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl
creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
running build_ext
building ‘lxml.etree‘ extension
creating build/temp.linux-x86_64-2.6
creating build/temp.linux-x86_64-2.6/src
creating build/temp.linux-x86_64-2.6/src/lxml
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Isrc/lxml/includes -I/usr/include/python2.6 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.6/src/lxml/lxml.etree.o -w
src/lxml/lxml.etree.c:82:20: error: Python.h: No such file or directory
src/lxml/lxml.etree.c:84:6: error: #error Python headers needed to compile C extensions, please install development version of Python.
Compile failed: command ‘gcc‘ failed with exit status 1
creating tmp
cc -I/usr/include/libxml2 -c /tmp/xmlXPathInitlSjq1o.c -o tmp/xmlXPathInitlSjq1o.o
/tmp/xmlXPathInitlSjq1o.c:1:26: error: libxml/xpath.h: No such file or directory
*********************************************************************************
Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
*********************************************************************************
error: command ‘gcc‘ failed with exit status 1
根据错误提示可知，导致安装出错的原因是libxml2和libxslt这两个库没有安装，而且找不到xslt-config这个可执行程序，解决方法如下：
# sudo yum install libxml2
# sudo yum install libxslt
# sudo yum install libxslt-devel
2、继续使用pip命令安装pyspider
# sudo pip install pyspider
...
Running setup.py install for lxml
Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__=‘/tmp/pip-build-EQanqI/lxml/setup.py‘;exec(compile(getattr(tokenize, ‘open‘, open)(__file__).read().replace(‘\r\n‘, ‘\n‘), __file__, ‘exec‘))" install --record /tmp/pip-WzvLSh-record/install-record.txt --single-version-externally-managed --compile:
Building lxml version 3.6.0.
Building without Cython.
Using build configuration of libxslt 1.1.26
Building against libxml2/libxslt in the following directory: /usr/lib64
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.6
creating build/lib.linux-x86_64-2.6/lxml
copying src/lxml/_elementpath.py -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/doctestcompare.py -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/cssselect.py -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/ElementInclude.py -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/__init__.py -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/builder.py -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/pyclasslookup.py -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/usedoctest.py -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/sax.py -> build/lib.linux-x86_64-2.6/lxml
creating build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/__init__.py -> build/lib.linux-x86_64-2.6/lxml/includes
creating build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/defs.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/html5parser.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/_html5builder.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/_setmixin.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/_diffcommand.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/__init__.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/builder.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/clean.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/ElementSoup.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/usedoctest.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/soupparser.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/formfill.py -> build/lib.linux-x86_64-2.6/lxml/html
copying src/lxml/html/diff.py -> build/lib.linux-x86_64-2.6/lxml/html
creating build/lib.linux-x86_64-2.6/lxml/isoschematron
copying src/lxml/isoschematron/__init__.py -> build/lib.linux-x86_64-2.6/lxml/isoschematron
copying src/lxml/lxml.etree.h -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/lxml.etree_api.h -> build/lib.linux-x86_64-2.6/lxml
copying src/lxml/includes/config.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/schematron.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/xmlschema.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/relaxng.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/c14n.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/xslt.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/xpath.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/uri.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/etreepublic.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/tree.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/xmlerror.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/htmlparser.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/dtdvalid.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/xmlparser.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/xinclude.pxd -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/etree_defs.h -> build/lib.linux-x86_64-2.6/lxml/includes
copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-2.6/lxml/includes
creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources
creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/rng
copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/rng
creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl
copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl
copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl
creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
running build_ext
building ‘lxml.etree‘ extension
creating build/temp.linux-x86_64-2.6
creating build/temp.linux-x86_64-2.6/src
creating build/temp.linux-x86_64-2.6/src/lxml
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/libxml2 -Isrc/lxml/includes -I/usr/include/python2.6 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.6/src/lxml/lxml.etree.o -w
src/lxml/lxml.etree.c:82:20: error: Python.h: No such file or directory
src/lxml/lxml.etree.c:84:6: error: #error Python headers needed to compile C extensions, please install development version of Python.
Compile failed: command ‘gcc‘ failed with exit status 1
creating tmp
cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInitd0Umi6.c -o tmp/xmlXPathInitd0Umi6.o
cc tmp/xmlXPathInitd0Umi6.o -L/usr/lib64 -lxml2 -o a.out
error: command ‘gcc‘ failed with exit status 1
根据错误提示可知，导致安装出错的原因是Python.h: No such file or directory，即找不到Python.h这个头文件，解决方法如下：
# sudo yum install python-devel
至此，再次执行sudo pip install pyspider就可以完成pyspider的安装。
3、启动pyspider，验证安装是否成功
# pyspider
Traceback (most recent call last):
File "/usr/bin/pyspider", line 5, in
from pkg_resources import load_entry_point
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2655, in
working_set.require(__requires__)
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 648, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: pyquery
根据错误提示可知，在执行pyspider时找不到pyquery，但实际上在安装pyspider时pyquery已经安装了，通过pip freeze也可以看到，导致该问题的原因不详，解决方法如下：
# sudo pip install -U setuptools
4、安装pyspider所需使用的驱动
通过单纯的pyspider命令来启动pyspider时，默认会在当前目录下创建data目录，其中存放了sqlite数据库文件，该情况下无需安装任务数据库connector。如果要以分布式集群的方式来运行pyspider的话，就需要使用mysql、mongodb或postgresql作为后台数据库，使用rabbitmq、redis、beanstalk或kombu作为消息队列。官方推荐的安装方式是采用pip install pyspider[all]命令来安装所有驱动，但该方法一方面会安装一些无用的驱动，另一方面极有可能导致安装失败。因此如果已经确定后台数据库软件和消息队列软件的话，那么选择安装需要的驱动即可。例如采用mongodb作为后台数据库，采用redis作为消息队列的话，就可以通过如下命令安装：
# sudo pip install pymongo
# sudo pip install redis
以上是关于CentOS 6.5安装pyspider过程记录的主要内容，如果未能解决你的问题，请参考以下文章
Centos 6.5 python 2.6.6 升级到 2.7.3，并安装easy_install和pip工具过程
CentOS 6.5 安装 Redis 执行 make #error "Newer version of jemalloc required"