python爬虫之scrapy安装

Posted 2020-10-11

 简介：

　　Scrapy，Python开发的一个快速、高层次的屏幕抓取和web抓取框架，用于抓取web站点并从页面中提取结构化的数据。Scrapy用途广泛，可以用于数据挖掘、监测和自动化测试。

　　Scrapy吸引人的地方在于它是一个框架，任何人都可以根据需求方便的修改。它也提供了多种类型爬虫的基类，如BaseSpider、sitemap爬虫等，最新版本又提供了web2.0爬虫的支持。

 

 安装环境：

　　Windows安装：

　　注意：

　　　　1、从上往下依次安装依赖包，.whl文件直接pip3 install 文件绝对路径和名字即可安装

　　　　2、注意你的pip版本，下载9.0以上版本。

　　Linux：

　　下面是介绍Centos6.5版本安装，注意yum源的配置。

　　安装依赖包。

yum install python3-dev

yum install -y python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev

　　检查pip版本是不是9.0.1

[[email protected] /]# pip3 --version
pip 9.0.1 from /usr/local/lib/python3.6/site-packages (python 3.6)

　　

　　开始安装scrapy。（linux系统不像windows那样依赖包需要我们一个一个安装，它会自动安装所有需要安装的依赖包，省略了很多步骤和问题）

[[email protected] /]# pip3 install scrapy
Collecting scrapy
  Downloading Scrapy-1.4.0-py2.py3-none-any.whl (248kB)
    100% |████████████████████████████████| 256kB 29kB/s
Collecting PyDispatcher>=2.0.5 (from scrapy)
  Downloading PyDispatcher-2.0.5.tar.gz
Collecting parsel>=1.1 (from scrapy)
  Downloading parsel-1.2.0-py2.py3-none-any.whl
Collecting service-identity (from scrapy)
  Downloading service_identity-17.0.0-py2.py3-none-any.whl
Collecting w3lib>=1.17.0 (from scrapy)
  Downloading w3lib-1.18.0-py2.py3-none-any.whl
Collecting queuelib (from scrapy)
  Downloading queuelib-1.4.2-py2.py3-none-any.whl
Collecting lxml (from scrapy)
  Downloading lxml-4.1.0-cp36-cp36m-manylinux1_x86_64.whl (5.6MB)
    100% |████████████████████████████████| 5.6MB 13kB/s
Collecting cssselect>=0.9 (from scrapy)
  Downloading cssselect-1.0.1-py2.py3-none-any.whl
Collecting six>=1.5.2 (from scrapy)
  Downloading six-1.11.0-py2.py3-none-any.whl
Collecting pyOpenSSL (from scrapy)
  Downloading pyOpenSSL-17.3.0-py2.py3-none-any.whl (51kB)
    100% |████████████████████████████████| 51kB 11kB/s
Collecting Twisted>=13.1.0 (from scrapy)
  Downloading Twisted-17.9.0.tar.bz2 (3.0MB)
    100% |████████████████████████████████| 3.0MB 20kB/s
Collecting attrs (from service-identity->scrapy)
  Downloading attrs-17.2.0-py2.py3-none-any.whl
Collecting pyasn1-modules (from service-identity->scrapy)
  Downloading pyasn1_modules-0.1.5-py2.py3-none-any.whl (60kB)
    100% |████████████████████████████████| 61kB 79kB/s
Collecting pyasn1 (from service-identity->scrapy)
  Downloading pyasn1-0.3.7-py2.py3-none-any.whl (63kB)
    100% |████████████████████████████████| 71kB 87kB/s
Collecting cryptography>=1.9 (from pyOpenSSL->scrapy)
  Downloading cryptography-2.1.2-cp36-cp36m-manylinux1_x86_64.whl (2.2MB)
    100% |████████████████████████████████| 2.2MB 16kB/s
Collecting zope.interface>=4.0.2 (from Twisted>=13.1.0->scrapy)
  Downloading zope.interface-4.4.3-cp36-cp36m-manylinux1_x86_64.whl (173kB)
    100% |████████████████████████████████| 174kB 20kB/s
Collecting constantly>=15.1 (from Twisted>=13.1.0->scrapy)
  Downloading constantly-15.1.0-py2.py3-none-any.whl
Collecting incremental>=16.10.1 (from Twisted>=13.1.0->scrapy)
  Downloading incremental-17.5.0-py2.py3-none-any.whl
Collecting Automat>=0.3.0 (from Twisted>=13.1.0->scrapy)
  Downloading Automat-0.6.0-py2.py3-none-any.whl
Collecting hyperlink>=17.1.1 (from Twisted>=13.1.0->scrapy)
  Downloading hyperlink-17.3.1-py2.py3-none-any.whl (73kB)
    100% |████████████████████████████████| 81kB 46kB/s
Collecting idna>=2.1 (from cryptography>=1.9->pyOpenSSL->scrapy)
  Downloading idna-2.6-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 48kB/s
Collecting asn1crypto>=0.21.0 (from cryptography>=1.9->pyOpenSSL->scrapy)
  Downloading asn1crypto-0.23.0-py2.py3-none-any.whl (99kB)
    100% |████████████████████████████████| 102kB 19kB/s
Collecting cffi>=1.7; platform_python_implementation != "PyPy" (from cryptography>=1.9->pyOpenSSL->scrapy)
  Downloading cffi-1.11.2-cp36-cp36m-manylinux1_x86_64.whl (419kB)
    100% |████████████████████████████████| 430kB 18kB/s
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/site-packages (from zope.interface>=4.0.2->Twisted>=13.1.0->scrapy)
Collecting pycparser (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=1.9->pyOpenSSL->scrapy)
  Downloading pycparser-2.18.tar.gz (245kB)
    100% |████████████████████████████████| 256kB 62kB/s
Installing collected packages: PyDispatcher, six, w3lib, lxml, cssselect, parsel, idna, asn1crypto, pycparser, cffi, cryptography, pyOpenSSL, attrs, pyasn1, pyasn1-modules, service-identity, queuelib, zope.interface, constantly, incremental, Automat, hyperlink, Twisted, scrapy
  Running setup.py install for PyDispatcher ... done
  Running setup.py install for pycparser ... done
  Running setup.py install for Twisted ... done
Successfully installed Automat-0.6.0 PyDispatcher-2.0.5 Twisted-17.9.0 asn1crypto-0.23.0 attrs-17.2.0 cffi-1.11.2 constantly-15.1.0 cryptography-2.1.2 cssselect-1.0.1 hyperlink-17.3.1 idna-2.6 incremental-17.5.0 lxml-4.1.0 parsel-1.2.0 pyOpenSSL-17.3.0 pyasn1-0.3.7 pyasn1-modules-0.1.5 pycparser-2.18 queuelib-1.4.2 scrapy-1.4.0 service-identity-17.0.0 six-1.11.0 w3lib-1.18.0 zope.interface-4.4.3
[[email protected] /]# python3
Python 3.6.3 (default, Oct 25 2017, 10:18:57)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import scrapy
>>>