python之urllib2简单解析HTML页面

Posted 2020-09-07

一、urllib2简单获取html页面

   #!/usr/bin/env python
   # -*- coding:utf-8 -*-
   
   import urllib2
   response = urllib2.urlopen(‘http://www.baidu.com‘);
   html = response.read();
   print html

简单的几行代码就能拿到html页面，接下来局势html的解析工作了。

想象很美好，实际操作就出问题了。baidu没有禁止机器人抓取可以正常抓取到页面，但是比如：https://b.ishadow.tech/是禁止机器人抓取的，简单模拟浏览器头部信息也不行。

 

然后想找个GitHub上的爬虫来试验一下行不行，因此找到了https://github.com/scrapy/scrapy，看样子好像比较叼。

按照readme安装了一下，安装失败了，仔细看了一下文档。

pip install scrapy

官方建议安装在Python的虚拟环境里，描述大概如下（https://doc.scrapy.org/en/latest/intro/install.html#intro-using-virtualenv）：

Using a virtual environment (recommended)
TL;DR: We recommend installing Scrapy inside a virtual environment on all platforms.

Python packages can be installed either globally (a.k.a system wide), or in user-space. We do not recommend installing scrapy system wide.

Instead, we recommend that you install scrapy within a so-called “virtual environment” (virtualenv). Virtualenvs allow you to not conflict with already-installed Python system packages (which could break some of your system tools and scripts), and still install packages normally with pip (without sudo and the likes).

然后决定安装一个Python虚拟环境，命令如下：

$ sudo pip install virtualenv

查看基本使用

$virtualenv -h
Usage: virtualenv [OPTIONS] DEST_DIR

只需要 virtualenv加目标目录就可以了。

因此新建虚拟环境

$virtualevn e27
New python executable in ~/e27/bin/python
Installing setuptools, pip, wheel...done.

启用环境

$source ./bin/activate

注意切换环境成功后当前目录会有标识，如下

?  e27 source ./bin/activate 
(e27) ?  e27 

 

退出环境

$deactivate

 

现在开始干正事，在虚拟环境中安装爬虫（https://github.com/scrapy/scrapy）

pip install scrapy

大约三分钟后安装完成，之前直接在全局环境安装尽然还失败了。成功后shell输出如下：

......
Successfully built lxml PyDispatcher Twisted pycparser
Installing collected packages: lxml, PyDispatcher, zope.interface, constantly, incremental, attrs, Automat, Twisted, ipaddress, asn1crypto, enum34, idna, pycparser, cffi, cryptography, pyOpenSSL, queuelib, w3lib, cssselect, parsel, pyasn1, pyasn1-modules, service-identity, scrapy
Successfully installed Automat-0.5.0 PyDispatcher-2.0.5 Twisted-17.1.0 asn1crypto-0.22.0 attrs-16.3.0 cffi-1.10.0 constantly-15.1.0 cryptography-1.8.1 cssselect-1.0.1 enum34-1.1.6 idna-2.5 incremental-16.10.1 ipaddress-1.0.18 lxml-3.7.3 parsel-1.1.0 pyOpenSSL-16.2.0 pyasn1-0.2.3 pyasn1-modules-0.0.8 pycparser-2.17 queuelib-1.4.2 scrapy-1.3.3 service-identity-16.0.0 w3lib-1.17.0 zope.interface-4.3.3

 安装好scrapy后尝试一个简单的连接

(e27) ?  e27 scrapy shell ‘http://quotes.toscrape.com/page/1/‘

得到一堆结果如下

2017-03-30 22:08:42 [scrapy.core.engine] INFO: Spider opened
2017-03-30 22:08:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None)
[s] Available Scrapy objects:
[s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s]   crawler    <scrapy.crawler.Crawler object at 0x1100bab50>
[s]   item       {}
[s]   request    <GET http://quotes.toscrape.com/page/1/>
[s]   response   <200 http://quotes.toscrape.com/page/1/>
[s]   settings   <scrapy.settings.Settings object at 0x1100baad0>
[s]   spider     <DefaultSpider ‘default‘ at 0x11037ebd0>
[s] Useful shortcuts:
[s]   fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s]   fetch(req)                  Fetch a scrapy.Request and update local objects 
[s]   shelp()           Shell help (print this help)
[s]   view(response)    View response in a browser

证明是可以工作的，然后试一下连接：https://b.ishadow.tech/

(e27) ?  e27 scrapy shell ‘https://b.ishadow.tech/‘  

结果如下：

2017-03-30 22:10:21 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-03-30 22:10:21 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-03-30 22:10:21 [scrapy.core.engine] INFO: Spider opened
2017-03-30 22:11:36 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://b.ishadow.tech/> (failed 1 times): TCP connection timed out: 60: Operation timed out.
2017-03-30 22:12:51 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://b.ishadow.tech/> (failed 2 times): TCP connection timed out: 60: Operation timed out.
2017-03-30 22:14:07 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET https://b.ishadow.tech/> (failed 3 times): TCP connection timed out: 60: Operation timed out.
Traceback (most recent call last):

爬去超时了，看来是被识别出来是机器人爬取内容被拒绝的（当然此时站点通过浏览器是可以访问的），厉害了我的哥！到这里你是不是已经猜到我的真实目的了，没有的话请打开我爬取得连接看看就知道了。??

及时没有爬取成功有时间研究一下这个爬虫还是挺好的，文档连接：https://doc.scrapy.org/en/latest/intro/overview.html

接下来慢慢研究怎么突破封锁。