Python爬虫入门遇到的坑

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Python爬虫入门遇到的坑相关的知识,希望对你有一定的参考价值。

1. 环境 

- Python
  mac os预装的python 

$ python -V  
Python 2.7.10
$ where python
/usr/bin/python
$ ls /System/Library/Frameworks/Python.framework/Versions
2.3     2.5     2.6     2.7     Current
$ ls /Library/Frameworks/Python.framework/Versions (用户安装的目录)

- IDE
  Pycharm
- 辅助
  安装pip

sudo easy_install pip

- Python库

sudo pip install requests (默认安装requests 2.13.0) 
sudo pip install BeautifulSoup (默认安装BeautifulSoup 3.2.1)
sudo pip install lxml (默认安装lxml 3.7.3)

 

2. 问题

- 问题1

代码:
soup = BeautifulSoup(html, ‘lxml‘)
报错:
Traceback (most recent call last):
File "/Users/cuizhenyu/Documents/Codes/Python/DownloadMeitu/LibBeautifulSoupTest.py", line 15, in <module>
soup = BeautifulSoup(html) #soup = BeautifulSoup(html, ‘lxml‘)报错
TypeError: ‘module‘ object is not callable
解决:
from BeautifulSoup import BeautifulSoup

- 问题2

代码:
soup = BeautifulSoup(html, ‘lxml‘)
报错:
Traceback (most recent call last):
File "/Users/cuizhenyu/Documents/Codes/Python/DownloadMeitu/LibBeautifulSoupTest.py", line 15, in <module>
soup = BeautifulSoup(html, ‘lxml‘) #soup = BeautifulSoup(html, ‘lxml‘)报错
File "/Library/Python/2.7/site-packages/BeautifulSoup.py", line 1522, in __init__
BeautifulStoneSoup.__init__(self, *args, **kwargs)
File "/Library/Python/2.7/site-packages/BeautifulSoup.py", line 1147, in __init__
self._feed(isHTML=isHTML)
File "/Library/Python/2.7/site-packages/BeautifulSoup.py", line 1189, in _feed
SGMLParser.feed(self, markup)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sgmllib.py", line 104, in feed
self.goahead(0)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sgmllib.py", line 138, in goahead
k = self.parse_starttag(i)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sgmllib.py", line 296, in parse_starttag
self.finish_starttag(tag, attrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sgmllib.py", line 338, in finish_starttag
self.unknown_starttag(tag, attrs)
File "/Library/Python/2.7/site-packages/BeautifulSoup.py", line 1338, in unknown_starttag
self.endData()
File "/Library/Python/2.7/site-packages/BeautifulSoup.py", line 1251, in endData
(not self.parseOnlyThese.text or \
AttributeError: ‘str‘ object has no attribute ‘text‘
解决:
当前BeautifulSoup是v3版,不支持lxml等,需用v4版。

 









































以上是关于Python爬虫入门遇到的坑的主要内容,如果未能解决你的问题,请参考以下文章

Python爬虫安装 pyQuery 遇到的坑 Could not find function xmlCheckVersion in library libxml2. Is libxml2 ins(代

python爬虫零基础入门——反爬的简单说明

python爬虫入门

Python scrapy 常见问题及解决 遇到的坑

scrapy按顺序启动多个爬虫代码片段(python3)

python print 方法 打印 unicode 遇到的坑