爬虫-BS4

Posted aspiration

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了爬虫-BS4相关的知识，希望对你有一定的参考价值。

简介
BeautifulSoup提供一些简单的、python式的函数用来处理导航、搜索、修改分析树等功能。它是一个工具箱，通过解析文档为用户提供需要抓取的数据，因为简单，所以不需要多少代码就可以写出一个完整的应用程序。BeautifulSoup自动将输入文档转换为Unicode编码，输出文档转换为utf-8编码。你不需要考虑编码方式，除非文档没有指定一个编码方式，这时你仅仅需要说明一下原始编码方式就可以了。BeautifulSoup已成为和lxml、html6lib一样出色的python解释器，为用户灵活地提供不同的解析策略或强劲的速度

from bs4 import BeautifulSoup

html = """
<html><head><title>The Dormouse‘s story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html, ‘lxml‘)
print(soup.prettify())
print(soup.title.string)

<html>
 <head>
  <title>
   The Dormouse‘s story
  </title>
 </head>
 <body>
  <p class="title" name="dromouse">
   <b>
    The Dormouse‘s story
   </b>
  </p>
  <p class="story">
   Once upon a time there were three little sisters; and their names were
   <a class="sister" href="http://example.com/elsie" id="link1">
    <!-- Elsie -->
   </a>
   ,
   <a class="sister" href="http://example.com/lacie" id="link2">
    Lacie
   </a>
   and
   <a class="sister" href="http://example.com/tillie" id="link3">
    Tillie
   </a>
   ;
and they lived at the bottom of a well.
  </p>
  <p class="story">
   ...
  </p>
 </body>
</html>
The Dormouse‘s story

标签选择器

# 选择元素
soup.p # 标签加其内部的所有内容，且只获取第一个匹配的标签

type(soup.p) # bs4.element.Tag

bs4.element.Tag

soup.p

<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>

data = []
for i in soup.find(‘body‘).children:
    data.append(i)

data

[‘
‘,
 <p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>,
 ‘
‘,
 <p class="story">Once upon a time there were three little sisters; and their names were
 <a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
 and they lived at the bottom of a well.</p>,
 ‘
‘,
 <p class="story">...</p>,
 ‘
‘]

print(soup.body.prettify())

<body>
 <p class="title" name="dromouse">
  <b>
   The Dormouse‘s story
  </b>
 </p>
 <p class="story">
  Once upon a time there were three little sisters; and their names were
  <a class="sister" href="http://example.com/elsie" id="link1">
   <!-- Elsie -->
  </a>
  ,
  <a class="sister" href="http://example.com/lacie" id="link2">
   Lacie
  </a>
  and
  <a class="sister" href="http://example.com/tillie" id="link3">
   Tillie
  </a>
  ;
and they lived at the bottom of a well.
 </p>
 <p class="story">
  ...
 </p>
</body>

# 提取信息
soup.p.string
# 获取名称
soup.p.name 
soup.p.parent.name
# 获取属性
print(soup.p.attrs) # 返回字典
print(soup.p.attrs[‘name‘])
print(soup.p[‘name‘])
# 获取内容
print(soup.p.string)

soup.title.string

type(soup.title.string)

type(soup.title.contents)

soup.head

soup.p

soup.p.string

"The Dormouse‘s story"

嵌套选择

# 每一返回结果是bs4.element.Tag
print(type(soup.p))
print(type(soup.p.b))

print(soup.p.string)
print(soup.p.b.string)

<class ‘bs4.element.Tag‘>
<class ‘bs4.element.Tag‘>
The Dormouse‘s story
The Dormouse‘s story

关联选择

# 选取到了一个节点元素之后，如果想要获取它的直接子节点可以调用contents属性
print(soup.body.contents)  # contents返回的是列表类型

[‘
‘, <p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>, ‘
‘, <p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>, ‘
‘, <p class="story">...</p>, ‘
‘]

print(soup.title.contents)

["The Dormouse‘s story"]

list_ = soup.body.contents
len(list_)

7

子节点和子孙节点

# print(soup.body.children)
for i,child in enumerate(soup.body.children):
    print(i,child)

0 

1 <p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
2 

3 <p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
4 

5 <p class="story">...</p>
6

?

print(soup.body.descendants) # 生成器,递归查询所有子孙节点
for i,childs in enumerate(soup.body.descendants):
    print(i,childs)

<generator object Tag.descendants at 0x000002435F075ED0>
0 

1 <p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
2 <b>The Dormouse‘s story</b>
3 The Dormouse‘s story
4 

5 <p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
6 Once upon a time there were three little sisters; and their names were

7 <a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>
8  Elsie 
9 ,

10 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
11 Lacie
12  and

13 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
14 Tillie
15 ;
and they lived at the bottom of a well.
16 

17 <p class="story">...</p>
18 ...
19

?

父节点和祖先节点

print(soup.a.parent) # 返回其直接父节点即其下内容
type(soup.a.parent) # bs4.element.Tag

<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>





bs4.element.Tag

print(soup.a.parents) # 
for i,des in enumerate(soup.a.parents):
    print(i,des)

<generator object Tag.descendants at 0x000002435F075ED0>
0 <p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
1 <body>
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
</body>
2 <html><head><title>The Dormouse‘s story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
</body></html>
3 <html><head><title>The Dormouse‘s story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
</body></html>

兄弟节点

html = """
<html>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            Hello
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, ‘lxml‘)
print(‘Next Sibling‘, soup.a.next_sibling)
print(‘Prev Sibling‘, soup.a.previous_sibling)
print(‘Next Siblings‘, list(enumerate(soup.a.next_siblings)))
print(‘Prev Siblings‘, list(enumerate(soup.a.previous_siblings)))

Next Sibling 
            Hello
            
Prev Sibling 
            Once upon a time there were three little sisters; and their names were
            
Next Siblings [(0, ‘
            Hello
            ‘), (1, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>), (2, ‘ 
            and
            ‘), (3, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>), (4, ‘
            and they lived at the bottom of a well.
        ‘)]
Prev Siblings [(0, ‘
            Once upon a time there were three little sisters; and their names were
            ‘)]

方法选择器

# find_all()、find()等方法，我们可以调用方法然后传入相应等参数就可以灵活地进行查询了
find_all(name , attrs , recursive , text , **kwargs)
find_all，顾名思义，就是查询所有符合条件的元素，可以给它传入一些属性或文本来得到符合条件的元素
soup.find_all(True) # 返回所有标签

html=‘‘‘
<div class="panel">
<div class="panel-heading">
        <h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
</ul>
</div>
</div>
‘‘‘
soup = BeautifulSoup(html, ‘lxml‘)

name属性

print(soup.find_all(name=‘ul‘)) # 查询所有ul标签，返回list
print(‘#‘*50)
print(type(soup.find_all(name=‘ul‘)))
print(type(soup.find_all(name=‘ul‘)[0]))

[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>, <ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>]
##################################################
<class ‘bs4.element.ResultSet‘>
<class ‘bs4.element.Tag‘>

for ul in soup.find_all(name=‘ul‘):
    print(ul.find_all(name=‘li‘)) # 结果返回列表类型，列表种的每个元素依然是Tagl类型
    for li in ul.find_all(name=‘li‘):
        print(li.string)

[]
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
Foo
Bar
Jay
[]
[<li class="element">Foo</li>, <li class="element">Bar</li>]
Foo
Bar

attrs属性

print(soup.find_all(attrs={‘class‘:‘element‘})) # 传入的是attrs参数，参数的类型是字典类型

print(soup.find_all(‘ul‘,attrs={‘id‘:‘list-1‘}))

[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]

常用属性如id,class可以不用attrs来传递，而直接用id和class_

print(soup.find_all(class_=‘element‘))

print(soup.find_all(id=‘list-1‘))

[<li class="element">Foo</li>,
 <li class="element">Bar</li>,
 <li class="element">Jay</li>,
 <li class="element">Foo</li>,
 <li class="element">Bar</li>]

text属性

# text参数可以用来匹配节点的文本，传入的形式可以是字符串，可以是正则表达式
import re
html=‘‘‘
<div class="panel">
    <div class="panel-body">
        <a>Hello, this is a link</a>
        <a>Hello, this is a link, too</a>
    </div>
</div>
‘‘‘
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, ‘lxml‘)
print(soup.find_all(text=re.compile(‘link‘))) # 返回所有匹配正则表达式的节点文本组成的列表

[‘Hello, this is a link‘, ‘Hello, this is a link, too‘]

find(name,attrs,recursive,text,**kwargs)
find()方法返回的是单个元素，也就是第一个匹配的元素

html=‘‘‘
<div class="panel">
<div class="panel-heading">
        <h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
</ul>
</div>
</div>
‘‘‘

from bs4 import BeautifulSoup
soup = BeautifulSoup(html,‘lxml‘)

print(soup.find(neme=‘div‘)) # Return only the first child of this Tag matching the given
print(‘#‘*10)
print(soup.find(name=‘ul‘))
print(‘#‘*10)
print(soup.find(attrs={‘panel-heading‘}))
print(‘#‘*10)
print(soup.find(class_=‘element‘))

None
##########
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
##########
<div class="panel-heading">
<h4>Hello</h4>
</div>
##########
<li class="element">Foo</li>

soup.find_parent() # 返回直接父节点
soup.find_parents() # 返回所有祖先节点
soup.find_next_sibling()# 返回后面第一个兄弟节点
soup.find_next_siblings() # 返回后面所有兄弟节点
soup.find_next() # 返回第一个符合条件的节点
soup.find_all_next() # 返回节点后所有符合条件的节点

CSS选择器

使用CSS选择器，只需要调用select()方法，传入相应的CSS选择器即可

html=‘‘‘
<div class="panel">
<div class="panel-heading">
        <h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
</ul>
</div>
</div>
‘‘‘
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, ‘lxml‘)

print(soup.select(‘.panel .panel-heading‘)) # 符合CSS选择器的节点组成的列表

print(soup.select(‘ul li‘)) # 符合CSS选择器的节点组成的列表

print(soup.select(‘#list-2 .element‘)) # 符合CSS选择器的节点组成的列表
print(type(soup.select(‘ul‘)[0])) # print(type(soup.select(‘ul‘)[0]))

[<div class="panel-heading">
<h4>Hello</h4>
</div>]

嵌套选择

soup.select(‘ul‘)

[<ul class="list" id="list-1">
 <li class="element">Foo</li>
 <li class="element">Bar</li>
 <li class="element">Jay</li>
 </ul>, <ul class="list list-small" id="list-2">
 <li class="element">Foo</li>
 <li class="element">Bar</li>
 </ul>]

for ul in soup.select(‘ul‘):
    print(ul.select(‘li‘))

[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]

# 获取属性
for ul in soup.select(‘ul‘):
    print(ul[‘id‘])
    print(ul.attrs[‘id‘])
    print(ul.get_text()) # 获取文本

list-1
list-1

Foo
Bar
Jay

list-2
list-2

Foo
Bar

?

print(soup.find_all(‘div‘)[1])
[out]
<div class="panel-heading">
<h4>Hello</h4>
</div>
soup.find_all(‘div‘)[1].get_text()
[out]
‘
Hello
‘
print(soup.find_all(‘div‘)[1].h4.string)
[out]
Hello
print(soup.find_all(‘div‘)[1].string)
[out]
None


以上是关于爬虫-BS4的主要内容，如果未能解决你的问题，请参考以下文章

Python爬虫入门——利用bs4库对HTML页面信息进行遍历读取

python-爬虫-bs4-BeautifulSoup

全网最全python爬虫系统进阶学习(附原代码)学完可就业

全网最全python爬虫系统进阶学习(附原代码)学完可就业

Python爬虫(十五)_案例：使用bs4的爬虫

Python爬虫(十五)_案例：使用bs4的爬虫