如何在 python 中使用 beautifulsoup4 来抓取标签中的内容

Posted 2023-05-05

小白入门阶段，主要用requests和beautifulsoup4库来爬取内容。目前遇到的问题是，使用beautifulsoup抓取标签内容出错。所以来咨询下过往前辈的建议。
1、像上图html文档中的滴滴出行，应该如何抓取？用select函数可以实现嘛？
2、像抓取战略投资，我使用了下面的语句，内容截取到了，但是还多了个括号。不知道怎么把括号去掉。
investment=soup.select('span[class="t-small c-green"]')[0].text.strip()
3、我光是select函数就用迷糊了。。。更别说添加别的函数了。
问题比较简单，但是已经卡了我很久了。求大神指点一二啊！

from bs4 import BeautifulSoup

html_doc = '''
<div class="line-title">
<span class="title">
<b>
滴滴出行
<span class="t-small c-green">
（战略投资）
</span>
</b>
</span>
<span class="sechovershow jzbtn c-lined small marl10 act-ugc-edit act-ugc-edit-base1" style="margin-top:-5px">
<i class="fa fa-pencil"></i>
编辑
</span>
</div>
'''

soup = BeautifulSoup(html_doc, "html.parser")
# 初级版
didi = soup.b.next_element.strip()
invest = soup.b.span.next_element.strip()

# 进阶版

didi, invest = soup.b.stripped_strings

参考技术A .lstrip("(").rstrip(")")

python 使用Beautiful Soup从页面中提取数据

# To run this, you can install BeautifulSoup
# https://pypi.python.org/pypi/beautifulsoup4

# Or download the file
# http://www.py4e.com/code3/bs4.zip
# and unzip it in the same directory as this file


from urllib.request import urlopen
from bs4 import BeautifulSoup
import ssl

# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = "http://py4e-data.dr-chuck.net/comments_40614.html"
html = urlopen(url, context=ctx).read()

# html.parser is the HTML parser included in the standard Python 3 library.
# information on other HTML parsers is here:
# http://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser
soup = BeautifulSoup(html, "html.parser")

# Retrieve all of the anchor tags
sum = 0
tags = soup('span')
for tag in tags:
    # Look at the parts of a tag
    # print('TAG:', tag)
    # print('URL:', tag.get('href', None))
    print('Contents:', tag.contents[0])
    sum += int(tag.contents[0])
    # print('Attrs:', tag.attrs)
print(sum)