从BeautifulSoup对象中删除非BMP字符

Posted 2021-04-05

我是python的初学者。我正在使用BeautifulSoup从网站中提取数据。但只要页面的源代码包含表情符号，我的程序就会停止。在解析之前/之前我应该​​做什么，以便删除表情符号/非BMP字符并抓取页面。

import bs4 as bs
import string
import urllib.request

str = 'http://www.storypick.com/harshad-mehta-scam-web-series/' #myurl
source = urllib.request.urlopen(str);
soup = bs.BeautifulSoup(source,'lxml');

match=soup.find('div',class_='td-post-content');
str=soup.title.text+"
";
name=soup.title.text;
for paragraph in match.find_all(['p' , 'h4' , 'h3' , 'h2' , 'blockquote']):
    str+=paragraph.text+"
";
print(str);

输出：

UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 161-161: Non-BMP character not supported in Tk

答案

我转而使用requests，这让事情更简单。这是一个比你想要做的更简单的例子，但确实有效。现在完成脚本应该没有问题。

import requests
from bs4 import BeautifulSoup

requestURL = 'http://www.storypick.com/harshad-mehta-scam-web-series'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (Khtml, like Gecko) Chrome/64.0.3282.186 Safari/537.36'}

with requests.Session() as session:
    r = session.get(requestURL, headers=headers)
    if r.ok:
        soup = BeautifulSoup(r.content, 'lxml')
        for paragraph in soup.find_all('p'):
            print (paragraph)

另一答案

为我完美工作！我稍微修改了一下代码

import bs4 as bs
import string
import urllib

str = 'http://www.storypick.com/harshad-mehta-scam-web-series/' #myurl
source = urllib.urlopen(str);
soup = bs.BeautifulSoup(source);

match=soup.find('div',class_='td-post-content');
str=soup.title.text+"
";
name=soup.title.text;
for paragraph in match.find_all(['p' , 'h4' , 'h3' , 'h2' , 'blockquote']):
    str+=paragraph.text+"
";
print(str);