使用 BeautifulSoup 提取 <script 中的 json 内容

Posted 2023-02-23

【中文标题】使用 BeautifulSoup 提取 <script 中的 json 内容

【英文标题】：Extract json content in <script using BeautifulSoup

【发布时间】：2019-04-15 19:31:43

【问题描述】：

我正在尝试使用漂亮的汤提取脚本的 json 部分，但它什么也没打印。怎么了？

import requests,json
from bs4 import BeautifulSoup

headers = 
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (Khtml, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
        'referrer': 'https://google.com',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
        'Accept-Encoding': 'gzip, deflate, br',
        'Accept-Language': 'en-US,en;q=0.9',
        'Pragma': 'no-cache',
    
r = requests.get('https://www.joox.com/id/id/single/xtMtD9ZdeLEdHp1w1fip8w==',headers=headers)
soup = BeautifulSoup(r.content, 'html5lib')
data = json.loads(soup.find('script').text)
print(data)

汤的结果是这样的：

<script>

      __NEXT_DATA__ = "props":"trackData":"album_url":"https://imgcache.qq.com/music/photo/mid_album_300/O/F/002tkoBZ2zJPOF.jpg","code":0,"country":"hk","encodeSongId":"xtMtD9ZdeLEdHp1w1fip8w==","express_domain":"http://stream.music.joox.com/","flag":0,"gososo":0,"has_hifi":false,"has_hq":false,"imgSrc":"https://imgcache.qq.com/music/photo/mid_album_300/O/F/002tkoBZ2zJPOF.jpg","kbps_map":"\"128\":4082449,\"192\":6522038,\"24\":825973,\"320\":10216761,\"48\":1708095,\"96\":3701266,\"ape\":0,\"flac\":0\n","ktrack_id":0,"m4aUrl":"https://hk.stream.music.joox.com/C4000035Jew421IKj2.m4a?

对此有什么想法吗？谢谢

【参考方案1】：

您可以使用子字符串和正则表达式来获取您正在寻找的内容。

    import requests,json, re
    from bs4 import BeautifulSoup

    headers = 
            'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
            'referrer': 'https://google.com',
            'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
            'Accept-Encoding': 'gzip, deflate, br',
            'Accept-Language': 'en-US,en;q=0.9',
            'Pragma': 'no-cache',
        
    r = requests.get('https://www.joox.com/id/id/single/xtMtD9ZdeLEdHp1w1fip8w==',headers=headers)
    soup = BeautifulSoup(r.content, 'html5lib')
    # finding the script element whihc contains the required data
    json_data = soup.find('script', text=re.compile("__NEXT_DATA__") )
    # identifying the required elements and obtaining it by sub string
    data = str(json_data)[str(json_data).find('__NEXT_DATA__ = '):str(json_data).find('module=')]
    data = json.loads(data.replace('__NEXT_DATA__ = ', ''))
    print(data)

使用切片技术，我们识别字符串模式并单独提取 json 字符串。之后你就可以开始了！

希望这会有所帮助！干杯!

【讨论】：

不得不为我的具体情况做一些修改，但这个答案大部分让我到了那里。

【参考方案2】：

即使BeautifulSoup还是需要Regex，为什么不直接用呢

import re

....
r = requests.get('https://www.joox.com/id/id/single/xtMtD9ZdeLEdHp1w1fip8w==',headers=headers)

jsData = re.search(r'__NEXT_DATA__\s+=\s+(.*)', r.text)
data = json.loads(jsData.group(1))
print(data)