使用动态滚动解析网页的所有链接

Posted 2021-05-05

在滚动到底部之后我试图将所有链接提取到页面末尾，但是在运行我的代码之后，我只获得了一些链接。

我尝试使用BeautifulSoup下面的代码来刮掉所有链接：

from bs4 import BeautifulSoup
import requests

r = requests.get('https://dir.indiamart.com/impcat/paper-dona-machine.html')
soup = BeautifulSoup(r.text,'lxml')

for links in soup.find_all('div',class_='r-cl b-gry'):
    link = links.find('a')
    print(link['href'])

我想在向下滚动页面后提取到最后的所有链接。

答案

您要解析的网页使用javascript在用户向下滚动时加载更多内容。 BeautifulSoup无法运行JavaScript，因此您无法单独使用BeautifulSoup来获取页面中的所有链接。

但是，您可以使用Selenium WebDriver轻松实现此目的（请参阅this post）：

import time
from selenium import webdriver
from bs4 import BeautifulSoup

# specify chrome driver path below
browser = webdriver.Chrome(r'C:UsersusernameDownloadschromedriver.exe')

browser.get("https://dir.indiamart.com/impcat/paper-dona-machine.html")
time.sleep(1)
body = browser.find_element_by_tag_name("body")

# Get scroll height
last_height = browser.execute_script("return document.body.scrollHeight")
new_height = 0
# scroll until the bottom of the page as been reached
while new_height != last_height:
    last_height = new_height
    # Scroll down to bottom
    browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    # Wait to load page
    time.sleep(0.2)
    # Calculate new scroll height
    new_height = browser.execute_script("return document.body.scrollHeight")

bodyHTML = browser.find_element_by_tag_name("body").get_attribute('innerHTML')

# parse the body using beautiful soup
soup = BeautifulSoup(bodyHTML,'lxml')

# store and print company links
result = []
for links in soup.find_all('div',class_='r-cl b-gry'):
    link = links.find('a')
    print(f"{link.text} - {link['href']}")
    result.append(link['href'])

print(f"{len(result)} links found")

这将打开Chrome浏览器窗口，自动向下滚动页面，然后在到达页面底部后使用BeautifulSoup解析HTML代码。请注意，要运行上述代码，您需要下载Chrome WebDriver并包含路径。

样本输出：

Goyal Industries - https://www.goyalbrothers.in/
N. S. Enterprises - https://www.indiamart.com/nsenterprise/
Bannariamman Traders - https://www.bannariammantraders.com/
...
NSN Enterprises - https://www.nsnenterprises.net/
Ruby Automation - https://www.indiamart.com/rubyautomation/
Shree Balaji Machinery (Unit Of Shree Balaji... - https://www.balajimachinery.in/
96 links found