如何使用Selenium和Python在网站内的页面中导航？

Posted 2021-04-17

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了如何使用Selenium和Python在网站内的页面中导航？相关的知识，希望对你有一定的参考价值。

我正在搜刮这个网站( http:/rera.rajasthan.gov.inProjectSearch。)使用Python和Selenium。我的代码可以用了，但是目前只刮取了第一页，我想遍历所有的页面，并刮取其中所有的VIEW，但是他们处理分页的方式很奇怪，我怎么才能一个一个地去刮取这些页面呢？

我的源码。

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException, WebDriverException
import time

opt = webdriver.ChromeOptions()
opt.add_argument("--ignore-certificate-errors")
opt.add_argument("--start-maximized")

driver = webdriver.Chrome(executable_path=r"C:Usersfit foodiePycharmProjectsSeleniumBrowserchromedriver.exe", options=opt)

driver.get(url="http://rera.rajasthan.gov.in/")
search= driver.find_element_by_xpath("//*[@id='liSearch']/a").click()
proj_src=driver.find_element_by_xpath("//*[@id='liSearch']/ul/li[1]/a").click()

search_btn = driver.find_element_by_xpath('//*[@id="btn_SearchProjectSubmit"]').click()


def page():
    while True:
        try:
            driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(
                EC.element_to_be_clickable((By.XPATH, "//*[@id='OuterProjectGrid']/div[4]/div[4]/a"))))
            driver.find_element_by_xpath("//*[@id='OuterProjectGrid']/div[4]/div[4]/a").click()
            print("Navigating to Next Page")
        except (TimeoutException, WebDriverException) as e:
            print("Last page reached")
            break

无法通过这个来分页

img:这就是

答案

对于 Pagination 使用以下内容 css 选择器，并在每次点击后提供延迟。

def page():
    i=2
    while True:
        try:
            driver.execute_script("arguments[0].scrollIntoView();", WebDriverWait(driver, 20).until(
                EC.element_to_be_clickable((By.CSS_SELECTOR, "a[data-p='{}']".format(i)))))
            driver.find_element_by_css_selector("a[data-p='{}']".format(i)).click()

            print("Navigating to Next Page " + str(i))
            i=i+1
            time.sleep(1)
        except (TimeoutException, WebDriverException) as e:
            print("Last page reached")
            break

page()

輸出：控制台快照

如果你的目标是获取所有页面的所有表格数据，你可以不需要硒你也可以试试python requests 模块，并使用职位请求。

import requests
data={
    "PageSize" :1250,
    "page": 1
}
res=requests.post("http://rera.rajasthan.gov.in/Home/GetProjectsList",data=data).json()
for item in res['Data']['Items']:
    print(item['DistrictName'],item['ProjectName'],item['ProjectTypeName'],item['PromoterName'],item['ApplicationNo'],item['CertificateNo'])

对所有这样的页面进行输出。

Jaipur ATHARV APPARTMENT Group Housing SHP HOME LLP Revoked Project Revoked Project
Jaipur JVJ DREAM RESIDENCY Group Housing JVJ DREAM DEVELOPERS LLP RAJ-RERA-APP-P-2020-2214 (19/03/2020) RAJ/P/2020/1262 (29/05/2020)
Chittorgarh SHARDA ROYAL GREENS Plotted Development Choudhary Infraheight Private Limited RAJ-RERA-APP-P-2020-2201 (17/03/2020) RAJ/P/2020/1261 (29/05/2020)
Tonk GREEN CITY-A BLOCK Plotted Development SUN INDIA REALHOME LLP RAJ-RERA-APP-P-2020-2173 (04/03/2020) RAJ/P/2020/1260 (29/05/2020)
Ajmer Dream Homz Group Housing G S DREAMHOME LLP RAJ-RERA-APP-P-2020-2188 (13/03/2020) RAJ/P/2020/1259 (20/05/2020)
Jaipur KEDIA'S AMARA Group Housing KEDIA BUILDERS AND COLONIZERS PRIVATE LIMITED RAJ-RERA-APP-P-2020-2224 (13/05/2020) RAJ/P/2020/1258 (18/05/2020)
Jaipur Kuber Garden Group Housing PUNIT ESTATES PRIVATE LIMITED RAJ-RERA-APP-P-2020-2221 (29/04/2020) RAJ/P/2020/1257 (04/05/2020)
Kota SHUBH SAVERA Plotted Development SANTOSH  SAINI RAJ-RERA-APP-P-2020-2222 (29/04/2020) RAJ/P/2020/1256 (02/05/2020)
Udaipur MIRACLE Group Housing BHOOMISHIV BUILDERS LLP RAJ-RERA-APP-P-2020-2117 (15/02/2020) RAJ/P/2020/1255 (02/05/2020)
Jaipur NANDAN PRIME VILLAS Group Housing NARENDRA KUMAR AGARWAL RAJ-RERA-APP-P-2020-2184 (11/03/2020) RAJ/P/2020/1254 (28/04/2020)
Jaipur Akshat Kanota Estate-Phase 3 Group Housing AKSHAT APARTMENTS PRIVATE LIMITED RAJ-RERA-APP-P-2020-2052 (24/01/2020) RAJ/P/2020/1253 (20/04/2020)
Jaipur SHREE RADHA KRISHNA APARTMENT Group Housing GURUSAIKRIPA BUILDERS LLP RAJ-RERA-APP-P-2020-2213 (19/03/2020) RAJ/P/2020/1252 (16/04/2020)
Jodhpur Mangaldeep Darshan Group Housing Mangaldeep DaRSHAN RAJ-RERA-APP-P-2020-2186 (12/03/2020) RAJ/P/2020/1251 (16/04/2020)
Sri Ganganagar SHREENATH ENCLAVE Plotted Development ANANDAM HEIGHTS DEVELOPERS PRIVATE LIMITED RAJ-RERA-APP-P-2020-2144 (27/02/2020) RAJ/P/2020/1250 (16/04/2020)
Jaipur SHEKHAWAT CREST Group Housing M R S B INFRA PROJECT PRIVATE LIMITED RAJ-RERA-APP-P-2020-2181 (11/03/2020) RAJ/P/2020/1249 (12/04/2020)
Kota S.S. TIRUPATI TOWER Mixed (Residential And Commercial) S S TIRUPATI INFRAPROJECTS RAJ-RERA-APP-P-2020-2123 (18/02/2020) RAJ/P/2020/1248 (12/04/2020)
Jhalawar Green Villas Group Housing CHAUDHARY BHOORAMAL DEVELOPERS RAJ-RERA-APP-P-2020-2139 (25/02/2020) RAJ/P/2020/1247 (09/04/2020)
Ajmer Samriddhi's Dynasty Group Housing SANKALP REALMART PVT LTD RAJ-RERA-APP-P-2020-2073 (01/02/2020) RAJ/P/2020/1246 (27/03/2020)
Udaipur ARCHI'S  LOTUS PARK Group Housing ARCHI BUILDMART PRIVATE LIMITED RAJ-RERA-APP-P-2020-2171 (03/03/2020) RAJ/P/2020/1245 (27/03/2020)
Alwar KRISHAN KUNJ Plotted Development CHHOTE LAL MEENA RAJ-RERA-APP-P-2020-2067 (29/01/2020) RAJ/P/2020/1244 (27/03/2020)
Jodhpur SHANKHESHWAR NAGAR Plotted Development BALWANT  RAM RAJ-RERA-APP-P-2020-2095 (10/02/2020) RAJ/P/2020/1243 (27/03/2020)
Jodhpur VEERPRATAP INDUSTRIAL PARK Plotted Development VICTORIA INFRA HOLDINGS PRIVATE LIMITED RAJ-RERA-APP-P-2019-1699 (23/10/2019) RAJ/P/2020/1242 (27/03/2020)
Jaipur Ram Awas Group Housing Shubhashish Builders and Developers RAJ-RERA-APP-P-2020-2023 (17/01/2020) RAJ/P/2020/1241 (27/03/2020)
Sikar SHREE HANUMAN HEIGHTS Commercial MAHADEV BUILDERS AND DEVELOPERS RAJ-RERA-APP-P-2020-2166 (03/03/2020) RAJ/P/2020/1240 (27/03/2020)
Sikar MADHUVAN HOMES Group Housing RAJENDRA SINGH KHICHAR RAJ-RERA-APP-P-2020-2155 (02/03/2020) RAJ/P/2020/1239 (27/03/2020)
Baran SUMERU SOHAM Mixed (Residential And Commercial) SUMERU LIFE SPACE INDIA PRIVATE LIMITED RAJ-RERA-APP-P-2020-2172 (03/03/2020) RAJ/P/2020/1238 (27/03/2020)
Jodhpur ASHAPURNA ANMOL PHASE-I Group Housing ASHAPURNA BUILDCON LIMITED RAJ-RERA-APP-P-2020-2090 (07/02/2020) RAJ/P/2020/1237 (27/03/2020)
Sirohi AYODHYAPURAM SHEOGANJ Group Housing RAMBHADEEP BUILDCON PRIVATE LIMITED  RAJ-RERA-APP-P-2020-2111 (14/02/2020) RAJ/P/2020/1236 (27/03/2020)
Jaipur Bhavyaa Green Zenith Group Housing BHAVYAA GREEN BUILDERS RAJ-RERA-APP-P-2020-2163 (03/03/2020) RAJ/P/2020/1235 (20/03/2020)
Dholpur G.K. CITY Group Housing G K Builders RAJ-RERA-APP-P-2020-2065 (29/01/2020) RAJ/P/2020/1234 (20/03/2020)
Udaipur ARCHI'S PEARL PARADISE Group Housing ARCHI CIVIL CONSTRUCTION PRIVATE LIMITED RAJ-RERA-APP-P-2020-2142 (27/02/2020) RAJ/P/2020/1233 (20/03/2020)
Jaipur Stareef Suites 88 Group Housing Arihant Prime Buildtech LLP RAJ-RERA-APP-P-2020-2083 (05/02/2020) RAJ/P/2020/1232 (20/03/2020)
Jaipur HARITWAL CITY - D Plotted Development BHARURAM  JAT RAJ-RERA-APP-P-2020-2119 (17/02/2020) RAJ/P/2020/1231 (19/03/2020)
Jodhpur CMJAY LORDI PANDIT JI PACKAGE-10 JODHPUR Group Housing JODHPUR DEVELOPMENT AUTHORITY RAJ-RERA-APP-P-2020-2191 (13/03/2020) RAJ/P/2020/1230 (18/03/2020)
Jaipur Vedic Villas Phase- II Group Housing KEDIA BUILDERS AND COLONIZERS PRIVATE LIMITED RAJ-RERA-APP-P-2020-2169 (03/03/2020) RAJ/P/2020/1229 (12/03/2020)
Tonk SHREE GANESH VATIKA Plotted Development RAM KRISHAN COLONIZERS AND DEVELOPEPRS PRIVATE LIMITED RAJ-RERA-APP-P-2020-2158 (02/03/2020) RAJ/P/2020/1228 (11/03/2020)
Jaipur Vinayak Residency A+B+C (Extension) Plotted Development Vinayak Developers RAJ-RERA-APP-P-2020-2092 (10/02/2020) RAJ/P/2020/1226 (11/03/2020)
Jaipur NIRANJAN VIHAR EXTENSION Plotted Development SHRI GOVARDHAN ESTATES PRIVATE LIMITED  RAJ-RERA-APP-P-2020-2099 (11/02/2020) RAJ/P/2020/1225 (11/03/2020)
Jaipur SHREE PARSHVANATH ENCLAVE Group Housing PARSHVANATH INFRA PROJECT RAJ-RERA-APP-P-2020-2030 (21/01/2020) RAJ/P/2020/1224 (11/03/2020)
Jaipur Vrinda Gardens Phase V Group Housing Vista Housing RAJ-RERA-APP-P-2020-2097 (11/02/2020) RAJ/P/2020/1223 (06/03/2020)
Jaipur Ashiana Amantran Phase II Group Housing Ashiana Housing Limited RAJ-RERA-APP-P-2020-2125 (19/02/2020) RAJ/P/2020/1221 (06/03/2020)
Jaipur MANGLAM AANANDA PHASE III (B) Group Housing MANGLAM BUILD DEVELOPERS LIMITED RAJ-RERA-APP-P-2020-2152 (29/02/2020) RAJ/P/2020/1220 (06/03/2020)
Sirohi Karan Heights Group Housing Samdarshi Builders RAJ-RERA-APP-P-2020-2043 (23/01/2020) RAJ/P/2020/1219 (04/03/2020)
Alwar Krish City Centre Commercial Narmada Asbestos Pipes Private Limited RAJ-RERA-APP-P-2020-2021 (16/01/2020) RAJ/P/2020/1218 (04/03/2020)
Bhilwara OSTWAL EMPIRE-1 Plotted Development KULDEEP UMRAOSINGH OSTWAL RAJ-RERA-APP-P-2020-2040 (22/01/2020) RAJ/P/2020/1217 (04/03/2020)
Bhilwara OSTWAL EMPIRE-2 Plotted Development UMRAOSINGH PRITHVIRAJ OSTWAL RAJ-RERA-APP-P-2020-2039 (22/01/2020) RAJ/P/2020/1216 (04/03/2020)
Kota AKANSHA DEEP HEIGHTS Group Housing AKANSHA INFRA HOUSING PROJECTS RAJ-RERA-APP-P-2020-2122 (17/02/2020) RAJ/P/2020/1215 (04/03/2020)
Jodhpur NAKSHATRA Group Housing VISION ASSOCIATES RAJ-RERA-APP-P-2020-2070 (31/01/2020) RA以上是关于如何使用Selenium和Python在网站内的页面中导航？的主要内容，如果未能解决你的问题，请参考以下文章 
 如何使用Selenium和Python在reCAPTCHA演示页面上找到多个框架内的多个元素并与之交互。
 如果通过 python selenium 和美丽的汤嵌套在视频内的源标签内，则无法抓取 src
 如何使用 Selenium 和 Python 在 Python 类中调用方法
 在新标签页中打开网页 Selenium + Python
 如何使用 selenium webdriver (python) 选择、复制和粘贴元素中的所有内容
 如何通过Selenium和Python按照网站点击文本为LOGIN的元素？