scrapy框架中如何使用selenuim

Posted 2021-01-28 duanhaoxin

主程序代码：

 1 import scrapy
 2 from selenium import  webdriver
 3 
 4 class SelenuimtestSpider(scrapy.Spider):
 5     name = ‘selenuimTest‘
 6     # allowed_domains = [‘www.xxx.com‘]
 7     start_urls = [‘http://war.163.com/‘]
 8 
 9     def __init__(self):
10         #本地谷歌浏览器的驱动存放地址
11         self.bro = webdriver.Chrome(r‘D:爬虫相关资料驱动程序chromedriver_win32chromedriver.exe‘)
12     def parse(self, response):
13         print(response.xpath(‘/html/body/div[1]/div[3]/div[4]/div[1]/div/div/ul/li/div/div[3]/div[1]/h3/a/text()‘).extract_first())
14 
15 
16     def closed(self,spider):
17         self.bro.quit()    

爬虫代码