scrapy爬取网址，进而爬取详情页问题

Posted 2021-01-09 zhz-8919

1、最容易出现的问题是爬取到的url大多为相对路径，如果直接将爬取到的url进行二次爬取就会出现以下报错：

　　raise ValueError(‘Missing scheme in request url: %s‘ % self._url)，该错误的意思是request的url为无效链接。

解决方法：将相对路径转换为绝对路径

 1 # -*- coding: utf-8 -*-
 2 import scrapy
 3 from pyquery import PyQuery
 4 from ..items import City10Item
 5 from scrapy.http import Request
 6 from scrapy.utils.response import get_base_url
 7 from urllib.parse import urljoin
 8 
 9 
10 class CitySpiderTest1Spider(scrapy.Spider):
11     name = ‘city_spider_test1‘
12     allowed_domains = [‘58.com‘]
13     start_urls = [‘https://cq.58.com/chuzu/‘]
14 
15     def parse(self, response):
16         jpy=PyQuery(response.text)
17         li=jpy(‘body > div.mainbox > div > div.content > div.listBox > ul > li‘).items()
18         for i in li:
19             item=City10Item()
20             item[‘name‘]=i(‘div.des > h2 > a‘).text()
21             item[‘url‘]=i(‘div.des > h2 > a‘).attr(‘href‘) #爬取url，此时为相对路径
22             item[‘price‘]=i(‘div.listliright > div.money > b‘).text()
23             base_url = get_base_url(response) #获取相对路径前级地址
24             abs_url=urljoin(base_url,item[‘url‘]) #转换为绝对路径
25 
26 
27             if item[‘url‘]:
28                 yield Request(abs_url,
29                               callback=self.detail_parse, #回调详情页函数
30                               meta={‘item‘:item}, #将参数传递给meta
31                               priority=10,
32                               dont_filter=True #强制不过滤
33 
34                 )
35 
36     def detail_parse(self,response):
37         jpy = PyQuery(response.text)
38         item = response.meta[‘item‘]   #接收item
39         item[‘introduce_item‘] = jpy(‘body > div.main-wrap > div.house-detail-desc > div.main-detail-info.fl > div.house-word-introduce.f16.c_555 > ul > li:nth-child(1) > span.a2‘).text()   #提取房屋亮点
40         item[‘address‘] = jpy(‘body > div.main-wrap > div.house-basic-info > div.house-basic-right.fr > div.house-basic-desc > div.house-desc-item.fl.c_333 > ul > li:nth-child(6) > span.dz‘).text()   #房屋详情地址
41         item[‘phone_number‘] = jpy(‘body > div.main-wrap > div.house-basic-info > div.house-basic-right.fr > div.house-fraud-tip > div.house-chat-phone > span‘).text()   #电话号码
42         yield item