Xpath不会返回content within (is not the issue)
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Xpath不会返回content within (is not the issue)相关的知识,希望对你有一定的参考价值。
以下代码使用scrapy + scrapy-splash + Python。我试图从这个网站上提取即将到来的比赛(包括:球队名称,比赛名称,开始时间):https://www.hltv.org/matches
我在回调'解析'函数中的代码是:
match_days = response.xpath("//div[@class = 'upcoming-matches']//div[@class = 'match-day']")
for match in match_days.xpath("./a"):
print(match.extract())
# tournament_name = match.xpath(".//td[@class='event']//span[@class='event-name']/text()").extract_first()
# team1_name = match.xpath(".//td[@class='team-cell'][1]//div[@class='team']/text()").extract_first()
它应该给我每个“<a>”元素的内容(即应该看起来像这样:
<a href="/matches/2318355/dkiss-vs-psychoactive-prowince-winner-winner-of-the-future-2017" class="a-reset block upcoming-match standard-box" data-zonedgrouping-entry-unix="1514028600000">
<table class="table">
<tbody>
<tr>
<td class="time">
<div class="time" data-time-format="HH:mm" data-unix="1514028600000">12:30</div>
</td>
<td class="team-cell">
<div class="line-align">
<img alt="DKISS" src="https://static.hltv.org/images/team/logo/8657" class="logo" title="DKISS">
<div class="team">DKISS</div>
</div>
</td>
<td class="vs">vs</td>
<td class="team-cell">
<div class="team">PSYCHOACTIVE/proWince winner</div>
</td>
<td class="event"><img alt="Winner of the Future 2017" src="https://static.hltv.org/images/eventLogos/3464.png" class="event-logo" title="Winner of the Future 2017"><span class="event-name">Winner of the Future 2017</span></td>
<td class="star-cell">
<div class="map-text">bo3</div>
</td>
</tr>
</tbody>
</table>
</a>
但我只为每个“<a>”得到这个:
<a href="/matches/2318355/dkiss-vs-psychoactive-prowince-winner-winner-of-the-future-2017" class="a-reset block upcoming-match standard-box" data-zonedgrouping-entry-unix="1514028600000">
</a>
我在scrapy shell中尝试了这个并且结果相同。
我尝试使用Chrome Developer工具,我可以在innerhtml属性中看到每个“<a>”的所有内容。
我不相信问题是“<tbody>”,因为我已经明白它在某些情况下被省略并被Web浏览器添加,因为当我从“响应”中打印出返回页面的html内容时“<tbody>”就在那里(顺便说一下,我通过scrapy-splash使用lua脚本向网址发出POST请求并获取html页面)
有谁知道为什么会这样?我已经花了几天时间没有答案,也没有任何关于更多测试的想法,以便在不应该的时候解决为什么会发生这种情况。
谢谢。
使用css
选择器对我来说更容易。
import scrapy
class MySpider(scrapy.Spider):
name = 'myspider'
start_urls = ['https://www.hltv.org/matches']
def parse(self, response):
print('url:', response.url)
days = response.css('.match-day')
for day in days:
date = day.css('.standard-headline::text').extract_first()
print('date:', date)
tables = day.css('table')
for table in tables:
time = table.css('div.time::text').extract_first()
teams = table.css('.team::text').extract()
event = table.css('.event-name::text').extract_first()
placeholder = table.css('.placeholder-text-cell::text').extract_first()
print(' time:', time)
if teams:
print(' teams 1:', teams[0])
print(' teams 2:', teams[1])
print(' event:', event)
else:
print(' placeholder:', placeholder)
# --- it runs without project and saves in `output.csv` ---
from scrapy.crawler import CrawlerProcess
c = CrawlerProcess({
'USER_AGENT': 'Mozilla/5.0',
# save in file as CSV, JSON or XML
'FEED_FORMAT': 'csv', # csv, json, xml
'FEED_URI': 'output.csv', #
})
c.crawl(MySpider)
c.start()
结果
url: https://www.hltv.org/matches
date: 2017-12-24
time: 03:00
teams 1: NSPR
teams 2: MiTH
event: WESG 2017 Thailand LAN
time: 06:00
teams 1: Signature
teams 2: Beyond
event: WESG 2017 Thailand LAN
time: 09:00
placeholder: WESG Thailand - Grand Final
time: 10:00
teams 1: DKISS
teams 2: Izako Boars
event: Winner of the Future 2017
date: 2017-12-26
time: 14:00
teams 1: Recca
teams 2: Signature
event: GOTV.GG Invitational #1
time: 17:00
teams 1: AGO
teams 2: Vega Squadron
event: LOOT.BET Cup 2
time: 20:00
teams 1: mousesports
teams 2: Spirit
event: LOOT.BET Cup 2
date: 2017-12-27
time: 12:00
teams 1: Singularity
teams 2: GoodJob
event: CSesport.com XMAS Cup
time: 13:00
placeholder: GOTV.GG - Semi-Final #1
time: 15:00
placeholder: GOTV.GG - Semi-Final #2
time: 15:00
teams 1: MANS NOT HOT
teams 2: VenatoreS
event: CSesport.com XMAS Cup
time: 17:00
teams 1: Heroic
teams 2: Valiance
event: LOOT.BET Cup 2
date: 2017-12-28
time: 13:00
placeholder: GOTV.GG - 3rd place decider
time: 15:00
placeholder: GOTV.GG - Grand Final
以上是关于Xpath不会返回content within (is not the issue)的主要内容,如果未能解决你的问题,请参考以下文章
pandas使用groupby函数进行分组聚合并使用agg函数将每个分组特定变量对应的多个内容组合到一起输出(merging content within a specific column of g