如何使用Scrapy Tor Privoxy和UserAgent匿名废弃? (Windows 10)
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了如何使用Scrapy Tor Privoxy和UserAgent匿名废弃? (Windows 10)相关的知识,希望对你有一定的参考价值。
这个问题的答案很难找到,因为信息是分散的,问题的标题有时会产生误导。下面的答案重新组合了一个地方所需的所有信息。
答案
你的蜘蛛应该是这样的。
# based on https://doc.scrapy.org/en/latest/intro/tutorial.html
import scrapy
import requests
class QuotesSpider(scrapy.Spider):
name = "quotes"
def start_requests(self):
urls = [
'http://quotes.toscrape.com/page/1/',
'http://quotes.toscrape.com/page/2/',
]
for url in urls:
print('
url:', url)
## use one of the yield below
# middleware will process the request
yield scrapy.Request(url=url, callback=self.parse)
# check if Tor has changed IP
#yield scrapy.Request('http://icanhazip.com/', callback=self.is_tor_and_privoxy_used)
def parse(self, response):
page = response.url.split("/")[-2]
filename = 'quotes-%s.html' % page
with open(filename, 'wb') as f:
f.write(response.body)
print('
Spider: Start')
print('Is proxy in response.meta?: ', response.meta)
print ("user_agent is: ",response.request.headers['User-Agent'])
print('
Spider: End')
self.log('Saved file --- %s' % filename)
def is_tor_and_privoxy_used(self, response):
print('
Spider: Start')
print("My IP is : " + str(response.body))
print("Is proxy in response.meta?: ", response.meta) # not header dispo
print('
Spider: End')
self.log('Saved file %s' % filename)
您还需要在middleware.py和settings.py中添加内容。如果你不知道怎么做this will help you
以上是关于如何使用Scrapy Tor Privoxy和UserAgent匿名废弃? (Windows 10)的主要内容,如果未能解决你的问题,请参考以下文章