scrapy 爬取知乎问题答案 ,并异步写入数据库(mysql)

Posted Chai-zz

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了scrapy 爬取知乎问题答案 ,并异步写入数据库(mysql)相关的知识,希望对你有一定的参考价值。

 

python版本  python2.7

爬取知乎流程:

 一 、分析 在访问知乎首页的时候(https://www.zhihu.com),在没有登录的情况下,会进行重定向到(https://www.zhihu.com/signup?next=%2F)这个页面,

  爬取知乎,首先要完成登录操作,登陆的时候观察往那个页面发送了post或者get请求。可以利用抓包工具来获取登录时密码表单等数据的提交地址。

 1、利用抓包工具,查看用户名密码数据的提交地址页就是post请求,将表单数据提交的网址,经过查看。是这个网址 ‘https://www.zhihu.com/api/v3/oauth/sign_in‘。

 2、通过抓取上述登录地址,在其请求的contenr字段中,发现post请求服务器不止包含用户名,密码,还有timetamp,lang,client_id,sihnature等表单数据,需要知道每一个表单数据的特点,而特点是我们数据变化 在每次登录的时候的变化来查找数据的规律。

 3、经过多次登录观察,这些表单数据中只有timetamp,和signature是变化的,其他的值是不变的。

   4、通过js发现 signature字段的值是有多个字段组合加密而成,其实timetamp时间戳是核心,每次根据时间的变化,生成不同的signature值。

5、考虑到signature的值加密较为复杂,直接将浏览器登陆成功后的时间戳timetamp和signature 复制到请求数据中,然后进行登录。
6、表单数据田中完毕,发送post请求时,出现了缺少验证码票据的错误(capsion_ticket) 经过分析验证码票据是为了获取验证码而提供的一种验证方式,
而抓包装工具中关于验证码的请求有两次, 一次获取的是:

{‘show_captcha‘:true}
而同时第二次获取的是:{‘img_base_64‘:Rfadausifpoauerfae}。
7、经过分析{‘show_captcha‘:true} 是获取验证码的关键信息,再抓包信息中发现第一次请求相应的set-cookie中,包含了capsion_ticket验证码票据信息。
8、在此模拟登陆又出现了错误‘ERR_xxx_AUTH_TOKEN‘错误信息,而她出现在我们很根据验证码票据获取验证码图片时,
我们从抓包中查看关于Authorization:oauth ce30dasjfsdjhfkiswdnf.所以将其在headers当中进行配置。
验证码问题:
验证码问题
-对于知乎的验证码,有两种情况,一种是英文的图片验证码,一种是点击倒立文字的验证码,当登录需要验证码的时候,回向这两个网站发送数据
倒立文字验证码:https://www.zhihu.com/api/v3/oauth/captcha?lang=cn
英文图片验证码:https://www.zhihu.com/api/v3/oauth/captcha?lang=en
-英文验证码得到数据是四个英文字母。可采用云打码在线识别。
   -倒立文字验证码是得到的是每个汉字有一定的范围,当登陆的时候点击验证码的时候,
从https://www.zhihu.com/api/v3/oauth/captcha?lang=cn该网站获取到的一个像素点(x,y),比如倒立文字在第三个和第五个,就会有一个可选范围,只要输入合适的像素点 就可以登录。
  -只对倒立文字进行验证
  -只是简单地爬取第一页的问题及回答
二、创建scrapy项目
  scrapy startproject ZhiHuSpider
  scrapy genspider zhihu zhihu.com
三、代码
  在zhihu.py中代码如下:
  
  1 # -*- coding: utf-8 -*-
  2 import base64
  3 import json
  4 import urlparse
  5 import re
  6 from datetime import datetime
  7 import scrapy
  8 from scrapy.loader import ItemLoader
  9 from ..items import ZhiHuQuestionItem, ZhiHuAnswerItem
 10 
 11 
 12 class ZhihuSpider(scrapy.Spider):
 13     name = zhihu
 14     allowed_domains = [www.zhihu.com]
 15     start_urls = [https://www.zhihu.com]
 16     start_answer_url = "https://www.zhihu.com/api/v4/questions/{}/answers?include=data%5B%2A%5D.is_normal%2Cadmin_closed_comment%2Creward_info%2Cis_collapsed%2Cannotation_action%2Cannotation_detail%2Ccollapse_reason%2Cis_sticky%2Ccollapsed_by%2Csuggest_edit%2Ccomment_count%2Ccan_comment%2Ccontent%2Ceditable_content%2Cvoteup_count%2Creshipment_settings%2Ccomment_permission%2Ccreated_time%2Cupdated_time%2Creview_info%2Crelevant_info%2Cquestion%2Cexcerpt%2Crelationship.is_authorized%2Cis_author%2Cvoting%2Cis_thanked%2Cis_nothelp%2Cupvoted_followees%3Bdata%5B%2A%5D.mark_infos%5B%2A%5D.url%3Bdata%5B%2A%5D.author.follower_count%2Cbadge%5B%3F%28type%3Dbest_answerer%29%5D.topics&limit=20&offset={}&sort_by=default"
 17 
 18     headers = {
 19         User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0,
 20         Referer: https://www.zhihu.com,
 21         HOST: www.zhihu.com,
 22         Authorization: oauth c3cef7c66a1843f8b3a9e6a1e3160e20
 23     }
 24     points_list = [[20, 27], [42, 25], [65, 20], [90, 25], [115, 32], [140, 25], [160, 25]]
 25 
 26     def start_requests(self):
 27         """
 28         重写父类的start_requests()函数,在这里设置爬虫的起始url为登录页面的url。
 29         :return:
 30         """
 31         yield scrapy.Request(
 32             url=https://www.zhihu.com/api/v3/oauth/captcha?lang=cn,
 33             callback=self.captcha,
 34             headers=self.headers,
 35         )
 36 
 37     def captcha(self, response):
 38         show_captcha = json.loads(response.body)[show_captcha]
 39         if show_captcha:
 40             print u有验证码
 41             yield scrapy.Request(
 42                 url=https://www.zhihu.com/api/v3/oauth/captcha?lang=cn,
 43                 method=PUT,
 44                 headers=self.headers,
 45                 callback=self.shi_bie
 46             )
 47         else:
 48             print u没有验证码
 49             # 直接进行登录的操作
 50             post_url = https://www.zhihu.com/api/v3/oauth/sign_in
 51             post_data = {
 52                 client_id: c3cef7c66a1843f8b3a9e6a1e3160e20,
 53                 grant_type: password,
 54                 timestamp: 1515391742289,
 55                 source: com.zhihu.web,
 56                 signature: 6d1d179e50a06d1c17d6e8b5c89f77db34f406ac,
 57                 username: ,#账号
 58                 password: ,#密码
 59                 captcha: ‘‘,
 60                 lang: cn,
 61                 ref_source: homepage,
 62                 utm_source: ‘‘
 63             }
 64 
 65             yield scrapy.FormRequest(
 66                 url=post_url,
 67                 headers=self.headers,
 68                 formdata=post_data,
 69                 callback=self.index_page
 70             )
 71 
 72     def shi_bie(self, response):
 73         try:
 74             img= json.loads(response.body)[img_base64]
 75         except Exception, e:
 76             print 获取img_base64的值失败,原因:%s%e
 77         else:
 78             print 成功获取加密后的图片地址
 79             # 将加密后的图片进行解密,同时保存到本地
 80             img = img.encode(utf-8)
 81             img_data = base64.b64decode(img)
 82             with open(zhihu_captcha.GIF, wb) as f:
 83                 f.write(img_data)
 84 
 85             captcha = raw_input(请输入倒立汉字的位置:)
 86             if len(captcha) == 2:
 87                 # 说明有两个倒立的汉字
 88                 pass
 89                 first_char = int(captcha[0]) - 1 # 第一个汉字对应列表中的索引
 90                 second_char = int(captcha[1]) - 1 # 第二个汉字对应列表中的索引
 91                 captcha = {"img_size":[200,44],"input_points":[%s,%s]} % (self.points_list[first_char], self.points_list[second_char])
 92             else:
 93                 # 说明只有一个倒立的汉字
 94                 pass
 95                 first_char = int(captcha[0]) - 1
 96                 captcha = {"img_size":[200,44],"input_points":[%s]} % (
 97             self.points_list[first_char])
 98 
 99             data = {
100                 input_text: captcha
101             }
102             yield scrapy.FormRequest(
103                 url=https://www.zhihu.com/api/v3/oauth/captcha?lang=cn,
104                 headers=self.headers,
105                 formdata=data,
106                 callback=self.get_result
107             )
108 
109     def get_result(self, response):
110         try:
111             yan_zheng_result = json.loads(response.body)[success]
112         except Exception, e:
113             print 关于验证码的POST请求响应失败,原因:{}.format(e)
114         else:
115             if yan_zheng_result:
116                 print u验证成功
117                 post_url = https://www.zhihu.com/api/v3/oauth/sign_in
118                 post_data = {
119                     client_id: c3cef7c66a1843f8b3a9e6a1e3160e20,
120                     grant_type: password,
121                     timestamp: 1515391742289,
122                     source: com.zhihu.web,
123                     signature: 6d1d179e50a06d1c17d6e8b5c89f77db34f406ac,
124                     username: ,#账号
125                     password: ,#密码
126                     captcha: ‘‘,
127                     lang: cn,
128                     ref_source: homepage,
129                     utm_source: ‘‘
130                 }
            #以上数据需要在抓包中获取
131 132 yield scrapy.FormRequest( 133 url=post_url, 134 headers=self.headers, 135 formdata=post_data, 136 callback=self.index_page 137 ) 138 else: 139 print u是错误的验证码! 140 141 def index_page(self, response): 142 for url in self.start_urls: 143 yield scrapy.Request( 144 url=url, 145 headers=self.headers 146 ) 147 148 def parse(self, response): 149 """ 150 提取首页中的所有问题的url,并对这些url进行进一步的追踪,爬取详情页的数据。 151 :param response: 152 :return: 153 """ 154 # /question/19618276/answer/267334062 155 all_urls = response.xpath(//a[@data-za-detail-view-element_name="Title"]/@href).extract() 156 all_urls = [urlparse.urljoin(response.url, url) for url in all_urls] 157 for url in all_urls: 158 # https://www.zhihu.com/question/19618276/answer/267334062 159 # 同时提取:详情的url;文章的ID; 160 result = re.search((.*zhihu.com/question/(\d+)), url) 161 if result: 162 detail_url = result.group(1) 163 question_id = result.group(2) 164 # 将详情url交由下载器去下载网页源码 165 yield scrapy.Request( 166 url=detail_url, 167 headers=self.headers, 168 callback=self.parse_detail_question, 169 meta={ 170 question_id: question_id, 171 } 172 ) 173 174 # 在向详情url发送请求的同时,根据问题的ID,同时向问题的url发送请求。由于问题和答案是两个独立的url。而答案其实是一个JSON的API接口,直接请求即可,不需要和问题url产生联系。 175 yield scrapy.Request( 176 # 参数:问题ID,偏移量。默认偏移量为0,从第一个答案开始请求 177 url=self.start_answer_url.format(question_id, 0), 178 headers=self.headers, 179 callback=self.parse_detail_answer, 180 meta={ 181 question_id: question_id 182 } 183 ) 184 185 break 186 187 def parse_detail_question(self, response): 188 """ 189 用于处理详情页面关于question问题的数据,比如:问题名称,简介,浏览数,关注者数等 190 :param response: 191 :return: 192 """ 193 item_loader = ItemLoader(item=ZhiHuQuestionItem(), response=response) 194 item_loader.add_value(question_id, response.meta[question_id]) 195 item_loader.add_xpath(question_title, //div[@class="QuestionHeader"]//h1/text()) 196 item_loader.add_xpath(question_topic, //div[@class="QuestionHeader-topics"]//div[@class="Popover"]/div/text()) 197 # 获取的问题中,可能会不存在简介 198 item_loader.add_xpath(question_content, //span[@class="RichText"]/text()) 199 item_loader.add_xpath(question_watch_num, //button[contains(@class, "NumberBoard-item")]//strong/text()) 200 item_loader.add_xpath(question_click_num, //div[@class="NumberBoard-item"]//strong/text()) 201 item_loader.add_xpath(question_answer_num, //h4[@class="List-headerText"]/span/text()) 202 item_loader.add_xpath(question_comment_num, //div[@class="QuestionHeader-Comment"]/button/text()) 203 item_loader.add_value(question_url, response.url) 204 item_loader.add_value(question_crawl_time, datetime.now()) 205 206 question_item = item_loader.load_item() 207 yield question_item 208 209 def parse_detail_answer(self, response): 210 """ 211 用于解析某一个问题ID对应的所有答案。 212 :param response: 213 :return: 214 """ 215 answer_dict = json.loads(response.body) 216 is_end = answer_dict[paging][is_end] 217 next_url = answer_dict[paging][next] 218 219 for answer in answer_dict[data]: 220 answer_item = ZhiHuAnswerItem() 221 answer_item[answer_id] = answer[id] 222 answer_item[answer_question_id] = answer[question][id] 223 answer_item[answer_author_id] = answer[author][id] 224 answer_item[answer_url] = answer[url] 225 answer_item[answer_comment_num] = answer[comment_count] 226 answer_item[answer_praise_num] = answer[voteup_count] 227 answer_item[answer_create_time] = answer[created_time] 228 answer_item[answer_content] = answer[content] 229 answer_item[answer_crawl_time] = datetime.now() 230 answer_item[answer_update_time] = answer[updated_time] 231 232 yield answer_item 233 234 # 判断is_end如果值为False,说明还有下一页 235 if not is_end: 236 yield scrapy.Request( 237 url=next_url, 238 headers=self.headers, 239 callback=self.parse_detail_answer 240 )

  item.py中代码:

    

 1 # -*- coding: utf-8 -*-
 2 
 3 # Define here the models for your scraped items
 4 #
 5 # See documentation in:
 6 # https://doc.scrapy.org/en/latest/topics/items.html
 7 
 8 from datetime import datetime
 9 import scrapy
10 from utils.common import extract_num
11 
12 
13 class ZhihuspiderItem(scrapy.Item):
14     # define the fields for your item here like:
15     # name = scrapy.Field()
16     pass
17 
18 
19 class ZhiHuQuestionItem(scrapy.Item):
20     question_id=scrapy.Field()              # 问题ID
21     question_title = scrapy.Field()         # 问题标题
22     question_topic = scrapy.Field()         # 问题分类
23     question_content = scrapy.Field()       # 问题内容
24     question_watch_num = scrapy.Field()     # 关注者数量
25     question_click_num = scrapy.Field()     # 浏览者数量
26     question_answer_num = scrapy.Field()    # 回答总数
27     question_comment_num = scrapy.Field()   # 评论数量
28     question_crawl_time = scrapy.Field()    # 爬取时间
29     question_url = scrapy.Field()           # 问题详情url
30 
31     def get_insert_sql(self):
32         insert_sql = "insert into zhihu_question(question_id, question_title, question_topic, question_content, question_watch_num, question_click_num, question_answer_num, question_comment_num, question_crawl_time, question_url) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE question_id=VALUES(question_id),question_title=VALUES(question_title),question_topic=VALUES(question_topic),question_content=VALUES(question_content),question_watch_num=VALUES(question_watch_num),question_click_num=VALUES(question_click_num),question_answer_num=VALUES(question_answer_num),question_comment_num=VALUES(question_comment_num),question_crawl_time=VALUES(question_crawl_time),question_url=VALUES(question_url)"
33 
34         # 整理字段对应的数据
35         question_id = str(self[question_id][0])
36         question_title = ‘‘.join(self[question_title])
37         question_topic = ",".join(self[question_topic])
38 
39         try:
40             question_content = ‘‘.join(self[question_content])
41         except Exception,e:
42             question_content = question_content内容为空
43 
44         question_watch_num = ‘‘.join(self[question_watch_num]).replace(,, ‘‘)
45         question_watch_num = extract_num(question_watch_num)
46 
47         question_click_num = ‘‘.join(self[question_click_num]).replace(,, ‘‘)
48         question_click_num = extract_num(question_click_num)
49         # ‘86 回答‘
50         question_answer_num = ‘‘.join(self[question_answer_num])
51         question_answer_num = extract_num(question_answer_num)
52         # ‘100 条评论‘
53         question_comment_num = ‘‘.join(self[question_comment_num])
54         question_comment_num = extract_num(question_comment_num)
55 
56         question_crawl_time = self[question_crawl_time][0]
57         question_url = self[question_url][0]
58 
59         args_tuple = (question_id, question_title, question_topic, question_content, question_watch_num, question_click_num, question_answer_num, question_comment_num, question_crawl_time, question_url)
60 
61         return insert_sql, args_tuple
62 
63 
64 class ZhiHuAnswerItem(scrapy.Item):
65     answer_id = scrapy.Field()                  # 答案的ID (zhihu_answer表的主键)
66     answer_question_id = scrapy.Field()         # 问题的ID (zhihu_question表的主键)
67     answer_author_id = scrapy.Field()           # 回答用户的ID
68     answer_url = scrapy.Field()                 # 回答的url
69     answer_comment_num = scrapy.Field()         # 该回答的总评论数
70     answer_praise_num = scrapy.Field()          # 该回答的总点赞数
71     answer_create_time = scrapy.Field()         # 该回答的创建时间
72     answer_content = scrapy.Field()             # 回答的内容
73     answer_update_time = scrapy.Field()         # 回答的更新时间
74 
75     answer_crawl_time = scrapy.Field()          # 爬虫的爬取时间
76 
77     def get_insert_sql(self):
78         insert_sql = "insert into zhihu_answer(answer_id, answer_question_id, answer_author_id, answer_url, answer_comment_num, answer_praise_num, answer_create_time, answer_content, answer_update_time, answer_crawl_time) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE answer_id=VALUES(answer_id),answer_question_id=VALUES(answer_question_id),answer_author_id=VALUES(answer_author_id),answer_url=VALUES(answer_url),answer_comment_num=VALUES(answer_comment_num),answer_praise_num=VALUES(answer_praise_num),answer_create_time=VALUES(answer_create_time),answer_content=VALUES(answer_content),answer_update_time=VALUES(answer_update_time),answer_crawl_time=VALUES(answer_crawl_time)"
79 
80         # 处理answer_item中的数据
81         # fromtimestamp(timestamp):将一个时间戳数据转化为一个date日期类型的数据
82         answer_id = self[answer_id]
83         answer_question_id = self[answer_question_id]
84         answer_author_id = self[answer_author_id]
85         answer_url = self[answer_url]
86         answer_comment_num = self[answer_comment_num]
87         answer_praise_num = self[answer_praise_num]
88         answer_content = self[answer_content]
89         answer_create_time = datetime.fromtimestamp(self[answer_create_time])
90         answer_update_time = datetime.fromtimestamp(self[answer_update_time])
91         answer_crawl_time = self[answer_crawl_time]
92 
93         args_tuple = (answer_id, answer_question_id, answer_author_id, answer_url, answer_comment_num, answer_praise_num, answer_create_time, answer_content, answer_update_time, answer_crawl_time)
94 
95         return insert_sql, args_tuple

  

    pipeline,py代码如下:

    

 1 # -*- coding: utf-8 -*-
 2 
 3 # Define your item pipelines here
 4 #
 5 # Don‘t forget to add your pipeline to the ITEM_PIPELINES setting
 6 # See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
 7 
 8 import mysqldb
 9 import MySQLdb.cursors
10 from twisted.enterprise import adbapi
11 
12 # 数据库的异步写入操作。因为execute()及commit()提交数据库的方式是同步插入数据,一旦数据量比较大,scrapy的解析是异步多线程的方式,解析速度非常快,而数据库的写入速度比较慢,可能会导致item中的数据插入数据库不及时,造成数据库写入的阻塞,最终导致数据库卡死或者数据丢失。
13 
14 
15 class ZhihuspiderPipeline(object):
16     def process_item(self, item, spider):
17         return item
18 
19 
20 class MySQLTwistedPipeline(object):
21     def __init__(self, dbpool):
22         self.dbpool = dbpool
23 
24     @classmethod
25     def from_settings(cls, settings):
26         args = dict(
27             host=settings[MYSQL_HOST],
28             db=settings[MYSQL_DB],
29             user=settings[MYSQL_USER],
30             passwd=settings[MYSQL_PASSWD],
31             charset=settings[MYSQL_CHARSET],
32             cursorclass=MySQLdb.cursors.DictCursor
33         )
34         # 创建一个线程池对象
35         # 参数1:用于连接MySQL数据库的驱动
36         # 参数2:数据库的链接信息(host, port, user等)
37         dbpool = adbapi.ConnectionPool(MySQLdb, **args)
38         return cls(dbpool)
39 
40     def process_item(self, item, spider):
41         # 在线程池dbpool中通过调用runInteraction()函数,来实现异步插入数据的操作。runInteraction()会insert_sql交由线程池中的某一个线程执行具体的插入操作。
42         query = self.dbpool.runInteraction(self.insert, item)
43         # addErrorback()数据库异步写入失败时,会执行addErrorback()内部的函数调用。
44         query.addErrback(self.handle_error, item)
45 
46     def handle_error(self, failure, item):
47         print u插入数据失败,原因:{},错误对象:{}.format(failure, item)
48 
49     def insert(self, cursor, item):
50         pass
51         # 当存在多张表时,每一个表对应的数据,解析时间是不确定的,不太可能保证问题,答案同时能够解析完成,并且同时进入到pipeline中执行Insert的操作。
52         # 所以,不能再这个函数中,对所有的表执行execute()的操作。
53         # 解决办法:将sql语句在每一个Item类中实现。
54         # insert_question = ‘‘
55         # insert_answer = ‘‘
56         # insert_user = ‘‘
57         insert_sql, args = item.get_insert_sql()
58         cursor.execute(insert_sql, args)

 

setting.py代码如下:
  
 1 # -*- coding: utf-8 -*-
 2 
 3 # Scrapy settings for ZhiHuSpider project
 4 #
 5 # For simplicity, this file contains only settings considered important or
 6 # commonly used. You can find more settings consulting the documentation:
 7 #
 8 #     https://doc.scrapy.org/en/latest/topics/settings.html
 9 #     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
10 #     https://doc.scrapy.org/en/latest/topics/spider-middleware.html
11 
12 BOT_NAME = ZhiHuSpider
13 
14 SPIDER_MODULES = [ZhiHuSpider.spiders]
15 NEWSPIDER_MODULE = ZhiHuSpider.spiders
16 
17 
18 # Crawl responsibly by identifying yourself (and your website) on the user-agent
19 #USER_AGENT = ‘ZhiHuSpider (+http://www.yourdomain.com)‘
20 
21 # Obey robots.txt rules
22 ROBOTSTXT_OBEY = False
23 
24 # Configure maximum concurrent requests performed by Scrapy (default: 16)
25 #CONCURRENT_REQUESTS = 32
26 
27 # Configure a delay for requests for the same website (default: 0)
28 # See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
29 # See also autothrottle settings and docs
30 #DOWNLOAD_DELAY = 3
31 # The download delay setting will honor only one of:
32 #CONCURRENT_REQUESTS_PER_DOMAIN = 16
33 #CONCURRENT_REQUESTS_PER_IP = 16
34 
35 # Disable cookies (enabled by default)
36 #COOKIES_ENABLED = False
37 
38 # Disable Telnet Console (enabled by default)
39 #TELNETCONSOLE_ENABLED = False
40 
41 # Override the default request headers:
42 # DEFAULT_REQUEST_HEADERS = {
43 #   ‘Accept‘: ‘text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8‘,
44 #   ‘Accept-Language‘: ‘en‘,
45 # }
46 
47 # Enable or disable spider middlewares
48 # See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
49 #SPIDER_MIDDLEWARES = {
50 #    ‘ZhiHuSpider.middlewares.ZhihuspiderSpiderMiddleware‘: 543,
51 #}
52 
53 # Enable or disable downloader middlewares
54 # See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
55 #DOWNLOADER_MIDDLEWARES = {
56 #    ‘ZhiHuSpider.middlewares.ZhihuspiderDownloaderMiddleware‘: 543,
57 #}
58 
59 # Enable or disable extensions
60 # See https://doc.scrapy.org/en/latest/topics/extensions.html
61 #EXTENSIONS = {
62 #    ‘scrapy.extensions.telnet.TelnetConsole‘: None,
63 #}
64 
65 # Configure item pipelines
66 # See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
67 ITEM_PIPELINES = {
68    # ‘ZhiHuSpider.pipelines.ZhihuspiderPipeline‘: 300,
69     ZhiHuSpider.pipelines.MySQLTwistedPipeline:1,
70 }
71 
72 # Enable and configure the AutoThrottle extension (disabled by default)
73 # See https://doc.scrapy.org/en/latest/topics/autothrottle.html
74 #AUTOTHROTTLE_ENABLED = True
75 # The initial download delay
76 #AUTOTHROTTLE_START_DELAY = 5
77 # The maximum download delay to be set in case of high latencies
78 #AUTOTHROTTLE_MAX_DELAY = 60
79 # The average number of requests Scrapy should be sending in parallel to
80 # each remote server
81 #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
82 # Enable showing throttling stats for every response received:
83 #AUTOTHROTTLE_DEBUG = False
84 
85 # Enable and configure HTTP caching (disabled by default)
86 # See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
87 #HTTPCACHE_ENABLED = True
88 #HTTPCACHE_EXPIRATION_SECS = 0
89 #HTTPCACHE_DIR = ‘httpcache‘
90 #HTTPCACHE_IGNORE_HTTP_CODES = []
91 #HTTPCACHE_STORAGE = ‘scrapy.extensions.httpcache.FilesystemCacheStorage‘
92 
93 MYSQL_HOST = localhost‘# 本机端口,
94 MYSQL_DB = ‘      #数据库名字
95 MYSQL_USER = ‘    #数据库用户名
96 MYSQL_PASSWD = ‘  #密码
97 MYSQL_CHARSET = utf8

  另外设置了一个工具模块新建了一个python package.用来过滤item数据

    需要在item中导入模块

      代码如下:

      

1 import re
2 
3 
4 def extract_num(value):
5     result = re.search(re.compile((\d+)), value)
6     res = int(result.group(1))
7     return res

 

    


  



 

  






























以上是关于scrapy 爬取知乎问题答案 ,并异步写入数据库(mysql)的主要内容,如果未能解决你的问题,请参考以下文章

Python爬虫实战,Scrapy实战,爬取知乎表情包

python scrapy简单爬虫记录(实现简单爬取知乎)

运维学python之爬虫高级篇scrapy爬取知乎关注用户存入mongodb

scrapy爬取知乎问答

使用scrapy爬取知乎图片

爬虫实战--利用Scrapy爬取知乎用户信息