scrapy爬取猫眼电影排行榜

Posted lattesea

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了scrapy爬取猫眼电影排行榜相关的知识,希望对你有一定的参考价值。

做爬虫的人,一定离不开的一个框架就是scrapy框架,写小项目的时候可以用requests模块就能得到结果,但是当爬取的数据量大的时候,就一定要用到框架.

下面先练练手,用scrapy写一个爬取猫眼电影的程序,环境配置和scrapy安装略过

第一步肯定是终端运行创建爬虫项目和文件

1 # 创建爬虫项目
2 scrapy startproject Maoyan
3 cd Maoyan
4 # 创建爬虫文件
5 scrapy genspider maoyan maoyan.com

然后在产生的items.py文件夹中定义需要爬取的数据结构

1 name = scrapy.Field()
2 star = scrapy.Field()
3 time = scrapy.Field()

之后打开maoyan.py文件,编写爬虫文件,记得导入items.py文件的MaoyanItem类,并实例化

 1 import scrapy
 2 from ..items import MaoyanItem
 3 ?
 4 class MaoyanSpider(scrapy.Spider):
 5     name = maoyan3
 6     allowed_domains = [maoyan.com]
 7     # 去掉start_urls变量
 8 ?
 9     # 重写start_requests()方法
10     def start_requests(self):
11         for offset in range(0,91,10):
12             url = https://maoyan.com/board/4?offset={}.format(offset)
13             yield scrapy.Request(url=url,callback=self.parse)
14 ?
15     def parse(self, response):
16         # 给items.py中的类:MaoyanItem(scrapy.Item)实例化
17         item = MaoyanItem()
18 ?
19         # 基准xpath
20         dd_list = response.xpath(//dl[@class="board-wrapper"]/dd)
21         # 依次遍历
22         for dd in dd_list:
23             # 是在给items.py中那些类变量赋值
24             item[name] = dd.xpath(./a/@title).get().strip()
25             item[star] = dd.xpath(.//p[@class="star"]/text()).get().strip()
26             item[time] = dd.xpath(.//p[@class="releasetime"]/text()).get().strip()
27 ?
28             # 把item对象交给管道文件处理
29             yield item

定义管道文件pipelines.py,进行持久化储存

 1 class MaoyanPipeline(object):
 2     # item: 从爬虫文件maoyan.py中yield的item数据
 3     def process_item(self, item, spider):
 4         print(item[name],item[time],item[star])
 5 ?
 6         return item
 7 ?
 8 ?
 9 import pymysql
10 from .settings import *
11 ?
12 # 自定义管道 - MySQL数据库
13 class MaoyanMysqlPipeline(object):
14     # 爬虫项目开始运行时执行此函数
15     def open_spider(self,spider):
16         print(我是open_spider函数输出)
17         # 一般用于建立数据库连接
18         self.db = pymysql.connect(
19             host = MYSQL_HOST,
20             user = MYSQL_USER,
21             password = MYSQL_PWD,
22             database = MYSQL_DB,
23             charset = MYSQL_CHAR
24         )
25         self.cursor = self.db.cursor()
26 ?
27     def process_item(self,item,spider):
28         ins = insert into filmtab values(%s,%s,%s)
29         # 因为execute()的第二个参数为列表
30         L = [
31             item[name],item[star],item[time]
32         ]
33         self.cursor.execute(ins,L)
34         self.db.commit()
35 ?
36         return item
37 ?
38     # 爬虫项目结束时执行此函数
39     def close_spider(self,spider):
40         print(我是close_spider函数输出)
41         # 一般用于断开数据库连接
42         self.cursor.close()
43         self.db.close()

接下来就是修改配置文件settings.py

 1 USER_AGENT = Mozilla/5.0
 2 ROBOTSTXT_OBEY = False
 3 DEFAULT_REQUEST_HEADERS = {
 4   Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8,
 5   Accept-Language: en,
 6 }
 7 ITEM_PIPELINES = {
 8    Maoyan.pipelines.MaoyanPipeline: 300,
 9    Maoyan.pipelines.MaoyanMysqlPipeline:200,
10 }
11 # 定义MySQL相关变量
12 MYSQL_HOST = 127.0.0.1
13 MYSQL_USER = root
14 MYSQL_PWD = 123456
15 MYSQL_DB = maoyandb
16 MYSQL_CHAR = utf8

最后,是创建run.py文件,然后就可以运行了

1 from scrapy import cmdline
2 cmdline.execute(scrapy crawl maoyan.split())

 

以上是关于scrapy爬取猫眼电影排行榜的主要内容,如果未能解决你的问题,请参考以下文章

对猫眼电影排行的爬取

使用xpath爬取猫眼电影排行榜

python爬取猫眼代码没

Python3编写网络爬虫04-爬取猫眼电影排行实例

Python实战之如何爬取猫眼电影排行?本文详解

抓取猫眼top100电影信息