Python爬虫应用

Posted 东菇敖

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Python爬虫应用相关的知识,希望对你有一定的参考价值。

个人灌水博文#1

本文使用python爬虫爬取学校内部网信箱内容,并将内容做成词云来直观获取学生最需要解决的问题

涉及到了爬虫,需要登陆验证网页的爬虫爬取,词云的制作

主要实现思路:用带有cookie信息的爬虫爬取学校内部网校务信箱信息,将信息通过jieba库分词并通过wordcloud库来生成词库

程序主体分为五个部分:

1、程序所使用的库的信息:

# coding:utf-8
import requests    
from bs4 import BeautifulSoup    
import re    
import jieba    
import wordcloud
from cv2 import imread

其中requests,BeautifulSoup,re库用于爬取信息,jieba,wordcloud,imread库用于生成词云 

2、爬取网页部分: 

def gethtml(Cookie,url):    #用于爬取网页内容
    headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.75 Chrome/73.0.3683.75 Safari/537.36',
    'Cookie': Cookie    #Cookie信息用于登陆网页
    }

    session = requests.Session()

    response = session.get(url, headers=headers)
    response.encoding = response.apparent_encoding
    return response.text

 关键在于使用cookie信息来登陆网页进行信息的爬取

3、将爬取到的信息放入列表中并过滤掉一些无用的信息:

def fill_List(List,HTML):   #将爬取的信息放于列表中
    soup = BeautifulSoup(HTML,"html.parser")
    for i in soup.find_all(string = re.compile("\\xa0")):
        o = "".join(i.split())
        if o != '' and o != '检索结果:2018年起信件共' and o != '共358页直接跳到第' and o != '信件内容':
            List.append(o)
    pass

 因为笔者爬取的网页是学校的校务信箱,较多内容固定,所以可以剔除掉一些无用的信息

4、由列表生成词云部分:

def WordCloud(list):    #词云部分,将列表分词并生成词云
    #mk = imread('C:/Users/BoletusAo/Desktop/wordcloud.png')
    w = wordcloud.WordCloud(width=1920,height=1080,font_path="msyh.ttc",stopwords = {"问题","的","疑问","关于","对于","故障","可以","情况","建议","投诉","反馈","真的","能","不能","能否","呢","还是","楼","你","中","与","为什么","的一些","请问","已经","回复","要","疑惑","点","了吗","人","怎么","吗","是","又","也","我们","级","无法","一直","很","是不是","等","意见","以及","处理","部分","好","多","这","为","被","未","后","就","吧","啊","里","了","时候","什么","还","一点","一个","使用","在","晚上","希望","何时","想","存在","不","和","有","让","没","及","请","到","通知","是否","有关","为何","用","对","严重","解决","不合理","让","没有","我","都","不了","新","正常","导致","出现","一下","开", \\
                                                                                     "深大","经常","差","说","作为","一些","最近","服务","稳定","人员","安排","吃","上","上课","再"},background_color="white")
    str = ",".join(list)
    jieba.setLogLevel(jieba.logging.INFO)
    w.generate(" ".join(jieba.lcut(str)))
    w.to_file('C:/Users/BoletusAo/Desktop/SZU2.png')
    pass

生成词云,关键在于屏蔽词的设置,信件标题有较多无用的词语需要屏蔽,例如:问题,的等 

5、main函数:

if __name__ == '__main__':
    List = []
    for i in range (1,359):    #校务信箱一共有358页
        if i == 1:
            url = 'url信息'
            Cookie = 'cooki信息'
            HTML = getHTML(Cookie,url)
            fill_List(List,HTML)
        else:
            url = 'url信息{}'.format(i)    #此处涉及到换页的相关操作,使用的是直接更改url信息的方法
            Cookie = 'coookie信息'
            HTML = getHTML(Cookie, url)
            fill_List(List, HTML)
    WordCloud(List)

具体url,cookie和翻页的处理因涉及到学校信息固不提供 

最终结果展示:

编写程序中遇到的问题:

1、需要登陆验证的网页该如何爬取

解决方法:使用cookie信息

2、如何在爬取的过程中进行翻页

更改url的相关信息实现翻页

以上是关于Python爬虫应用的主要内容,如果未能解决你的问题,请参考以下文章

Python练习册 第 0013 题: 用 Python 写一个爬图片的程序,爬 这个链接里的日本妹子图片 :-),(http://tieba.baidu.com/p/2166231880)(代码片段

python爬虫学习笔记-M3U8流视频数据爬虫

爬虫遇到头疼的验证码?Python实战讲解弹窗处理和验证码识别

python网络爬虫

我在哪里更改此 Python 代码片段以将临时文件保存在 tmp 文件夹中?

python 爬虫 ip池怎么做