Python爬虫应用

Posted 2021-09-02 东菇敖

个人灌水博文#1

本文使用python爬虫爬取学校内部网信箱内容，并将内容做成词云来直观获取学生最需要解决的问题

涉及到了爬虫，需要登陆验证网页的爬虫爬取，词云的制作

主要实现思路：用带有cookie信息的爬虫爬取学校内部网校务信箱信息，将信息通过jieba库分词并通过wordcloud库来生成词库

程序主体分为五个部分：

1、程序所使用的库的信息：

# coding:utf-8
import requests    
from bs4 import BeautifulSoup    
import re    
import jieba    
import wordcloud
from cv2 import imread

其中requests，BeautifulSoup，re库用于爬取信息，jieba，wordcloud，imread库用于生成词云 

2、爬取网页部分： 

def gethtml(Cookie,url):    #用于爬取网页内容
    headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.75 Chrome/73.0.3683.75 Safari/537.36',
    'Cookie': Cookie    #Cookie信息用于登陆网页
    }

    session = requests.Session()

    response = session.get(url, headers=headers)
    response.encoding = response.apparent_encoding
    return response.text

 关键在于使用cookie信息来登陆网页进行信息的爬取

3、将爬取到的信息放入列表中并过滤掉一些无用的信息：

def fill_List(List,HTML):   #将爬取的信息放于列表中
    soup = BeautifulSoup(HTML,"html.parser")
    for i in soup.find_all(string = re.compile("\\xa0")):
        o = "".join(i.split())
        if o != '' and o != '检索结果：2018年起信件共' and o != '共358页直接跳到第' and o != '信件内容':
            List.append(o)
    pass

 因为笔者爬取的网页是学校的校务信箱，较多内容固定，所以可以剔除掉一些无用的信息

4、由列表生成词云部分：

def WordCloud(list):    #词云部分，将列表分词并生成词云
    #mk = imread('C:/Users/BoletusAo/Desktop/wordcloud.png')
    w = wordcloud.WordCloud(width=1920,height=1080,font_path="msyh.ttc",stopwords = {"问题","的","疑问","关于","对于","故障","可以","情况","建议","投诉","反馈","真的","能","不能","能否","呢","还是","楼","你","中","与","为什么","的一些","请问","已经","回复","要","疑惑","点","了吗","人","怎么","吗","是","又","也","我们","级","无法","一直","很","是不是","等","意见","以及","处理","部分","好","多","这","为","被","未","后","就","吧","啊","里","了","时候","什么","还","一点","一个","使用","在","晚上","希望","何时","想","存在","不","和","有","让","没","及","请","到","通知","是否","有关","为何","用","对","严重","解决","不合理","让","没有","我","都","不了","新","正常","导致","出现","一下","开", \\
                                                                                     "深大","经常","差","说","作为","一些","最近","服务","稳定","人员","安排","吃","上","上课","再"},background_color="white")
    str = ",".join(list)
    jieba.setLogLevel(jieba.logging.INFO)
    w.generate(" ".join(jieba.lcut(str)))
    w.to_file('C:/Users/BoletusAo/Desktop/SZU2.png')
    pass

生成词云，关键在于屏蔽词的设置，信件标题有较多无用的词语需要屏蔽，例如：问题，的等 

5、main函数：

if __name__ == '__main__':
    List = []
    for i in range (1,359):    #校务信箱一共有358页
        if i == 1:
            url = 'url信息'
            Cookie = 'cooki信息'
            HTML = getHTML(Cookie,url)
            fill_List(List,HTML)
        else:
            url = 'url信息{}'.format(i)    #此处涉及到换页的相关操作，使用的是直接更改url信息的方法
            Cookie = 'coookie信息'
            HTML = getHTML(Cookie, url)
            fill_List(List, HTML)
    WordCloud(List)

具体url，cookie和翻页的处理因涉及到学校信息固不提供 

最终结果展示：

编写程序中遇到的问题：

1、需要登陆验证的网页该如何爬取

解决方法：使用cookie信息

2、如何在爬取的过程中进行翻页

更改url的相关信息实现翻页