爬取校园新闻首页的新闻

Posted 2020-10-29 Ming.杰

1. 用requests库和BeautifulSoup库，爬取校园新闻首页新闻的标题、链接、正文。

import requests
newsurl = \'http://news.gzcc.cn/html/xiaoyuanxinwen/\'
res = requests.get(newsurl)  # 返回response对象
res.encoding = \'utf-8\'
from bs4 import BeautifulSoup
soup = BeautifulSoup(res.text, \'html.parser\')

for news in soup.select("li"):

    if(len(news.select(\'.news-list-info\'))>0):
        print(news.select(\'a\')[0].attrs[\'href\'])
        print(news.select(\'.news-list-title\')[0].text)
        a = news.select(\'a\')[0].attrs[\'href\']
        rq = requests.get(a)
        rq.encoding="utf-8"
        soup2 =  BeautifulSoup(rq.text, \'html.parser\')
        print(soup2.select("#content")[0].text)

 

2. 分析字符串，获取每篇新闻的发布时间，作者，来源，摄影等信息。

info = soup.select(".show-info")[0].text

    author = info[info.find(\'作者：\'):].split()[0].lstrip(\'作者：\')
    source = info[info.find(\'来源：\'):].split()[0].lstrip(\'来源：\')
    Auditing = info[info.find(\'审核：\'):].split()[0].lstrip(\'审核：\')
    photo = info[info.find(\'摄影：\'):].split()[0].lstrip(\'摄影：\')

    print(author,source,Auditing,photo)

3. 将其中的发布时间由str转换成datetime类型。

        from datetime import datetime
        da = datetime.strptime(dt,"%Y-%m-%d %H:%M:%S")
        print(type(da))
        print(da,writer,sh,ly)
        break;