2020寒假生活学习日记（十五）

Posted 2021-03-10 jccjcc

后来在用JAVA爬取北京信件内容过程中出现好多问题。

我该用python爬取。

 

 这个是我爬取出来的各个信件网址的后缀即（http://www.beijing.gov.cn/hudong/hdjl/com.web.suggest.suggesDetail.flow?originalId=AH20021200370）

然后编写代码：

import requests
import re
import xlwt
# #https://flightaware.com/live/flight/CCA101/history/80
url = ‘http://www.beijing.gov.cn/hudong/hdjl/com.web.consult.consultDetail.flow?originalId=AH20021300174‘
headers = {
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (Khtml, like Gecko) Chrome/77.0.3865.90 Safari/537.36"
}
def get_page(url):
    try:
        response = requests.get(url, headers=headers)
        if response.status_code == 200:
            #print(‘获取网页成功‘)
            return response.text
        else:
            print(‘获取网页失败‘)
    except Exception as e:
        print(e)
fopen = open(‘C:UsershpDesktoplist.txt‘, ‘r‘)//这个是存取信件内容网址后缀
lines = fopen.readlines()
urls = [‘http://www.beijing.gov.cn/hudong/hdjl/com.web.consult.consultDetail.flow?originalId={}‘.format(line) for line in lines]
for url in urls:
    print(url)
    page = get_page(url)
    items = re.findall(‘‘,page,re.S)
    print(items)
    print(len(items))

　　但是在用正则法爬取内容的时候出现了一些问题。之后问题解决再次编辑。