05数据爬取-补

Posted 2020-12-17 minadukirinno

修改自一个爬取数据并输入省份查询的脚本。

 

import requests
import json
import mysql.connector
import time

def Down_data():#获取JSON
    url = ‘https://view.inews.qq.com/g2/getOnsInfo?name=disease_h5‘
    print="%int(time.mktime( t )*1000)‘"
    headers = {#伪装浏览器请求头
        ‘user-agent‘: ‘Mozilla/5.0 (Linux; android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (Khtml, like Gecko) Chrome/78.0.3904.108 Mobile Safari/537.36‘
    }
    r = requests.get(url, headers)
    res = json.loads(r.text)
    data_res = json.loads(res[‘data‘])
    return data_res
 
mydb = mysql.connector.connect(#连接数据库
  host="localhost",
  user="root",
  passwd="[e-mail]",
  database="uu_zaa_db"
)
mycursor = mydb.cursor()
#开始搞事
data = Down_data()[‘areaTree‘][0][‘children‘]#便于后面操作
lastTime = Down_data()[‘lastUpdateTime‘]#爬取的JSON更新日期在data--lastUpdateTime
tm = 1
for i in data:
    #if i[‘children‘]:#判断是否有children,以此筛选掉外国的数据(根据分析JSON发现外国的数据没有children,当然也可以写死所有省份名然后筛选,但是我懒得写)
    if tm:
        sql = "INSERT INTO info1 (Date, Province, City, Confirmed_num, Yisi_num, Cured_num, Dead_num) VALUES (%s, %s, %s, %s, %s, %s, %s)"
        ipt = (str(lastTime), str(i[‘name‘]), str(‘‘), str(i[‘total‘][‘confirm‘]), str(‘‘), str(i[‘total‘][‘heal‘]), str(i[‘total‘][‘dead‘]))#储存省份信息,根据表里规则省份的那行的City为空串
        mycursor.execute(sql, ipt)
        for item in i[‘children‘]:#各个城市信息,
            ipt = (str(lastTime), str(i[‘name‘]), str(item[‘name‘]), str(item[‘total‘][‘confirm‘]), str(‘‘), str(item[‘total‘][‘heal‘]), str(item[‘total‘][‘dead‘]))
            mycursor.execute(sql, ipt)

mydb.commit()#提交更改

 

Python其实上手很快，个人因为本身就有(自认为)规范的缩进习惯，对于Python的强制缩进等等没什么感受。