Python鐖彇鍙ュ瓙杩?鑾庡＋姣斾簹璇綍
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Python鐖彇鍙ュ瓙杩?鑾庡＋姣斾簹璇綍相关的知识，希望对你有一定的参考价值。
鏍囩锛?a href='http://www.mamicode.com/so/1/%e5%ae%8c%e6%95%b4' title='瀹屾暣'>瀹屾暣
list 瀛樺偍 涓嬭浇 鍘熺悊 line 鎵嬫満 name 璁剧疆 宸ュ叿浣跨敤鐨勬槸 Python3.7 + requests + BeautifulSoup4 + 绾跨▼
棣栧厛鍙ュ瓙杩锋槸鍏锋湁涓€瀹氬弽鐖▼搴忕殑锛屽叿浣撴槸鎬庢牱鐨勫氨涓嶆竻妤氾紝浣嗘槸鏂囧瓧杩樻槸鏁存暣榻愰綈鎽嗘斁鐫€锛岃繖涓繕鏄緢鍙嬪ソ鐨勶紝鍓嶇椤甸潰鍒嗘瀽瀹屽氨寮€濮嬫垜鏁版鐖彇灏濊瘯銆?/p>
绗簩銆佷笁娆″皾璇?/h3>
鎴戜竴鏃╄捣鏉ワ紝鍙戠幇鍙ュ瓙杩疯繖涓綉绔欏張鑳借闂簡锛屽紑蹇冪殑寰堝憪锛岃繕浠ヤ负琚皝浜嗗氨涓嶈兘鍐嶈闂簡鍛紝鐒跺悗灏卞紑濮嬩簡鎴戠浜屾灏濊瘯锛岃繖娆℃垜鎯崇潃涓嶈兘杩欎箞鏄庣洰寮犺儐浜嗭紝寰楃尌鐞愪竴鐐圭殑锛岀敤鏃堕棿鎹㈠彇瀹屾暣涓旀垚鍔熺殑缁撴灉銆?/p>
褰撶劧杩欎釜鏂规硶灏辨槸鍒╃敤sleep杩涜浼鍜紝鍥犱负涓€鍏?29椤碉紝鎴戣缃瘡鐖彇5椤靛氨闅忔満浼戞伅5浠ュ唴绉掑摝锛?code>time.sleep(random.randint(1,5))
绗簲娆″皾璇?/h3>
杩欎竴娆℃垜鎯充娇鐢ㄧ殑鏄嚎绋嬶紝閫氳繃html浠ｇ爜涓?code>pager-last
閮ㄥ垎浠ｇ爜濡備笅
#coding=utf-8
import requests
from requests import codes
import io
import bs4
import os
import random
import time
import threading
from bs4 import BeautifulSoup
webUrl = 鈥榟ttps://www.juzimi.com/鈥?headers = {鈥榰ser-agent鈥?鈥楳ozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36鈥榼
def downLoadPageExtra(url):
try:
realUrl = webUrl + 鈥?writer/%E8%8E%8E%E5%A3%AB%E6%AF%94%E4%BA%9A鈥?"?page="+str(url)
content = requests.get(realUrl,headers = headers)
content.encoding = "UTF-8"
if codes.ok == content.status_code:
soup = BeautifulSoup(content.text,鈥榟tml.parser鈥?
downLoadTxt = soup.findAll(鈥榓鈥?class_=鈥榵listju鈥?
f = open("juzimi"+ "/" + str(url) + ".txt", 鈥榓鈥? encoding=鈥榰tf-8鈥?
for dlt in downLoadTxt:
f.writelines((dlt.text)+"
")
print(str(url)+"OK")
f.close()
else:
print("鐘舵€佺爜鍑洪敊")
except:
print("getPageCodeError1")
def downLoadPage(url):
try:
content = requests.get(url,headers = headers)
content.encoding = "UTF-8"
if codes.ok == content.status_code:
soup = BeautifulSoup(content.text,鈥榟tml.parser鈥?
downLoadTxt = soup.findAll(鈥榓鈥?class_=鈥榵listju鈥?
#鑾峰彇鏍囬鍒涘缓鏂囦欢
title = str(soup.title.string).strip(鈥?鈥?.split(鈥?鈥?[0]
#鑾峰彇椤垫暟
pageCounts = soup.find(鈥榣i鈥?class_=鈥榩ager-last鈥?.find_all(鈥榓鈥?[0].text
f=open("juzimi"+"/"+ title + ".txt", 鈥榓鈥?encoding=鈥榰tf-8鈥?
for dlt in downLoadTxt:
f.writelines(str(dlt.text)+"
")
f.close()
print("绗竴鍙ュ畬鎴?)
for num in range(1,int(pageCounts)):
t = threading.Thread(target=downLoadPageExtra,args=(num,))
t.start()
print(鈥樼埇鍙栧畬鎴?..鈥?
except requests.ConnectionError:
print("getPageCodeError2")
if __name__ == 鈥榑_main__鈥?
dirSave = 鈥榡uzimi/鈥? if os.path.exists(dirSave) is False:
os.makedirs(dirSave)
downLoadPage(webUrl + 鈥?writer/%E8%8E%8E%E5%A3%AB%E6%AF%94%E4%BA%9A鈥?
浠ｇ爜鍏蜂綋濡備笂锛屼笉淇濊瘉鐧惧垎鐧剧洿鎺ヨ兘杩愯锛屽洜涓烘槸浠庨」鐩腑澶嶅埗杩囨潵骞朵笖鍋氫簡閮ㄥ垎淇敼锛屼絾鏄ぇ鑷村師鐞嗗凡璇存槑锛屼娇鐢ㄥ绾跨▼锛屽洜涓烘槸IO瀵嗛泦鍨嬶紝鎵€浠ュ绾跨▼杩樻槸鎸哄ソ浣跨殑锛岄€熷害閭ｆ槸鐩稿綋鐨勫揩鍟婏紝鍑犲崄绉掑氨閫氶€氭悶瀹氾紝褰撶劧鎴戜滑濡傛灉闇€瑕佸皢鎵€鏈塼xt鏂囦欢鍚堝苟鎴愪竴涓殑璇濓紝鎴戝張鍐欎簡涓€涓悎骞剁▼搴忋€?/p>
浠ｇ爜濡備笅
import io
# 閫氳繃html椤甸潰鏍囬鍛藉悕鐨則xt鏂囦欢锛屽苟涓斿瓨鍌ㄧ殑绗竴椤电殑鍙ュ瓙
mergeTxtName = 鈥樿帋澹瘮浜氱粡鍏歌褰昣鍚嶈█_鍚嶅彞璧忔瀽_鍙ュ瓙杩?txt鈥?# 寮€濮嬫爣璁?start = 1
# 缁撴潫鏍囪
end = 228
f = open(mergeTxtName,鈥榓鈥?encoding=鈥榰tf-8鈥?
for n in range(start,end + 1):
g = open(str(n)+鈥?txt鈥?鈥榬鈥?encoding=鈥榰tf-8鈥?
f.write(g.read())
print(str(n)+"瀹屾垚")
g.close()
f.close()
print("ok")
姝や唬鐮佹枃浠跺簲褰撴斁缃湪涓嬭浇鐩綍涓紝鍚﹀垯璇蜂慨鏀瑰搴旇矾寰勶紝鑻ュ苟涓嶆槸鑾峰彇鑾庡＋姣斾簹鐨勫彞瀛愶紝杩橀渶瑕佷慨鏀筸ergeTxtName瀛楃涓诧紝姝や唬鐮佷笉鎻愪緵鍒犻櫎宸插悎骞剁殑txt鏂囦欢锛岃嫢闇€瑕佸彲鑷澧炲姞...鑷啓绋嬪簭鏃惰娉ㄦ剰鏂囦欢璇诲啓鎿嶄綔鏃剁殑瀛椾綋缂栫爜闂...
鎬荤粨
鍐欑埇铏綋澶氬缁冩墜锛屽熀纭€鐨勭啛缁冿紝鍐嶆湁閽堝鐨勬彁楂橀毦搴︼紝寰佹湇鏇村缃戠珯锛岀涓€娆″皾璇曚娇鐢ㄥ崥瀹㈠洯鐨刴arkdown椋庢牸锛岀浉姣旇緝鏈夐亾浜戠瑪璁版湰鐨勪功鍐欙紝杩欓噷涓嶆槸寰堜範鎯紝鍏堣瘯鐫€涓€娆★紝涓汉鎰熻鏁堟灉濂界殑璇濅互鍚庡氨鐢ㄨ繖绉嶆柟寮忓啓浜嗭紝瑕佹槸鏈変竴涓疄鏃堕瑙堣鏈夊濂藉晩銆?/p>
鐖彇浜嗚帋澹瘮浜氱殑鍙ュ瓙鍚庡苟涓嶆槸婊¤冻浜庢妧鏈殑鎻愬崌锛岃鍒板簳骞舵病鏈夊灏戞彁鍗囷紝鎬诲綊鐨勬潵璇磋繕鏄睘浜庨潤鎬佺晫闈紝鍒濆績鍒欐槸鎯抽槄瑙堜竴涓嬭繖浜涙湁瓒ｇ殑鍙ュ瓙锛屾湁涓€浜涜瘽璇啓鐨勭湡鐨勫緢濂斤紝骞舵垚涓烘垜鐨勫骇鍙抽摥锛?/strong>
鎴戣崚搴熶簡鏃堕棿锛屾椂闂翠究鎶婃垜鑽掑簾浜嗐€?/p>
以上是关于Python鐖彇鍙ュ瓙杩?鑾庡＋姣斾簹璇綍的主要内容，如果未能解决你的问题，请参考以下文章
python 鐖彇涔屼簯鎵€鏈夊巶鍟嗗悕瀛楋紝url锛屾紡娲炴€绘暟 骞跺瓨鍏ユ暟鎹簱
Python鐖櫕瀹炶返 鈥斺€?3.鍒╃敤鐖櫕鎻愬彇杩斿洖鍊硷紝妯℃嫙鏈夐亾璇嶅吀鎺ュ彛