#yyds干货盘点#Python爬虫之Urllib用法合集

Posted 2022-03-02 是Dream呀

@TOC

一、何为爬虫

网络爬虫（又被称为网页蜘蛛，网络机器人，在FOAF社区中间，更经常的称为网页追逐者），是一种按照一定的规则，自动地抓取万维网信息的程序或者脚本。
我们平时的上网就是浏览器提交请求->下载网页代码->解析/渲染成页面。而我们的爬虫就是模拟浏览器发送请求->下载网页代码->只提取有用的数据->存放于数据库或文件中。所以，我们的爬虫程序只提取网页代码中对我们有用的数据。
如果我们把互联网比作一张大的蜘蛛网，那计算机上的数据便是蜘蛛网上的一个猎物，而`爬虫程序就是一只小蜘蛛，沿着蜘蛛网抓取自己想要的数据。

`

二、爬虫核心

三、爬虫的用途

1.数据分析 / 人工数据集
2.社交软件冷启动
3.舆情控制
4.竞争对手监控

四、爬虫分类

1.通用爬虫：

实例： 百度、360、google、sougou等搜索引擎‐‐‐伯乐在线
功能： 访问网页‐>抓取数据‐>数据存储‐>数据处理‐>提供检索服务
robots协议 一个约定俗成的协议，添加robots.txt文件，来说明本网站哪些内容不可以被抓取，起不到限制作用 自己写的爬虫无需遵守
网站排名(SEO)

1. 根据pagerank算法值进行排名（参考个网站流量、点击率等指标）
2. 百度竞价排名

缺点：
1.抓取的数据大多是无用的
2.不能根据用户的需求来精准获取数据

2.聚焦爬虫

功能： 根据需求，实现爬虫程序，抓取需要的数据
设计思路：
1.确定要爬取的url 如何获取Url
2.模拟浏览器通过http协议访问url，获取服务器返回的html代码 如何访问
3.解析html字符串（根据一定规则提取需要的数据） 如何解析

五、反爬手段

1.User‐Agent：

User Agent中文名为用户代理，简称 UA，它是一个特殊字符串头，使得服务器能够识别客户使用的操作系统及版，urllib库使用 7.请求对象的定制 扩展：编码的由来 User Agent中文名为用户代理，简称 UA，它是一个特殊字符串头，使得服务器能够识别客户使用的操作系统及版 本、CPU 类型、浏览器及版本、浏览器渲染引擎、浏览器语言、浏览器插件等。

2.代理IP

西次代理
快代理
什么是高匿名、匿名和透明代理？它们有什么区别？
1.使用透明代理，对方服务器可以知道你使用了代理，并且也知道你的真实IP。
2.使用匿名代理，对方服务器可以知道你使用了代理，但不知道你的真实IP。
3.使用高匿名代理，对方服务器不知道你使用了代理，更不知道你的真实IP。

3.验证码访问

打码平台 ：云打码平台

4.动态加载网页

网站返回的是js数据 并不是网页的真实数据
selenium驱动真实的浏览器发送请求

5.数据加密

分析js代码

六、.urllib库使用

1.爬取网站界面

urllib.request.urlopen() 模拟浏览器向服务器发送请求
response 服务器返回的数据 response的数据类型是HttpResponse

# -*-coding:utf-8 -*-
# @Author:到点了,心疼徐哥哥
# 奥利给干！！！
#
# 使用urllib获取百度首页源码
import urllib.request

# 1.定义一个url 就是要访问的地址
url = https://www.bilibili.com/

# 2.模拟浏览器向服务器发送请求 response响应
response = urllib.request.urlopen(url)

# 3.获取响应中的页面中的源码 content内容
# read方法 返回的是字节形式的二进制字
# 将二进制的数据转换为字符串
# 二进制到字符串 解码！ decode(编码的格式)
content = response.read().decode(utf-8)

# 打印数据
print(content)

2.一个类型，六个方法

# -*-coding:utf-8 -*-
# @Author:到点了,心疼徐哥哥
# 奥利给干！！！
import urllib.request

url = https://www.bilibili.com/

# 模拟浏览器向服务器发送请求
response = urllib.request.urlopen(url)

# 一个类型:HTTPResponse
# 六个方法:read readline readlines getcode geturl getheader

# 返回多少个字节
# content = response.read(5)
# print(content)

# 读取一行
# content=response.readline()
# print(content)

# 一行一行的读，直至读完
# content=response.readlines()
# print(content)

# 返回状态码 如果是200，就证明我们的逻辑没有错误
# print(response.getcode())

# 返回url地址
# print(response.geturl())

# 获取一个状态信息
print(response.getheaders())

3.下载网页、图片and视频

网页：

import urllib.request
# 下载网页
# url_page=https://www.bilibili.com/
# 
# url代表的是下载的路径 filename是文件的名字
# 在Python中 可以是变量的名字也可以是直接写值
# urllib.request.urlretrieve(url_page,bilibili.html)  # 后缀很重要

图片：

# 下载图片
# url_img=data:image/jpeg;base64,/9j/4AAQSkZJRgABAgAAAQABAAD//gAgbGliZmZtcGVnLTMuMi4yLTEwMTEtZzY5MTdmYzYA/9sAQwAIBAQEBAQFBQUFBQUGBgYGBgYGBgYGBgYGBwcHCAgIBwcHBgYHBwgICAgJCQkICAgICQkKCgoMDAsLDg4OEREU/8QBogAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoLAQADAQEBAQEBAQEBAAAAAAAAAQIDBAUGBwgJCgsQAAIBAwMCBAMFBQQEAAABfQECAwAEEQUSITFBBhNRYQcicRQygZGhCCNCscEVUtHwJDNicoIJChYXGBkaJSYnKCkqNDU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6g4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2drh4uPk5ebn6Onq8fLz9PX29/j5+hEAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/8AAEQgBLAEsAwEiAAIRAAMRAP/aAAwDAQACEQMRAD8A8Uo6UlFMBaKKKAClpKWgNRKKKKNQCiiigAooo6UC3FooooGGaKSigAooooAKQ0tJmmrgFFG6jigAzTqbS9qTADRRS0dACkNFJxQgA9aUUUCh7Ah606kWlFSy18Iopw5popwpMezCnAUg604UMBrComzmpn6VA/3qI7A9BnSkpaQ1e5mHNLSUtABRRRQAE0UmKWhAFFFFABRRSUWAXNFFAoADRRiigAooAooADSUppKYAaKKKACnCmilpMBaTNFFKwBmjijFFVsAUUUCk9gQ9aeKavSnCpZcBRS9qTtSikwe4LTxTAadQxoR+lRN1qVqiPWmtEDIieaOKWk71X9ehmFLRSDNACilpKKACk70pooQBmiik709AClpOlAosAtFFFIBaSiijXuAtBoopdQEoxRRTAKTFLRQAUUUtDYCUUEUdaFewBRRRRe4CZpRSGlWh7ASL0pwNMWnVJcXoLmlzSUUDY5eTTqanWnikwQ1qiPWpmHWoj1pxBkVJQc0ZqrmYClooo6AFLSUtACGkpaSgAxQeBS0YpgJRQAfWlouAgpaKKTYCc5pRRRR+IC0UUGkhvZCUUUU0IKKKKYBS0lOApMEJSU402hAIaAaCaOKa2ACKVaSlApPYB6U4UiUtJlx2FFLSClFJjYoFOBpBQKQCmoW61K3Som601sBHSGlpKZmGKM0tJRYBaWkpaYCGijtRQAUc0UUfeACiijFIAo4paKHsAUUUUvQAoPSig0wEozRRimAUUUUAFOHFNpwpaXBAabS0lPqDE60ClpAKewB3pRRQKTAkXpSikWlFJlx2FFLSUtJjY4UtIKWkAjdKjPWpGPFRN1poCKiiin2MwpaSinewCiiilpagJRRRTAKWkFKAT2NGoBRTnXbt+VlO0E7vfuOBwR0ptGzAKKXBpMUmAUUUUAFBopKLAFFFFNAFFFFFwCnLTaclIEDU2nt0plNAwopATS0AFANFKtD2BD1FKKFpallx6CilFJQKGNi0ZoopBqIaY3WnmmGmgIjzRRRimZh0oHWiigBaWkpaLgIamFqf3a8tI6B9gHQH7o46kjn2BpLW1lvJ44IlLPIyooAJJLHAGB710Nl4cUFom8wXCtGiv91SoXqZcqqEMMbTjjHzda0o0Z1NrerHGLkzP0rQBduqzSLFuPfoB6k//rrpIPA0tjPCZI4RbkKUumXzIHbjiTJymT0J4HBqRfDl7ZS5u4I1O1RF5yu0MxAGMkMpO7pvUkZ5Gau6JdeK4WnhbR7qG3tmjeXYJlWEBiDl2VgItjsTkMm3vXQsPTopc63+1v8Af0RvSpwjbmi/Xoi1qvw/03XbRUhgiGtWULPDDzGmowJlth+Xa7JyVA5I4yw5rP0LwZpIhjk+wWk0jZDnUWmXOeD5UEJjwFz3LEdeK9M8J6VYanPH9uE1o2F8iSFYo1V25RiF3GJzwVkhbypRyD2rSb4SeFNYuL6W6/tOy1COd/Mlhul8t1k+ZJ442iaMJIOoAGGDCprSpU3zct36fjccoQhVfOlay1Sv+B5rJ8NfCN1b4doIDgnfbS36lfoJZJkb/v3WBqvwos0Rjp+plm2s0ayrvEhH8IKIjfX93x1PFet6x8ItVMP/ABKdSt71E4VZlEU4A7bgTGx/4En0rlrnTde0S7Wz1zTruwaMsbfUYoVeJgeA0gG6GUD+IcMByDUxlQq6WivwKnTocjcVGfppNfI8g1bRb/RbmS3vIijI20n+E8ZGM4PzLyMgZFU69W+JfhYa7o9jdRTQm8t5jbNLnEN0rRmRI1bblCuH8tZM4ZmXI4FeX3tpNZTvDNG0bqeh9OxyOCCOQRwe1Z16LpT/ALvRnNJfdfQhpKWkqNVsyQooooAWkoooAKctNpy0MaBqaacaae9CExMUtJS02AUopKVaTBEi0tItLUsuOwtFFKKBsMUpWgUtIBhFMPWpGNRk800BEKKDR2pozCgUUop9AFFKiF2AFIoLEAda7D4ceDl8QXE8kqeZEkQMIVfkYq6CQyN1GAcEHkjOO1VSputNRXUErlPSY4dNhin8szSRYdQhKQxllA82VtpaU55UA7PqKrGe7inBmTzMoHJ25+U/0Hpiuw16202XTLiO2iS2NrJC00gXqM+VubjrkgsCOpHSsy3sTeCOOO5jSdI1IYoTHOqZj5b76ZAAJVMEkk12LCtJKL27dfky+Vwstrq5L4autehEdxYfZb+EsIpbGfAS4RAC0IWQ+VvaMjy8MH/umul0f4gS+HPErQqwW2McbCG7BzCjorBGJ+ZVKsEYkAKw545rM8NW+oW0syWUkRlcJKpjaO4i82MshguQuV2yoSN0qqQ6rk+m1rXhHTvF2mNepGLPVLeNjLD1kyM7yhByyhhjawz+dbewU6b5d9nB/D/wBqrOD5b6dfM64eJvD0MUN0sRsTMAnlIm62ZnG7EZA2hTkNtHCtyvU1m+LfE/i7UQiaYBZ26okFxetOkTgb1eMMGZWfgkjaDkE461xfhrX4tLuF0XWYW8kqVt5pR5ghfI2Mvy8D6ZAPOK7rw34NsPFiGHVtRmOWXH2aaOCV2QMo3K4ljO9GyrhN6Mv7thXNXw/LSqNa8vfePeNtzVVqbcZO6t8366s6TwNq9//ZqpeBnnT5ZTjqV4z78c5qfxxod94l0sLpV1b2t0vKvcbhH7o4CNlW6dOK0tE8NW2ixeVHNPcBcBGnZXlVR0UuqJux2LDd6k1Nq0tnp9m9zcyCKJOXc8Yrilf/MIuLrrku237tlv8mcLbfDjxE9nLFeWtlKZBCz/AGa9yvmQk4ZdyxkNz8p9gD0BrifHnw1lt7SRZLYOVY+UbrAa3diT5PnonmR27Z3RsSyo/wB75STU3jr9onVJdbksPDyvbWdrIY2lBXdKQcF2JB49uKueCviZd+Kr2903U51uZkg8yBkwgfHLR5/vL99T6gjvW2GxfPKFGorxnpGTWuuxVTB1Zxqzi6cnHWcFurb+Tfex47qWg6hpzv5ltMgXO4MpJT/gS/u3XHIkjJUj0qjXr/ivT9RPmC1hsL9CQgtr60eJmkIBOy9tnUF3XJCybCewNcFquhWH2iRJbOfRpjkhTL59oZM/6ppTuaInorMNufvYzmrr4OVN6N/Nfk1oc0qaSundeasc5RTpEMTsh6qxU8g8g46jg/UcU3rWFrMkKTmlooAKeg4plSJ0pMcVcGHFRmpW6VE1CCSsJS0UUxBSrSZpRQwJBS0gpallx6C04UlKKTH1FpCaWmmgBDTD1pxppPNNClsRZo70GjimQFLSZpRTvoBo6JpE+oTLsClX/dn5lXG/5fmZvlQck7j6cDNeo6RpumeCtImsNPkSdpIB/aVwQ6iUuATCmcNnPO44woVVGSTXCeANUjtZJ7eeGJ45wyDejMjvs3APsIfClFwykFc5Br0HVfE/hzTYEtGjWEyRo0rCM3Crv++QN+4gHvzgY4rty6lR5edu3dvp95cIvSSe25Q1e3ub/StVv3ISK6ggSNflD8XC4GCRln2kAjJGQa56wuNE0yKNNWuiZUD+TJEtx5kQY5x5kSEcMS2TnBq1JrVxqdx9lDJLaRvmJYCkwcZ+XdDI8Zb5cfLx+VZ2vCBiPs1zDdSDGYzbpbsoBOVMThh37Ejjg1u/dTnBxck9Lq916X1+Q8ROMmuS9kvxNa1s9GmMF0NRlEjK4tb23ljZ5kbIKzK0cZV1fqruN3rVoeJdV0Qh5y90Ub5L60ifdgDG6eFwrcgAMQCMY5NcGb2e1eRQWTeRvibpkdCpHT6U9dVuWwBPNDH6RMVUH6dPrWbx8E27OE+ttvmn/wAORFtHaWmtW2sXkHm2toJpGzGZH2ozsd2GBCmPn/lmDgV3fgy8g+1y/wDCRR6bai2aJgHtYmt+cgP9qkLIRhQV+6wJOcV43a3V/NO8STzM7gfMjfLNt5G5GGNy9jjPpXX2uuXFr4cggmZ1nedQzuQrvDGv3WDEg5Y5V8YwtZ1swhVjJSurReqUbs2pUZSSWzb7/wBdD6AtbzTgn+jmBR6R7QP/AB2q/iPRbHxVpE2m3TyJDMBuaJtsi46FWwQD9QR6ivM/Beqa3aoJo9RGo28IxNZyhTN5PVZrWVcHOznyXBUkFQQa7+z1lZIUKOMOFKknbww469z2FcUnF7arzK9jOhOM4S96LupK+j+ZSfS/h58ONBZp7WyjgQBHe4jhkuLtz0EjsmZGPvwB2rhbj4tfDuDVYp9N8FWKSQmSLz47e3glMb5DlRCAGBycbxkdVIrpPHHwxn8f6xptxd6w9vpdsrGeziU/aJZM8GJzmNNw+UuykqPug0+58LeJNDs1s/A2k6BpyqoHnTxrJdS4GMyTSBmYnuWzzS97o9tur/4B04GrltOPNiYe3qzvfmqSpQhfe/Km2/kZV7ceG9asP+Ek0ZVNq4FpqVvKxV7Zjysco/uvz5bnBV+hGTXH+K/DNhC8GoSpfXNrKN6NBNJLPBnOPl2sjr8uWHAzzgV6B4f0D4o3X9oQeKbLRUtru2kgMtsIDMz4PlO4iXa6Kxzh+etYOnPbrb3lhexoiJPJ5A+Y/ZZk2rc25xlkjOA20HMZw616GXVvrNB06ivODXxJXcejtfWxy472NKrJ0JKdPok3KyfS8oxvb0PLtZ8PeG2bdZ375k5UtF5JDMN3lyCR9m/r3jrnLm0ktnKtwffg/kf5gkV6R4l8P21pdXMN+Hks5F3GeBoi6ITxI8Vy4D7M/ehuI+eoySK4vU/CM9uzPZX9hqdtnKm2n/0hQegktJdtwjepCsno5rPHUOWWkIrzjp96f6HO9de5i9KKdLG8MhR1ZWHBVgVYfUHpTa5QDFSJ0qMVInSkxw3FPSozUhqM0IchKKKKZIUopKVetDBbki06mrTs1LNI7C0ZpKOlJgxeaQ0tIaECY002nmmGqQpkVLR+lJ3oIF4pycn1pppRQB0HhGO1e8huJBMgtnWQurALkMuABgZbucnGBitv4g2aGaSUNmNJcB8AEBkUglVHHJrL8CW1xe3Ey/wRW80o3EDHyn5hnksei/3etOuNdOradJYzYWfgE5A3tHwrr6kgYYfjXVTlFYblf27tebRtTt7Fp9br5mJbX81lcq8M+0qRhuSOD3B6j2NbV1rVrqtmmPJtrqElpFhkEVvdDqZkUqGhnT+NASjDlQORXPT2s0TkMphppiplWN7YKQAVPLd+azp16lNOPTs7/gZNP0LmoRwXKwzxyB1lUhuRlJkOGXjsy7W9OeKopvDbeeDTDEY+O2c8GpI3xU1anPLmas+tuo4rUvafI0ci4OCCCMe3pXWW1vHqFmrO/IKOuQcH1U98EeneuMgZt4I/Cty31SSC2VY2P+FYVH2OrDy93W533hl7exLCN3R42BVc/wCstyPmjHb5G+eP2O2uu0/UYvIjCuHACj/vnp+VeP23iO4jkjkBIKDjHHXtWva/EGe2jwqFuPyP+FReRpC0lZ3/AMj1+z12BB+9kVQOuTin3HxD8O6ahMt5ACP9of0rxXUPHGtXoKoDGp/Osh5Li4YtPIzE+pNNTkTLCQbvqey3nx/8MW7lFaaXnGUQ7R+ORXK6/wCMNCvNTk1vRmSQXroL6zlykRulVgJdp4QzxnDSDpJGD/Ea4Ty4QOcVWvpZLW1mkhO0jYR9Q2R/+vtW2CxE6WJhPXzXdGeIw0YUZa9Dq/F2uWF5pMWoQO0c1tJHA6tKuyW3uUZ0juYnUxnYVlhzt3EcA4xXnWtSRee0aeUYwS0PlyNIqo3K/fztPPzKpxnPFL/b+oqJv9LlYTKEmikZmV1HTrkZB5B4INUZJWkOW5P0A/lxXRi8WsRsrHIrrS4wknrRRRXPYYVJH0qOnxnikxxeo49KiPWpCajNCBsKKKKa0EFAooFDBEi9KcKavSnVLLjsKKM0UooY2FIaKQmkC6CGmHrTyaYapCmMooopogKcoycdc+lJToXaOQOvUcigDrfh1cCA6lvI2JF++PUoCpijA7Y3sM+9c7qiOl9Mv912x+ddF4OtfL8PeIZDlJLmCNUUdMwyrO3vny0fPpwK5y9vBczM0yBmB27wdpIHAz68d62mn7GlfTRtP5mkpL2EY9ea5H9tu1AUyP8A8C5/nToFEmWkJI/u9Mn8KZmNsrk8dG6ip9OiguZTbS3CWxfBilkz5W/+65H3Qf72OKyle25NPWaX3X7kM9wEO3Zj0zz/ADot51OAQc1uXvhi4ubdkkT/AEqHlJrYxXFvOmOhZJAVPo3I9RTLDQbfTY3m1F/l2jZF8u6RuvIDHj2yD6kVLkrPU2+pYn2ifLyxevM7KNvvKtl5bvjvW1ZWkJUFsVmtLaSE+XCsePukfe/HHFOivpYuM8VnLV6HRDlpKzafmjWa2gzgVLHBBGvOKzYdRLdTT3u2YcGlYftEti7LNEo+XFVZHLmoBM+fmNPMyAdadhe1beoO+wZrJ1PVo5Y5LdT3wT6//WrQkbzvk3qm47dznCrnux7AdzXP3sD211LE+MqxGQQyt7qykqwPYgkGroqUW5L0OfG1XZQXXVkXeiikq/uOcKKKKQBTlptKOKGCFam04mmmjqNhRRRQIKUUlAoAkXpTqYpp1KxcWOozTc0tIYUhpaKAY3FJTqaetNClsRmilpMUEvRiirOmWy3V15RON0cu0k7QGEbFefqOlVsVLF5iMpjBJXkY9cU09VfUEdl4O1m3kvmsXSKO1fS3iQHGRK1qYXc99xclt3JwTXJanbeRdSp6OQPcZ4/SlsrmWzuFnyQw6c8itLU44r+yS4TAkxubpk+vA9Ov0rVz9pSs/ii2/VMu3tYu1rxX4GIh2vzT3w1NdSHP1ozUIzHQ3VzCfklkX2DsB/OrKXTy/fZie+4kn9areVvGQRUySySBFkG5kwokJ52AYVD6hex644qZRRVOpKL3foTB2B4p/mtilt4SXU4zWpL4Z1COKOUxIVmGVwenGefTNRoa88ktWZ0EvNXIpExyajGkyuG2DawzlW46dR9RUL29xEdrKRRZFRq9Cea5jHA5qFrrnrTl06d13H5c+tQTWskbc0A5yewTySvBIUbGByScdTjr71ltnPNdBZ6S9/pWoLGpMqQNcKB1Ig+dh+Iz+Nc8etVF2RjXbctewlLRRTuiBKKWjFF0AlLRigUrggpDS0hoBhRRRTAKKKB1oewD1p1NWl70hpi0tIKWkWkFFFFAxDTTTjTTTRMhlFFFLqS9QHFO8xz3ptLT+Yhd7E5JJqzHcyJEoVjw+fbFVRTlbHWmmUnYkuHSR96qEz1A6f8A1qjxS5o61V7kvRiJu7VZg5zn0qFVp+4rSaBOzR0Hh/TBe25k5LZ4x2x3rstEgleNILpFbYQVbru+o7GqPw40/VNW0tjpmiXd6mdhnRFEYYYyu52UFvXHNdDL4e8UWADTaPfRD1EXmYI9fKL9ajkqXvySt6M1ly8vxw1/vK5j+JvDkFvKLm0+64zInoem4f1rH+xoy5KjIrpp7hm2pPHInBUhkYfhhsVSksbV4mALJIGIDfw+q5/lTt8vIzT5fM565jC5xWfLtZiDir+otLDIyOuCD+fuKzXYls1JvR1Oi+H8CS6hOuAV8gqw9QXTI/EZrhNWsH03VL2ybrb3EsPPojkA/iMV3vw4kWO/nJ6lAB+dc78UrH7J4uuZAuFuo47kH1JBRv8Ax5TTjuRiF+8fyOcxRSiimZCYowKXFBosAlAopRQNCEUlPJFMoeoMMUlLRRr2sIQ0DrRRTActO70i0tIYopaSlzSLuFFFFAxKbTqacUImQyiiihClsFLSUtMnQKVQSeAT3NLDGZZFQbiWIACgszEnAVQOSxPQVr22iPD8tx+6PRoxzLnuJG/hPqo5HQ04QlN6BtuO0Xw3pd/bpLdazb28riSSO0UAvIsQUsjzO6pBI4LCMyKUyOpqL/hGL2XWTplpJa3JacQRXC3EaWkm4/KwuJjHGE5xvbaKvtDaswPkxIQFHyoF4UAdFA5wOT3613XhH9nrxL4ls4dQuJ4tFtZ1EkJm3STyIwyriFCCAw5G9l4rX2UYrV2+dribNDwf+ytZXNhDd+INeaSSUBvI0gwvCgP8P2uVZBI3qUjC+hNdonwJ+G1vo6aR/Y8UqEZe5lbOou4/j+1Lhhz2XC9sYqPw58Gb7wmqyaV401yC4+QyDy7eS0mKjGHtpt4ZcE4Uvx1HNdfsl+V5DmQAAvt2q2Ovyj7uTzjmleKfuy09BPXc888NaVP8NdWn0QkSabO5ktt5jXD42gxHCFtyhA4OSJMgnJGe1jiS6T5XkioscA9M9evcGpNZ0iz1uya3uI0J+9G+xXeGUD5Zow4K7l+nI4rzq+8WeK/BWqtp17cLdiMzSPJJD9mWaI4aOS3eTzFfI3gojHB4IzVxlouhLg2d9cWt6Bsliiuo/wC+8SyKQOzxtnr6rn6VRutC8NXoUXGi6a0jdfLh8sgD0aIJ83oCcetVvDPjOLxFbjyr/wAt8YMUkKsyH6rMmVPZgMetav8AZ0h3SxXixP8A8tC8ONx9yGYH0709PtW9bEuMov3br5nEfFnwL4Ss/B93qtlYGyvIHg8vbNMdweVUZHSR3QjaSRt5GOtePste4fFPRtZu/CV/M11azW8ASeVUYlnCNtGxGj+TG7P3q8RkIXPtWGJSjNW7HblrcqUm3tL9DY8GyCK8PvitPxn4IuPGclibG6tIL1Q8MMd3KlvFc7iGEQmdgqv94ru+XrkisTwq5N8K6nXZbq10aS8tpGhuLLF1DIpwVaMjJzg/wlqiNlJXFil7z9DzvxL4C8XeDpAuuaRd2Stwk5USWsn+5cwl4W+gfPtVSPQNXn0+PUYrK4ltpLk2iSRxvJmYBTswoPXcAv8AebgcivpT4T+IpfGfhiUzxwzy2ki29xHKkbRXcbRhlZ48vErj5kK5PTnBzXQWWgaBafu7PTYNP25IS3jWFYmLFiVjUeWuWJO5VB5rRwRgpJrax4L4P/Zz8Ra1DFd6/dp4egfn7PJE0+osvqYAyrFnt5rg/wCzVT4tfBC++HPk3tlczatpUow900Iie1l7RzqhZRvH+rYcHkcV9BvphhnLNIGRcuVbiRgvPy4yGGcZIPHcVLFC93bzrfW0VxBdgrLBKqyxlG42SIdykEdVI4oUdHdW7BKSTSWvc+OiuKK9h+L37Olxp/2nX/B0b3FplpbnSAC1xajks9qTzLCOvl/6xB03CvIHRkJUggjOQRgg+hFS00MYaSlpKEAUmaKKACiiihgOWl70i07rSHa4opcCm80uaTLVrC0UlFAAaYetPphpomQlJRRSB7BS0VLZWs19dQ20C75ZpEijX1d2CqPzIp7knbeBtCi8PeBNT8dXUQ+2SXP9j+H/ADOizzJi5vlB4L28O8QN/DId3UCsaCdWlxkuSevUk/1r0j446GfDXwy8IaJbKot9MuFhuWUffuHtmLuf9+TzDnrzXkiXLQsrKSMcjFbRl7OysJ6nonwm+H9z4u8UR3F5Cf7KsCtxcs4+Wc5Pl24z97ew+f0UH2r6CilUIFxgDAAHQY9u1cJ+z3rWjar8PraCGSNbyG5uo7tCFV95YOpOO2wrtPeu1eN4m5GB2PY0qsuaVtbdP8xLQtHpTCAw6dqZFNuXH5VDJqCwS7X4Of0/GpUX0CUl1HshUnbz6r3/AArG8V+ErLxVaeTJNJayjcFnjAYFWGGinib5JYm6sjcbgDjittZYrhN8bA/TvUMuQNyqSe+PT3qk3cLnjN7pWp+GtVvJFd4zDK4KfZmtDiPGZIDGv2d0CsjN5ZAUttxXaeC/HyaikVtfOpkwEWTtKvYY6bvbv2rT8b6Jbatpc8hSUTxQTmMwJuklyhHkyKATJGeDtI6gV5VcmLSpF+zTGJg6I1s7EXKSeUkxZVDSDYFdSWEjBWO3NXfS/TqFuZnrmueFINctJ4oLh40nilR4d3yPvjIHB9yD9a+dtTspIbyaHHzJIyEe4OD+te4+A/GLahYQpc7mnjPlSY74+6+P9oenfNeaeONCNj4s1dUjOz7bM6rzwkreYh5HTawrPFpuMH8jXLpcs6sNno7dNDE8OQyWlyjuOCfxFd5Bpjavplxboqs0sEqBW/2oyP61zGl6ZdXEgWKIucjnHA/GvRPCOjTWyBpeG4/CsIayNcTZxvfU5f4DeJpPCGrahFqh8u2vDbxSsFXbAUwsMsnl/OOrq25AoHzF817i6wTFTkbsZVx6HnIPQjHPpivD/GelN4M8QmK30wvo+qSC5lumNw+yViXliEvCxyK6g248xgUbaF3ZFa198YYtC8AXukPdzPqtxZmPSJBA/wAsNw/lMJZdoiWSBC5jReQoAPIrfl5rW32ZzO1v0Oe+LnxKuNX8afa9Ju54IdF32Wmz28hTdKG/0i7G04+ZvkTqCoBNVdJ/aE+IGlzq8t3aainG9Lu0hG8e72wgfcf7xJrA8M+E9f8AHeoNY6JZSXLW8eZXLrBbwKM8zTyfIpds4BO5j0HFZraRqbX13ZR6TqNxcWcz29zHbwS3BhljYqyMYUcZyPXntWkpQWnu2WglTm1flf3Hu3gb9onwp4lkjtNZRfD94xCpJJJ5lhK3tOQrQknoJRt/26yPjP8AAC08RC48S+E0SO+dfOubCLb5F9nlp7crwsrA7mUZWTquD18mufDWsafB9ovtH1jToeP3t3ZTwx8/7Ukar+tdJ4C+L/ifwMiW0N4uo6arA/YLzcyovcW8oPmQ57BSY8/wVPs09Y29L6MfvRfvJo85uYGt5pYmBDRuyMCCCGU4IIPIOexqGvSfizpPhzxlYy+PfCETR7pUTxFpuAJrG4lzsu9ifKYpyCGkTClxkgEmvNj1rKceV2He4hooooQCE0q0h5oBof6APWlzikBpSaXQYoNLTQaUUmUncU0maDSUA2FITSmm0xSYlFFLSsDfYK674HaKNZ+JOjKy7o7NpNQl9MWq7kz9Zdgrka9U/Zg00Pq/iDUj/wAsLKG1XjoZ5d5wfpFV0lea+/7ib2udn8ebWS/+Hd+6ru+yXNpdk9wFl8tjj02yHJrwGSvpbxXYjWdA1jTzwLqyuYFyQAXMZ2Hn/axivmpwQSp6jg+xHBrSqrWJiaXgrxhqfhPU/OtbiSFJcJMFYgH0bjuP5V7V4Q+NNwRHDqX7xDjEjYxj/a6n8a+f2iz0rtfAtre6zpLtGu42bLFIc9QwyhP4Aj8Klcz218im4xXvfefQumeJdD1YBoLhY2P8O7Kfhn+hpPFWnTX+kyyWb/vkXKuhJyvccDcOO+K8Zg1bVdJkCLlAhz9CPSul8O/FS+tMec7Yz8xeQs31Ax+lNWT7Ps9mRKPMu68tzY8OeK7i0uDDJu2x/I27rkHkn159uK6+31BLlA6HqMmudstd8I+LGDTwJFcMf+Pq3AhkLDvIuMNz/fGTXJ/GjRPidBp5k0Kd73w6sWbn+zS636gcs15GD5rRe8BZAPvgU/8AF8mTbW0X8jrfEvxY8AeF5zFeazE9xnDwWSNeSRHODv8AJyic9VZgfavPNcvfCniG/Gq+H9Sk2XE15dTQyDypY5gNjwvbyt5AhnMglEg2ykN1+Tjy5nOef1q3oepPpWoxXaKkgXcrxuMrIjAqykHjoeMgjPUGmn0RVu56n4ZlvdHu9kqtE+QjrnkZ5wcE+vr0wehrrNV8K6f4gs01KNC0yKsNyef3kf8ABJ16pnaT6Y9K4Hwr4utNQtY1vPJRIi6TT5Jlg+UvbsyjDOFC+U5RH3MQBgV6b4Vv/LiMTDMcse4KeM4HzLg+qnIpuKnBr7rk88qdVTXo/NHOWGjz+FZzcJGtxF3U4yB9D3+ldDYzfbY/NiXaG5H/ANek1a3xMEYbo3G6NscMuf8A0IdCKlsVgtVAiwo/u9qw5bO21uh0c14p2u3rfyMTx+u/QJbW5dEM01skAc/fl85Cqp6sRn+tcf4G+GGteLki03WbloLLSLxZZtlxFM0ykMiwQrFnyX2qC0spLNGygL8or0HVdGbVta0+/d8xWFvdGKBzmA3Uu0RzMnQtCFJViDjPHNavhiCw0jS9sBT97I88jIiRhpJDk7FRQAvpx70+a2iWolFy97Sye1iGytNM8M266Po2mRafZoPMuXt0CL0x87nLyzSdNzktjJp1hqFxA3k6dbQ2kMsjFkjRUXcxy0r7QNznqzMSxPU1NfXi3AMaD5c5IHc+/qaqw2UsbGQzOqDnbnCjPoKLLqbKa5NElb1d33LPiO0vr+2uLVNT08QTWrpPb3lu9xHNuBBVz5gVEbpnaxB5xXyrqKXNhqF1ZzqYpLeeWF4yfutG5Uj3xjr3r2X4geJtRv8AUX0/SpJRbW7Dz5od2ZZe0Snvt7471meOvAHhfxXos2tadb3mneJmtjeS2iFprW/+zxDzcRbd0UjqpZWQ7WfqvOaFdd/0FWpe0guVpyjdy2XT8Thvh/4w/wCEX1xJblPtGnXaNY6taNnZc2M/yyqR/eQHzIj2dRVb4ieD5PBniW5sFYzWcm250656i7spxvgmGOOVOGx0YEVkw8MM11+o6i3jf4fR2k2JNV8Kr5lq+C0t1o0jKkkOfvFrGUo4z/yxY/3aq3PB+Wxz7P1OI70UtJUadxid6X0ooB5oYDh1oNA60HmjoMUU6milzSaHFpICeKSikoQNgaSlpKBCUtJS0BbXQK9r/ZjgVfDmuzlcebqEMW7+8IrfOP8Ax+vFB1r3r9nu2a0+HSSsMfatRvJV9wuyLP5oRV0PifoKWiZ196Q7cjA9D3A/lXz38RtCHh3xlqtki7IWl+1W4/6Y3A8xRz/dJK/hX0HdEg8JnPUZzj3APH1ryr9oHRpDLpWtAA8PYzMBz3lhL/8AkRR9K0qK8PTUiL9482rpvhP4ot/DvifZd4+yX8JtZc8hZM7oXweOH+XPYNXME1ExZWBBwQcg1ClyyTXQpq6a7nr/AIiaKV5Jowg34AyFU/XC8ZHr3rPtNBuJ8TK6AZ7np9azfCb3/iS2icMWbIjckknf3OPXHNdhZaMtgv8ApDmT+5sYYbn7209fpxWns/aataGE6ro+6pa+hTtobiyUTQoxKkDhCiSMDzgkdPQ4ruPCfih4Io2kuC3AxGCJGUd87SePy45xWPaizVflG+MYH3gzH3xyqnrVm2t/skgmso0AY/MGTk++1ccf7VX7JJW6djNVZN/qN+JXwR0LxxaTav4chh0zW8NI0UYWOy1I4ztdB8kM7fwyrhWP3x3rwm/sbzSL2exvreW1ubeRopoJVKSRuvVWU9DX054c1O73gTQmEHgN2YdtoH/1q5r9oT4bQeJtCbxNp8apqemxZuSowbyzHXfgcyQdVY/wZB6CsmuSdu50QnzxueHadfy2rkxOUJxzkjlTuX8Qen5Hg17j4J1mbV9C07UyySSSIBKyEbftEXyuPl6bx94cYJIrwEM8b7SCCODnrXafCzxv/YGoixvp/L067ba7tylpK33ZvaMtgTeg+fsaqMuZWCUdbnuqpC8aQ3as0MuJIpE+9ESPvKens6nrWVexS6fqD28nblH7SIeVcfUfl0rS0u5SSLyZMHptOcjnrj1HcEetLr1gZLWObG54MIdvOYWOVP8AwFv0as68bx5luiqEnGXI9U9ipDKGQg9CCD+IpmmROll5KZYhm/n/ACpnkyugjiVneT5VVQSxz6AV0GheHpbGEtOVDuqfJ97Zgc5PTP0rNX3NoyjFO7+XoZ9rbNGMyDmoNVubZLeT7TOLaBFZ5pCcYQDnn+Vbd7aGIFsceo5FedeObq91fURaQYazjI3hDzPID/Hj+BT27nmqgrlSmraP5kTa9HKGk0rR7FLRSVtBch3uLhj1k2Ky/f6ksTxTLKPUdLvD4p8SXcMDRD/RbG0HlbsD5VbDfdHpmoZrs+H7cSSxxqdv32GWUf3Y/T8Oa4rxX4tutduPs4LiJeCWPJ9gBwBVS91a6vt0Jotynokl17tepB8VdY0DxPdx6tY6XDpV75rQ3n2cKsN6pG5J2VQFEykFXYD5wRnkVzmhaq+j6lBdqvmKu5Joc4E8EqmOaFvaWJmQ/XNWfFVtdaeljDLbXFus8Qu0eWN0WeN+EeIsMOnB+YcVkbu9KDcUr7ixLhOrLkSS0WndLUdr+mLpWpzQROZYG2zWkpGPNtpl3xP9dhAf0cEdqpVp3c32/SYkIzLYlyj/AMTW8hBZDz0ik+ZO/wC8as2lNKL02epCEFAxmloHWpYCjrTjSLSmgYhNApKKAFzSUUUBuFFFFAISlpKWgNriqcMK+ivhFGYPhp4eXYFLW8spA4z5lxK2fxHJNfOo619KeBIPs3gjw9CCU2aVZ845BeJWP6tmtcP9r5E1PhNK4lwg2tx3LHJA/CuX+JWirrXhHVLVEMkoi+0RDkZmt/3ilRzliAy4HYmunRgpYEsAOAzgAMfYVmalcb96RJ9wHcxGRk9u4zWtr3RB86duKbtrX8b6MNA8UahZopWEyefb5/54zfOv/fJJT6rWaq5UN2zj8ax5dWi76HZ/BHUIxrFzo8u3N3GZbYk4xLEPnH1MeT/wE16RPpcsKkvNJs5Iw+19p9TjO3/awTXh+iapNomr2OpQcvaXEc6rnG/Y3KEjnDrlT7Gvoq2MOuabZ6jaBJre8hSePZg4Ei57jnHQ4xgitqU7QSfQxr0k5cy6/mYscUVuY2eQkk/IA3yN7FBwcdAT+dbNhbNcInlHA/3RtyR2x8wx9cU+y0ALNykeBjHHI9OufxrasdM2kbTwOw6fSidRdCKdF31ILHTpYlAaQ/72MZ9+/StiGLzLV4JwJo3Ro5Fb7ro42sCPRlODTobWMLjb+VOfEQ46DOcispS5mbxgoI+Yfib4VXw34t1bTYUdBbTFoA+3c9vIokjPykg/KcDucc81N8P/AIV+KvHEiSwRfYLDdhr26BVGHfyY+HmP0wmerV1n7S1tbw+IdG1KMYku7KWCb3NrINjZHfbLj8Kt/Bj4krdrBoGpXGyaJQljMT/ro1/5YE9N6D7hP3l46ir5U3frbYnmklpt3O90fw4fDWi6fYrdzX5sokgknmC+Y6D7h+Xoqj92ASSAACTW1YR/a4SpTcpUoSem09R9RToEhuFWSLr0YddwPYjvmrlqYo0ESx+Wo7fXqTnnNRKTs9CoR6/cV7HSl02MvCElmYEb2zwP7q4B257nvUsd3cghZYg57iM52e3+2fYU5w8Em5D8j/8Ajp/wNMutUt7CLdIAXxwoGM+5PYVMY9kmW33ZMR9pUg48sjBQjlsjowI4+lYWoeCLGBJp9PtoklYszJ1J9kLd/atGw1afUmk+zxQJtHMjFiB9QMZ/Osjxr4yfw/ZqY4xLNNuSOVG8yNH56t90dDgdeKqFObnbRW38iZTUV1d/xPGvHHiG7vNWntTvK27tGVYbdrKcEbcDbjoRjrUnw3+HM/i+8nuZ5Bb20IJUsM/aJ/4YlBI+UdZG7cDqaofFi9vIja6zHGvm3peG8mwMeeg3JIQON0sWRk9TETyag+DfjXU9J1yG0llea0u5isqu5/cu3ImQ9umGXoRSkrVbSd9dP0LlVSo+4rO13+p6P4j0Wx8T6QfDHiSIW00Azp19Gnz2kgGF28f6ojiSPoR05Arw/wAQaHf+HdTudOvIyklvIUJ6q4/hkRujI64ZGHBBr6V16wTWdO+0hT9stlLMAMGVRzkf3mA/MV5t8RfDo8V6W19HEV1KwjISPGDc2y5ZoSpIwycvEQCc5XvVyipLszGnNxlbdHk8UuxvUHII9QeoqCRcORVqSDjcmSP1H1qGVNwDDr0rKaaXoax1ZDQBTthr0b4U/Az/AIS2xj1zX7mSx0yQ5t4I8LPdqDguWOfLjJ+7gFm6jApQhKo7JBP3PiPOlFDCvoW4+Dfwma1NoumbDjaLhLibzwf728yHJ+ox7V478Sfh/eeAta+zF2ubG4BksbvGPNTvG+OBLH0Ydxhh1q6mHqU48z2JjWhN2W5zVFFFZlWCiiigYcUUUUAJRRSijUG1a9hRX05oivb6DpMIUHbYWid/4YEzgda+ZrYAyoCAQWAOfc19OIPLt440+VVihVcdhsA/lW2G2l8iamwju+d7h2wcA8bR/Un8BTJwxTPDKfX73P0HJ9KJmOzfk5woxk7ehPAzgHPpTCzO+wn5UHQd+QOf/rYrQzPNPjX4fkltLbV1iO+1kaC4b+LyZWzGXx/ck+X0G6vOFY17f40s4rnTr22m3SRy2t0r7iMnYhKHp1UgEH1FeHjtWddWlfui4bEqmva/2cfFDaho154euZAZNPYXFpuOW+yzH50Ge0cvQdg9eJoeRXXfBnUrrT/iLoPkMFF1c/Y5gRkPDMpDKfyBB7ECimEz6OS2Xj5RirEcZUY4/rTU+UDHc0/eefak2CJUwBj+dZ/iLUorC0Z2dQ38IP8AE3YAVbZ2UA/zryT46eMtc0WR/sksalZBFGzJuMW5Mlk+bG/0LA/SnTjdtvZaiqv3UlvLQ5L48eKBrPiCxskKn+z7djLg52y3BVthPqqBc/WuX8I2t1q3iPTLC3uHtZJ7qFRcpHJKbb5h++KRAsQnXsM4yRWXJcT3l00k8jyPI5aR2JLuzHJJJySSa+hv2e9A0VfCsd+lhbJeLvtzdLGoneMnfh5MZbDdPbjtQpOUnLZLoFuSMY7neaU0f9nQRwurtHGqeYAqtKygAuwXgFj8xHTNPuMQATJ/wIHn60xYY7fzHiUITk8cDk88D1rNur+6E7Rh8Lx09+vWhRu79B36FyTXrVcow+Qfe56/h1FY3irUo52thbkl3GNg5OAT6epIxST/ACzTj/aI/ICk8KwRX/iG6edd/wBkAWAfwrjjOO5759avljBOVtkTzN/ebGj6fLp+kokhEckhMkwJ/wDHc+w61598S9UgN1eWcF/DJD+5uRCgOY5ydskec4DY+fjJwccV2vxDvbiw8PahPA2144l2H03uEJ/AHI968YRft2s28c5Lh5Y9/Jy245Yk9cmin8PN1kJ6ybe0RNSjh17TbjSbl8CePEHQiG4TmGUn2f5W5+4zCuC0G4m0rVMPmKWGXaynqrxthlP0IxXc7FW4IAHyOQPz7+tcl8QIo7TxO8sKhGmgt7iTHRpXDB2x/t7dzerEmprrRS6p2Li9bdGrnuXw+8axa5ZQRTzx/aI8bXyoPT7kidx6EDFP8X6EYJzqFurbXOWUciN+pP0P8PUV494V1G7t5YZYpCj5Xlcj+te5eDLybxB4YVr/AGykLIuQMZCngHJNEXzx/P8AzInBQat8r/keEfEPQToOqi9gUi1vmdiuPlin6vH9G++v4jtWAIo5slOPVf8ACvY/iToOmz6bqts8RKRwSXEfPzRyRoXVlOOOePpxXisLMp4NRbVxeppTltNaNWZ1XgP4f6R4oi1C41fVxpEFuFjiYBC0krAn+P8AhUdhyc12mqeP4tGtLPR7CVXgsrWC2R4+Efyowu8Y7NjP415tYTyvbSRljtyGwD36Z/KrO4taREnJG5fwDECqwUpUZTjo3bf5nTm/1bEYahUp03Tu9Vp1jqtN1daXOmbx5qc8yssrKA3IzXUeTpvxH8Kz6VqciiQjdaznG+2uADskXvjPDj+JCRXl8cjIeK6n4b3Mz65CWYsFZcKc7evpXSqntIyjLW6PM9i4zjKMrNM4a/8ACPiPT7zULSbS7wyadIY7sxwySxwkDO4uildrKQytnlSDWWRivqDxjqU2j2omtY4A8yqZS8YbzcKF+fpn5QBz2FeIfGPTbC01ywvLW2itW1Gy+0XEcC7IfNWVk3InO3cAN3OCea4px5TvlRtRVS/qjjqM0Ud6kmwUUUhNAj//2Q==

# urllib.request.urlretrieve(url=url_img,filename=xzq.jpg)

视频：

# 下载视频
url_video=https://vd2.bdstatic.com/mda-mj01i9fur5bzkwj7/sc/cae_h264/1633050469983308417/mda-mj01i9fur5bzkwj7.mp4?v_from_s=hkapp-haokan-hnb&auth_key=1633070432-0-0-e7d194ed5654813f4560e2184fbff9e0&bcevod_channel=searchbox_feed&pd=1&pt=3&abtest=
urllib.request.urlretrieve(url_video,阅兵.mp4)

七.请求对象的定制

UA介绍：User Agent中文名为用户代理，简称 UA，它是一个特殊字符串头，使得服务器能够识别客户使用的操作系统 及版本、CPU 类型、浏览器及版本。浏览器内核、浏览器渲染引擎、浏览器语言、浏览器插件等

编码的由来：

# -*-coding:utf-8 -*-
# @Author:到点了,心疼徐哥哥
# 奥利给干！！！
import urllib.request
import urllib.parse

url = https://www.baidu.com/s?wd=
# 请求对象的定制为了解决反爬的第一种手段
headers = 
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36

# 将周杰伦三个字变为unicode编码的格式
# 我们需要依赖于urllib.parse
name = urllib.parse.quote(薛之谦)

url = url + name
# print(url)

# 请求对象的定制
request = urllib.request.Request(url=url,headers=headers)

# 模拟浏览器向服务器发送请求
response = urllib.request.urlopen(request)

# 获取响应的内容
content = response.read().decode(utf-8)

# 打印数据
print(content)

2.get请求方式：urllib.parse.urlencode（）

# -*-coding:utf-8 -*-
# @Author:到点了,心疼徐哥哥
# 奥利给干！！！
# urlencode 应用场景:多个参数的时候
import urllib.request
import urllib.parse
base_url=https://www.baidu.com/s?

data = 
    wd:周杰伦,
    sex:男
 

new_data = urllib.parse.urlencode(data)

# 请求资源路径
url = base_url +new_data

# print(new_data)
headers = 
    User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36

# 请求对象的定制
request = urllib.request.Request(url=url,headers=headers)

# 模拟浏览器向服务器发送请求
response = urllib.request.urlopen(request)

# 获取网页源码的数据
content = response.read().decode(utf-8)
print(content)

3.post请求方式

# -*-coding:utf-8 -*-
# @Author:到点了,心疼徐哥哥
# 奥利给干！！！
import urllib.request
import urllib.parse

url = https://fanyi.baidu.com/sug
headers = 
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36

data = 
    kw:spider


# post请求的参数 必须进行编码
data = urllib.parse.urlencode(data).encode(utf8)

# post请求的参数，不会拼接在url的后立案，需要放在请求对象定制的参数中
request = urllib.request.Request(url=url,data=data,headers=headers)

# 模拟服务器发送求求
response=urllib.request.urlopen(request)

# 获取响应的数据
content = response.read().decode(utf8)
# print(content)

# post请求的参数必须编码：data = urllib.parse.urlencode(data)
# 编码之后 必须调用encode方法:data = urllib.parse.urlencode(data).encode(utf8)
# 参数是放在请求对象定制的方法中:request = urllib.request.Request(url=url,data=data,headers=headers)

# 字符串———》json对象
import json

obj = json.loads(content)
print(obj)

总结：post和get区别