python基础===requests学习笔记

Posted

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了python基础===requests学习笔记相关的知识，希望对你有一定的参考价值。

 技术分享

Requests is the only Non-GMO HTTP library for Python, safe for human consumption.

Note

The use of Python 3 is highly preferred over Python 2. Consider upgrading your applications and infrastructure if you find yourself still using Python 2 in production today. If you are using Python 3, congratulations — you are indeed a person of excellent taste. ———

 

>>> import requests

>>> r = requests.get(‘http://www.baidu.com‘)

 

 Now, we have a response object called r. We can get all the information we need from this object.

For example, this is how you make an HTTP POST request:

>>> r = requests.post(‘http://xxxxx.com/post‘, data = {‘key‘:‘value‘})

 What about the other HTTP request types: PUT, DELETE, HEAD and OPTIONS? These are all just as simple:

>>> r = requests.put(‘http://xxxxxx.org/put‘, data = {‘key‘:‘value‘})
>>> r = requests.delete(‘http://xxxxxxxx.org/delete‘)
>>> r = requests.head(‘http://xxxxxxx.org/get‘)
>>> r = requests.options(‘http://xxxxxxxxxx.org/get‘)

 

  •  Passing Parameters In URLs

We often want to send some sort of data in the URL‘s query string.

As an example, if you wanted to pass key1=value1 and key2=value2 to httpbin.org/get, you would use the following code:

 

>>> payload = {‘key1‘: ‘value1‘, ‘key2‘: ‘value2‘}
>>> r = requests.get(‘http://xxxxx.org/get‘, params=payload)

 

 You can see that the URL has been correctly encoded；

>>> print(r.url)
http://httpbin.org/get?key2=value2&key1=value1

 You can also pass a list of items as a value:

>>> payload = {‘key1‘: ‘value1‘, ‘key2‘: [‘value2‘, ‘value3‘]}

>>> r = requests.get(‘http://xxxx.org/get‘, params=payload)
>>> print(r.url)
http://xxxx.org/get?key1=value1&key2=value2&key2=value3

 

  • Response Content

>>> import requests

>>> r = requests.get("http://www.baidu.com")
>>> r.text
‘<!DOCTYPE html>\\r\\n<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge>

When you make a request, Requests makes educated guesses about the encoding of the response based on the HTTP headers. The text encoding guessed by Requests is used when you access r.text. You can find out what encoding Requests is using, and change it, using the r.encoding property:

>>> r.encoding
‘ISO-8859-1‘
>>> r.encoding = ‘utf-8‘
>>> r.text
‘<!DOCTYPE html>\\r\\n<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=alway

 If you change the encoding, Requests will use the new value of r.encoding whenever you call r.text.

If you have created your own encoding and registered it with the codecs module, you can simply use the codec name as the value of r.encoding and Requests will handle the decoding for you.

 

  • Binary Response Content

>>> r.content
b‘<!DOCTYPE html>\\r\\n<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always

 You can also access the response body as bytes, for non-text requests.

The gzip and deflate transfer-encodings are automatically decoded for you.

For example, to create an image from binary data returned by a request, you can use the following code:

>>> from PIL import Image
>>> from io import BytesIO

>>> i = Image.open(BytesIO(r.content))

 

  • JSON Response Content

 

>>> r = requests.get(‘https://api.github.com/events‘)
>>> r.json()
[{‘id‘: ‘6736180556‘, ‘type‘: ‘PushEvent‘, ‘actor‘: {‘id‘: 14111893, ‘login‘: ‘rvitorgomes‘, ‘display_login‘: ‘rvitorgomes‘, ‘gravatar_id‘: ‘‘, ‘url‘: ‘https://api.github.com/users/rvitorgomes‘, ‘avatar_url‘: ‘https://avatars.g

 

 In case the JSON decoding fails, r.json() raises an exception. For example, if the response gets a 204 (No Content), or if the response contains invalid JSON, attempting r.json() raises ValueError: No JSON object could be decoded.

 

 It should be noted that the success of the call to r.json() does not indicate the success of the response. Some servers may return a JSON object in a failed response (e.g. error details with HTTP 500). Such JSON will be decoded and returned. To check that a request is successful, use r.raise_for_status() or check r.status_code is what you expect.

  •  Raw Response Content

In the rare case that you‘d like to get the raw socket response from the server, you can access r.raw. If you want to do this, make sure you set stream=True in your initial request. Once you do, you can do this:

>>> r = requests.get(‘https://api.github.com/events‘, stream=True)

>>> r.raw
<requests.packages.urllib3.response.HTTPResponse object at 0x101194810>

>>> r.raw.read(10)
‘\\x1f\\x8b\\x08\\x00\\x00\\x00\\x00\\x00\\x00\\x03‘

 

  •  Custom Headers

>>> url = ‘https://api.github.com/some/endpoint‘
>>> headers = {‘user-agent‘: ‘my-app/0.0.1‘}

>>> r = requests.get(url, headers=headers)

 

Note: Custom headers are given less precedence than more specific sources of information. For instance:

  1. Authorization headers set with headers= will be overridden if credentials are specified in .netrc, which in turn will be overridden by the auth= parameter.
  2. Authorization headers will be removed if you get redirected off-host.
  3. Proxy-Authorization headers will be overridden by proxy credentials provided in the URL.
  4. Content-Length headers will be overridden when we can determine the length of the content.

Furthermore, Requests does not change its behavior at all based on which custom headers are specified. The headers are simply passed on into the final request.

Note: All header values must be a string, bytestring, or unicode. While permitted, it‘s advised to avoid passing unicode header values.

 

  •  More complicated POST requests

 Typically, you want to send some form-encoded data — much like an HTML form. To do this, simply pass a dictionary to the data argument. Your dictionary of data will automatically be form-encoded when the request is made:

>>> payload = {‘key1‘: ‘value1‘, ‘key2‘: ‘value2‘}

>>> r = requests.post("http://httpbin.org/post", data=payload)
>>> print(r.text)
{
  ...
  "form": {
    "key2": "value2",
    "key1": "value1"
  },
  ...
}

 You can also pass a list of tuples to the data argument. This is particularly useful when the form has multiple elements that use the same key:

>>> payload = ((‘key1‘, ‘value1‘), (‘key1‘, ‘value2‘))
>>> r = requests.post(‘http://httpbin.org/post‘, data=payload)
>>> print(r.text)
{
  ...
  "form": {
    "key1": [
      "value1",
      "value2"
    ]
  },
  ...
}

 

There are times that you may want to send data that is not form-encoded. If you pass in a string instead of a dict, that data will be posted directly.

For example, the GitHub API v3 accepts JSON-Encoded POST/PATCH data:

 

>>> import json
>>>import requests

>>> url = ‘https://api.github.com/some/endpoint‘
>>> payload = {‘some‘: ‘data‘}

>>> r = requests.post(url, data=json.dumps(payload))

 Instead of encoding the dict yourself, you can also pass it directly using the json parameter (added in version 2.4.2) and it will be encoded automatically:

>>> url = ‘https://api.github.com/some/endpoint‘
>>> payload = {‘some‘: ‘data‘}

>>> r = requests.post(url, json=payload)
>>> print(r.text)
{
  "args": {}, 
  "data": "{\\"key1\\": \\"value1\\", \\"key2\\": \\"value2\\"}", 
  "files": {}, 
  "form": {},

  •  POST a Multipart-Encoded File(多重编码文件)

Requests makes it simple to upload Multipart-encoded files:

>>> url = ‘http://httpbin.org/post‘
>>> files = {‘file‘: open(‘report.xls‘, ‘rb‘)}

>>> r = requests.post(url, files=files)
>>> r.text
{
  ...
  "files": {
    "file": "<censored...binary...data>"
  },
  ...
}

 You can set the filename, content_type and headers explicitly:

>>> url = ‘http://httpbin.org/post‘
>>> files = {‘file‘: (‘report.xls‘, open(‘report.xls‘, ‘rb‘), ‘application/vnd.ms-excel‘, {‘Expires‘: ‘0‘})}

>>> r = requests.post(url, files=files)
>>> r.text
{
  ...
  "files": {
    "file": "<censored...binary...data>"
  },
  ...
}

  •  Response Status Codes

 *详见之前博客，http响应码

 

>>> r = requests.get(‘http://www.baidu.com‘)
>>> r.status_code
200

 Requests also comes with a built-in status code lookup object for easy reference:

>>> r.status_code == requests.codes.ok
True

 If we made a bad request (a 4XX client error or 5XX server error response), we can raise it with Response.raise_for_status():

>>> bad_r = requests.get(‘http://httpbin.org/status/404‘)
>>> bad_r.status_code
404

>>> bad_r.raise_for_status()
Traceback (most recent call last):
  File "requests/models.py", line 832, in raise_for_status
    raise http_error
requests.exceptions.HTTPError: 404 Client Error

 

 But, since our status_code for r was 200, when we call raise_for_status() we get:

>>> r.raise_for_status()
None

 

  •  Response Headers

 We can view the server‘s response headers using a Python dictionary:

>>> r.headers
{‘Connection‘: ‘keep-alive‘, ‘Server‘: ‘meinheld/0.6.1‘, ‘Date‘: ‘Thu, 19 Oct 2017 02:34:53 GMT‘, ‘Content-Type‘: ‘application/json‘, ‘Access-Control-Allow-Origin‘: ‘*‘, ‘Access-Control-Allow-Credentials‘: ‘true‘, ‘X-Powered-By‘: ‘Flask‘, ‘X-Processed-Time‘: ‘0.000764846801758‘, ‘Content-Length‘: ‘488‘, ‘Via‘: ‘1.1 vegur‘}

 or,we can access the headers using any capitalization we want:

>>> r.headers["Date"]
‘Thu, 19 Oct 2017 02:34:53 GMT‘
>>> r.headers["Connection"]
‘keep-alive‘

  •  Cookies

 If a response contains some Cookies, you can quickly access them:

>>> url = ‘http://example.com/some/cookie/setting/url‘
>>> r = requests.get(url)

>>> r.cookies[‘example_cookie_name‘]
‘example_cookie_value‘

我这里执行失败的，先记录下来吧

 

 To send your own cookies to the server, you can use the cookies parameter:

>>> url = ‘http://httpbin.org/cookies‘
>>> cookies = dict(cookies_are = ‘working‘)
>>> r = requests.get(url,cookies = cookies)
>>> r.text
‘{\\n  "cookies": {\\n    "cookies_are": "working"\\n  }\\n}\\n‘

 

 Cookies are returned in a RequestsCookieJar, which acts like a dict but also offers a more complete interface, suitable for use over multiple domains or paths. Cookie jars can also be passed in to requests:

>>> jar = requests.cookies.RequestsCookieJar()
>>> jar.set(‘tasty_cookie‘, ‘yum‘, domain=‘httpbin.org‘, path=‘/cookies‘)
Cookie(version=0, name=‘tasty_cookie‘, value=‘yum‘, port=None, port_specified=False, domain=‘httpbin.org‘, domain_specified=True, domain_initial_dot=False, path=‘/cookies‘, path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={‘HttpOnly‘: None}, rfc2109=False)
>>> jar.set(‘gross_cookie‘, ‘blech‘, domain=‘httpbin.org‘, path=‘/elsewhere‘)
Cookie(version=0, name=‘gross_cookie‘, value=‘blech‘, port=None, port_specified=False, domain=‘httpbin.org‘, domain_specified=True, domain_initial_dot=False, path=‘/elsewhere‘, path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={‘HttpOnly‘: None}, rfc2109=False)
>>> url = ‘http://httpbin.org/cookies‘
>>> r = requests.get(url, cookies=jar)
>>> r.text
‘{\\n  "cookies": {\\n    "tasty_cookie": "yum"\\n  }\\n}\\n‘

 

  •  Timeouts

>>> requests.get("http://www.baidu.com",timeout = 0.001)
Traceback (most recent call last):
  File "C:\\Users\\yangbo\\AppData\\Local\\Programs\\Python\\Python36-32\\lib\\site-packages\\urllib3\\connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "C:\\Users\\yangbo\\AppData\\Local\\Programs\\Python\\Python36-32\\lib\\site-packages\\urllib3\\util\\connection.py", line 83, in create_connection
    raise err
  File "C:\\Users\\yangbo\\AppData\\Local\\Programs\\Python\\Python36-32\\lib\\site-packages\\urllib3\\util\\connection.py", line 73, in create_connection
    sock.connect(sa)
socket.timeout: timed out

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 














以上是关于python基础===requests学习笔记的主要内容，如果未能解决你的问题，请参考以下文章

python requests库学习笔记（下）

python requests库学习笔记（上）

MOOC《Python网络爬虫与信息提取》学习过程笔记requests库第一周1-3

Python学习笔记__13.2章 requests

requests源码阅读学习笔记

python：网络爬虫的学习笔记