The Python requests library in Python allows you to send HTTP requests easily. The library abstracts much of the complexity of making requests, by using a simple and consistent API. This guide provides an in-depth view of the essential aspects of the requests library and points to further guides that go even deeper.
By the end of this guide, you’ll have learned:
- How to make HTTP requests, including
GETandPOSTrequests - How to customize HTTP requests by using headers and query strings
- How to authenticate requests using different authentication methods
- How to understand the responses you get back, including converting them to Python dictionaries
- How to use proxies for your HTTP requests
- How to create sessions to streamline making multi-part requests
- How to work with timeouts to keep your program running optimally
Table of Contents
What is the Python Requests Library?
The Python requests library is a Python library that handles making HTTP requests. The library is well known for being simple and elegant, by abstracting away much of the complexity of working with HTTP requests.
The library is also one of the more popular libraries available in Python: it currently draws around 30,000,000 downloads per week and is in use by over 1,000,000 repositories on Github.
How Do You Install the Python Requests Library?
The simplest way to install the requests library is to use the Python pip package manager. In order to install the latest version of the library, you can simply call the following command in the command prompt or terminal.
python -m pip install requestsTo install a specific version of the library, such as version 2.28.1, you can write the following command:
python -m pip install requests==2.28.1It’s as easy as that! In the following section, you’ll learn how to install the requests library on macOS using the pip package manager.
Making HTTP GET Requests with the Python Requests Library
An HTTP GET request is used to retrieve data from the specified resource, such as a website. When using the Python requests library, you can use the .get() function to create a GET request for a specified resource.
The .get() function accepts two parameters:
- A
url, which points to a resource, and params, which accepts a dictionary or tuples to send in the query string
Let’s see how we can use the get() function to make a GET request:
# Making a GET Request with requests.get()
import requests
resp = requests.get('https://reqres.in/api/users')
print(resp)
# Returns:
# <Response [200]>Let’s break down what we did in the code above:
- We imported the
requestslibrary - We created a new variable,
resp, which is the result of passing our endpoint into theget()function - We then printed out the variable, which returned a
Responseobject
The Response object has a number of different properties and methods that we can access to learn more about the data that has been returned.
Some of the key properties and methods available to a Response object include:
.status_code, which is the integer code of the responded HTTP status, such as 200 or 404.ok, which returnTrueis the status code of the response is less than 400, otherwiseFalse.json(), which returns the JSON-encoded content of a response.content, which contains the content of the response.next, which returns a request for the new request in a redirect chain, if there is one
In the following section, you’ll learn how to make POST requests using the Python requests library.
Making HTTP POST Requests with the Python Requests Library
An HTTP POST request is used to send data to a server, where data are shared via the body of a request. In the request.post() function, data are sent with the data parameter, which accepts a dictionary, a list of tuples, bytes or a file object.
Let’s take a look at what the requests.post() function looks like in Python:
# Understanding the requests.post() Function
import requests
requests.post(
url,
data=None,
json=None,
**kwargs
)We can see that the function accepts three parameters:
url, which defines the URL to post todata, which accepts a dictionary, list of tuples, butes, or a file-like object to send in the body of the requestjson, which represents json data to send in the body of the request
To illustrate using this function, we’ll be using the https://httpbin.org/ website, which allows us to place actual POST requests.
Let’s create our first POST request using Python:
# Creating a POST Request with requests.post
import requests
resp = requests.post('https://httpbin.org/post')
print(resp)
# Returns:
# <Response [200]>In the example above, we simply pass the URL to the /post endpoint into the requests.post() function. We then print the response that we get back, which returns a Response object.
Customizing Headers in HTTP Requests Using Python Requests
HTTP headers allow the client and server to pass additional information while sending an HTTP request or receiving a subsequent response. These headers are case-insensitive, meaning that the header of 'User-Agent' could also be written as 'user-agent'.
Additionally, HTTP headers in the requests library are passed in using Python dictionaries, where the key represents the header name and the value represents the header value.
HTTP headers allow the client and server to pass additional information while sending an HTTP request or receiving a subsequent response. These headers are case-insensitive, meaning that the header of 'User-Agent' could also be written as 'user-agent'.
Additionally, HTTP headers in the requests library are passed in using Python dictionaries, where the key represents the header name and the value represents the header value.
# Passing HTTP Heades into a Python requests.get() Function
import requests
url = 'https://httpbin.org/get'
headers = {'Content-Type': 'text/html'}
print(requests.get(url, headers=headers))
# Returns: <Response [200]>Customizing Query String in HTTP Requests Using Python Requests
Query string parameters allow you to customize a GET request by passing values directly into the URL or into the params= parameter.
There are three ways to accomplish creating query parameters:
- Pass in a dictionary of parameters into the
params=parameter, such as{'page': '2'} - Pass in a list of tuples into the
params=parameter, such as[('page', '2')] - Directly modify the URL string by appending
?page=2to the URL
Let’s see how we can use the first method to accomplish this:
# Passing in Query Parameters Into Our GET Request
import requests
params = {'page': '2'}
resp = requests.get('https://reqres.in/api/users', params=params)Even when you don’t pass in headers, headers will be returned in a Response object. The headers are accessible via the .headers attribute, which returns a dictionary.
Let’s see how we can return the headers available in a Response object:
# Checking the Headers of a Response Object
import requests
response = requests.get('https://httpbin.org/get')
print(response.headers)
# Returns:
# {'Date': 'Wed, 03 Aug 2022 14:12:50 GMT', 'Content-Type': 'application/json', 'Content-Length': '312', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true'}In the code block above, we used a get() function call to get a Response object. We then accessed the headers of the response by accessing the .headers attribute.
How to Understand Responses from HTTP Requests in Python Requests
The Python requests library returns a Response object when any type of request is made, including POST and GET requests. The Response object contains useful attributes and methods that help in understand the response.
Before diving further, let’s see how we can create a Response object and verify its type:
# Getting a requests Response Object
import requests
resp = requests.get('https://reqres.in/api/users')
print(type(resp))
# Returns: <class 'requests.models.Response'>We can see that when we make a request, in this case a GET request, a Response object is returned.
Understanding whether or not a request was successful is, of course, a very important part of understanding a response. The requests.Response object provides a number of different ways to do this. Let’s explore two key attributes:
Response.status_codereturns an integer value representing the status of the responseResponse.okreturns a boolean value indicating success
A common task you’ll want to take on is seeing the body of a response that gets returned when a request is made. There are a number of simple ways in which we can do this:
.contentreturns the actual content in bytes.textreturns the content converted to a string, using a character encoding such as UTF-8
Both of these methods work well for accessing the content of the response. Using the .text attribute simply converts the values to a string for easier access.
How to Convert Python Requests Responses to JSON
The requests library comes with a helpful method, .json(), that helps serialize the response of the request. In order to look at an example, let’s the public endpoints provided by the website reqres.in. These endpoints work without signing up, so following along with the tutorial is easy.
Let’s see how we can access the /users endpoint and serialize the response into a Python dictionary using the .json() method:
# Serializing a GET Request with .json()
import requests
resp = requests.get('https://reqres.in/api/users')
resp_dict = resp.json()
print(type(resp_dict))
# Returns: <class 'dict'>From the code above, we can see that applying the .json() method to our response created a Python dictionary.
How to Authenticate HTTP Requests with Python Requests
The Python requests library provides a number of different ways to authenticate requests, which are covered in detail in this post. Let’s first take a look at basic authentication.
Basic authentication refers to using a username and password for authentication a request. Generally, this is done by using the HTTPBasicAuth class provided by the requests library. However, as you’ll later learn, the requests library makes this much easier, as well, by using the auth= parameter.
Let’s see how we can pass in a username and password into a simple GET request using the HTTPBasicAuth class:
# Authentication in Requests with HTTPBasicAuth
import requests
from requests.auth import HTTPBasicAuth
auth = HTTPBasicAuth('user', 'pass')
print(requests.get('https://httpbin.org/basic-auth/user/pass', auth=auth))
# Returns: <Response [200]>Let’s break down what we did in the code above:
- We imported both the
requestslibrary as well as only theHTTPBasicAuthclass - We then created a new
HTTPBasicAuthobject,auth, which contains a string for the username and password - Finally, we printed the
Responsewe returned when passing ourauthvariable into theauth=parameter of theget()function
If you were using this method, you’d change 'user' and 'pass' to the username and password of your choice.
Because the basic authentication method is used so frequently, the requests library abstracts away some of this complexity. Rather than needing to create a new HTTPBasicAuth object each time, you can simply pass a tuple containing your username and password into the auth= parameter.
Let’s see what this looks like:
# Simplifying Basic Authentication with Python requests
import requests
print(requests.get('https://httpbin.org/basic-auth/user/pass', auth=('user', 'pass')))
# Returns: <Response [200]>In the code above, we were able to significantly reduce the complexity of our code. The Python requests library handles a lot of the boilerplate code for us!
To learn more about other forms of authentication, check out this dedicated article on authenticating HTTP requests with Python.
How to Create Sessions with Python Requests
A Session object in the Python requests library allows you to persist parameters across different requests being made. Similarly, it allows you to persist cookies across requests. Finally, it also used connection pooling, which can make subsequent requests gain significant performance improvements.
What’s great about requests Session objects is that they allow you to persist items, such as authentication, across multiple requests. This allows you to, for example, scrape websites much more elegantly than you may otherwise.
An important thing to note is that a session object has access to all the methods of the main Requests API.
You can create a session object by instantiating a Session class, as shown below:
# Creating a Session Object
import requests
s = requests.Session()In the following section, you’ll learn how to set headers for a Python requests Session.
For example, you can persist in authenticating requests using Python Session objects. This allows you to pass in authentication to the Session object. This can be done by setting the .auth attribute of the session directly.
Let’s see how this can be done using Python:
# Persisting Authentication for Python requests Sessions
import requests
s = requests.Session()
s.auth = ('username', 'password')In the example above, we created a Session object. Then, we assigned a tuple containing a username and password to the .auth attribute of the Session object.
How to Set Timeouts for HTTP Requests with Python Requests
By default, the Python requests library does not set a timeout for any request it sends. This is true for GET, POST, and PUT requests. While this can prevent unexpected errors, it can result in your request running indefinitely.
Because of this, it’s important to set a timeout to prevent unexpected behavior.
In order to set a timeout in an HTTP request made via the requests library, you can use the timeout parameter. The parameter accepts either an integer or a floating point value, which describes the time in seconds.
It’s important to note that this behavior is different from many other HTTP request libraries, such as those in JavaScript. In other libraries or languages, this behavior tends to be expressed in milliseconds.
Let’s take a look at an example of how we can send a GET request with a timeout:
# Setting a Timeout on a GET Request with an Integer
import requests
resp = requests.get('http://datagy.io', timeout=3)In the example above, we set a timeout of 3 seconds. We used an integer to represent the time of our timeout. If we wanted to be more precise, we could also pass in a floating point value:
# Setting a Timeout on a GET Request with a Floating Point Value
import requests
resp = requests.get('http://datagy.io', timeout=3.5)By passing in a single value, we set a timeout for the request. If we wanted to set different timeouts for connecting and reading a request, we can pass in a tuple of values.
How to Use Proxies When Using Python Requests
In order to use proxies in the requests Python library, you need to create a dictionary that defines the HTTP, HTTPS, and FTP connections. This allows each connection to map to an individual URL and port. This process is the same for any request being made, including GET requests and POST requests.
It’s important to note that while the connections map to individual URLs and ports, they can actually point to the same URL and port.
Let’s see how you can define a set of proxies for the Python requests library:
# Setting up Proxies with the requests Library
import requests
proxy_servers = {
'http': 'http://proxy.sample.com:8080',
'https': 'http://secureproxy.sample.com:8080',
}
response = requests.get('sample.abc', proxies=proxy_servers)Let’s break down what we did above:
- We imported the
requestslibrary - We defined a dictionary,
proxy_servers, which mapped URLs and ports to HTTP and HTTPS connections - We then made a GET request and passed in our dictionary into the
proxies=argument
Conclusion
In this guide, you learned how to use the Python requests library to make HTTP requests. The library provides a lot of functionality to streamline and simplify making HTTP requests and working with their responses.
First, you learned how to make HTTP requests, including GET and POST requests. Then, you learned how to customize these requests using headers and query strings. You also learned how to authenticate requests in various ways.
Following this, you learned how to understand the responses that get returned. You also learned how to use sessions to make requests simpler and persistent. Then, you also learned how to use proxies and timeouts to manage the requests that you make.
Additional Resources
To learn more about related topics, check out the tutorials below: