Python requests and Persistent Sessions

In this tutorial, you’ll learn how to use the Python requests Session object to persist certain parameters across requests. Doing this allows you to persist cookies across all of the requests made from the Session and will use the same connection pooling. Sessions allow you to work across HTTP GET and HTTP POST requests, as well as other types of requests.

By the end of this tutorial, you’ll have learned:

• What Python requests Session objects are and why you would want to use them
• How to set headers and authenticate data when using Python requests sessions
• How to set and check cookies when working with Python requests Session objects

Python Requests Session Objects Explained

A Session object in the Python requests library allows you to persist parameters across different requests being made. Similarly, it allows you to persist cookies across requests. Finally, it also used connection pooling, which can make subsequent requests gain significant performance improvements.

What’s great about requests Session objects is that they allow you to persist items, such as authentication, across multiple requests. This allows you to, for example, scrape websites much more elegantly than you may otherwise.

An important thing to note is that a session object has access to all the methods of the main Requests API.

You can create a session object by instantiating a Session class, as shown below:

# Creating a Session Object
import requests
s = requests.Session()

In the following section, you’ll learn how to set headers for a Python requests Session.

How to Set Headers for a Python requests Session Object

When using Python requests Session objects, you can persist headers of requests being made within the session by providing data to the properties of the session. This allows you to send the same headers in each request that’s part of the session, without needing to specify the headers each time.

Let’s see how you can set headers for a Python Session object:

# Setting Persistent Headers for a Python requests Session
import requests

s = requests.Session()
s.headers.update({'accept': 'datagy'})

resp = s.get('https://httpbin.org/headers')
print(resp.text)

# Returns:
# {
#   "headers": {
#     "Accept": "datagy", 
#     "Accept-Encoding": "gzip, deflate, br", 
#     "Host": "httpbin.org", 
#     "User-Agent": "python-requests/2.27.1", 
#   }
# }

We can see here that the headers were updated at the session level and persisted across requests.

It’s important to note that if we had passed headers into a method call (rather than updating the session) the headers aren’t persisted across the session.

In the following section, you’ll learn how to set authentication methods for Python requests Session objects.

How to Set Authentication for a Python requests Session Object

Similar to persisting headers, you can persist authenticating requests using Python Session objects. This allows you to pass in authentication to the Session object. This can be done by setting the .auth attribute of the session directly.

Let’s see how this can be done using Python:

# Persisting Authentication for Python requests Sessions
import requests

s = requests.Session()
s.auth = ('username', 'password')

In the example above, we created a Session object. Then, we assigned a tuple containing a username and password to the .auth attribute of the Session object.

How to Persist Cookies for a Python requests Session Object

A main advantage of using sessions is that the cookies are persisted between requests. We can use the httpbin service to set and check cookies of a Session object. This way, we can easily first set some cookies and then return cookies from another request.

Let’s see how this can be done using Python:

# Setting and Persisting Cookies Across Sessions
import requests

s = requests.Session()
s.get('https://httpbin.org/cookies/set/sessioncookie/datagyiscool')
resp = s.get('https://httpbin.org/cookies')

print(resp.text)

# Returns:
# {
#   "cookies": {
#     "sessioncookie": "datagyiscool"
#   }
# }

Let’s break down what the code above does:

1. We created a session object, s
2. We then placed a GET request to set a session cookie, labeling the cookie 'datagyiscool'
3. We then sent another request to get the cookies of the session
4. Finally, we printed the .text attribute of the Response object

Conclusion

In this tutorial, you learned how to use sessions in Python requests. This allows you to easily persist data across multiple requests. This allows your code to be much cleaner and more consistent. Additionally, using sessions has the potential of making your requests much more performant.

You first learned how to create Session objects. Then, you learned how to persist headers. Then, you learned how to set and persist authentication across multiple requests. Finally, you learned how to set and persist cookies across multiple requests.

Additional Resources

To learn more about related topics, check out the tutorials below:

• response.json() – Working with JSON in Python requests
• How to Install requests Package in Python – Windows, macOS, and Linux
• Python requests: POST Request Explained
• Official Documentation: Python requests sessions