response.json() – Working with JSON in Python requests

In this tutorial, you’ll learn how to parse a Python requests response as JSON and convert it to a Python dictionary. Whenever the requests library is used to make a request, a Response object is returned. The Python requests library provides a helpful method, json(), to convert a Response object to a Python dictionary.

By the end of this tutorial, you’ll have learned:

• How to parse a requests.Response to a Python dictionary using the Response.json() method
• How to convert a Python requests JSON response to a dictionary
• How to handle errors when parsing a Python requests JSON response
• How to convert a Python requests JSON object to a Pandas DataFrame

Parsing Python requests Response JSON Content

Every request that is made using the Python requests library returns a Response object. This is true for any type of request made, including GET, POST, and PUT requests. The requests library offers a number of different ways to access the content of a response object:

• .content returns the actual content in bytes
• .text returns the content converted to a string, using a character encoding such as UTF-8

Since you’re reading an article about how to parse the JSON from a response into a Python dictionary, the two options above may not be ideal. While you could use the json library to serialize the response, this adds an additional step and complexity to your code.

The requests library comes with a helpful method, .json(), that helps serialize the response of the request. In order to look at an example, let’s the public endpoints provided by the website reqres.in. These endpoints work without signing up, so following along with the tutorial is easy.

Let’s see how we can access the /users endpoint and serialize the response into a Python dictionary using the .json() method:

# Serializing a GET Request with .json()
import requests
resp = requests.get('https://reqres.in/api/users')
resp_dict = resp.json()

print(type(resp_dict))

# Returns: <class 'dict'>

From the code above, we can see that applying the .json() method to our response created a Python dictionary. In the following section, you’ll learn how to work with the resulting dictionary to access some content.

How to Access Python requests Response Content as a Dictionary

After applying the .json() method to a response created by the requests library, we created a dictionary. This means that we can access the data in the dictionary using common dictionary methods, such as square-bracket indexing or the .get() method.

Let’s see how we can access the 'page' key in the data:

# Accessing Data in a Python Request Response
import requests
resp = requests.get('https://reqres.in/api/users')
resp_dict = resp.json()

print(resp_dict.get('page'))

# Returns: 1

In the code above, we applied the .get() method to access the value corresponding with the key 'page'. Using the .get() method is a safer way of handling this operation. This is because the method will simply return None, if a key doesn’t exist.

How to Handle Errors When Parsing a JSON Response from Python requests

In some cases, the data that’s returned from a Response object can’t be serialized. In these cases, you’ll encounter a JSONDecodeError. You can safely handle these errors by wrapping your request in a try-except block.

Let’s see how we can safely handle a JSONDecodeError using the requests library:

# Handling a JSONDecodeError in Python
from json import JSONDecodeError
import requests
resp = requests.get('https://reqres.in/api/users/page4')
try:
    resp_dict = resp.json()
except JSONDecodeError:
    print('Response could not be serialized')

How to Pretty Print a JSON Object From Python requests

In this section, we’ll take a look at how to pretty print a JSON object that is returned from using the Python requests library. Pretty printing a JSON file is made easy using the json.dumps() function.

Let’s see how we can pretty print a JSON object from the Python requests library:

import json
import requests

resp = requests.get('https://reqres.in/api/users')
resp_dict = resp.json()
pretty = json.dumps(resp_dict, indent=4)
print(pretty)

# Returns: 
# {
#     "page": 1,
#     "per_page": 6,
#     "total": 12,
#     "total_pages": 2,
#     "data": [
#         {
#             "id": 1,
#             "email": "[email protected]",
#             "first_name": "George",
#             "last_name": "Bluth",
#             "avatar": "https://reqres.in/img/faces/1-image.jpg"
#         },
#         ...
#         {
#             "id": 6,
#             "email": "[email protected]",
#             "first_name": "Tracey",
#             "last_name": "Ramos",
#             "avatar": "https://reqres.in/img/faces/6-image.jpg"
#         }
#     ],
#     "support": {
#         "url": "https://reqres.in/#support-heading",
#         "text": "To keep ReqRes free, contributions towards server costs are appreciated!"
#     }
# }

Let’s break down what we did in the code above:

1. We loaded the response the GET request
2. We serialized the response using the .json() method
3. We then used the dumps() function with an indent of 4 to pretty print the response

How to Convert a Python requests JSON Object to a Pandas DataFrame

In this final section, we’ll take a look at how to convert a requests.Reponse object into a Pandas DataFrame. Because Pandas allows you to easily create DataFrames from a list of dictionaries, we can simply pass in part of our serialized response.

From the code above, we know that the data are stored in a key named 'data' as a list of dictionaries. We can easily pass this dictionary into a Pandas DataFrame constructor and convert it to a DataFrame:

# Converting JSON Data to a Pandas DataFrame
import requests
import pandas as pd

resp = requests.get('https://reqres.in/api/users')
resp_dict = resp.json()
df = pd.DataFrame(resp_dict.get('data'))
print(df)

# Returns: 
#    id                     email first_name last_name                                   avatar
# 0   1    [email protected]     George     Bluth  https://reqres.in/img/faces/1-image.jpg
# 1   2    [email protected]      Janet    Weaver  https://reqres.in/img/faces/2-image.jpg
# 2   3       [email protected]       Emma      Wong  https://reqres.in/img/faces/3-image.jpg
# 3   4        [email protected]        Eve      Holt  https://reqres.in/img/faces/4-image.jpg
# 4   5  [email protected]    Charles    Morris  https://reqres.in/img/faces/5-image.jpg
# 5   6    [email protected]     Tracey     Ramos  https://reqres.in/img/faces/6-image.jpg

In the code above, we serialized our response to a dictionary. From there, we passed the values of the 'data' key into the pd.DataFrame constructor, which accepts a list of dictionaries.

Conclusion

In this tutorial, you learned how to use the Python requests library’s Response object and serialize the JSON data using the .json() method. You first learned how to use the .json() method. Then, you learned how to access data from the resulting dictionary. You also learned how to handle errors caused by this method, how to pretty print the resulting data, and how to load it into a Pandas DataFrame.

Additional Resources

To learn more about related topics, check out the tutorials below:

• Python requests Response Object Explained
• Python requests: POST Request Explained
• Python requests: GET Request Explained
• How to Install requests Package in Python – Windows, macOS, and Linux