PyTorch Tensors: The Ultimate Guide

In this guide, you’ll learn all you need to know to work with PyTorch tensors, including how to create them, manipulate them, and discover their attributes. PyTorch tensors are a fundamental building block of deep-learning models. Understanding how tensors work will make learning how to build neural networks much, much easier.

By the end of this guide, you’ll have learned the following:

• How to build tensors in PyTorch (including from NumPy arrays),
• How to perform operations on PyTorch tensors, and
• How to understand the various attributes of PyTorch tensors

What are PyTorch Tensors?

PyTorch tensors are a convernstone data structure in PyTorch that are used to represent multi-dimensional arrrays. Similar to NumPy arrays, they allow you to create scalars, vectors, and matrices. However, they go much beyond NumPy arrays, by allowing you to work with GPU acceleration, create computational graphs, as well as automatic differentiation (which allows for training neural networks).

Let’s now explore some more of these points and why they’re important:

1. PyTorch Tensors are multi-dimensional arrays: PyTorch arrays are similar to mathematical tensors, meaning they can have different dimensions, including 1D, 2D and higher. You can also perform many mathematical operations on tensors, including reshaping, multiplication, and more.
2. PyToch Tensors can use GPU acceleration: PyTorch tensors can seamlessly use GPUs (Graphical Processing Units) for computations. Because deep learning models require significant numerical calculations, using a GPU can accelerate this. PyTorch makes it incredibly easy to switch between GPUs and CPUs, including for training and inference.
3. PyTorch Tensors can perform automatic differentiation: PyTorch tensors are designed to work seamlessly with PyTorch’s autograd functionality, which provides automatic differentiation. This means that when you perform operations on tensors to build and train neural networks, PyTorch automatically tracks the operations and computes gradients with respect to the input tensors.
4. PyTorch Tensors Integrate with Neural Networks: PyTorch tensors are the primary data type used in building neural networks. As you build you neural networks, you construct computational graphs. This allows you to perform forward and backward passes through your model, allowing you to optimize your model’s parameters.

Now that you’ve learned about what tensors are and why they matter, let’s dive into creating and using these PyTorch tensors!

Importing Our Libraries

To start, let’s load our libraries. We’ll, of course, use PyTorch, but we’ll also import NumPy in order to demonstrate the interoperability between the two libraries.

# Importing Our Libraries
import torch
import numpy

Now that we have the libraries ready to go, let’s start creating some tensors.

Creating PyTorch Tensors

The simplest way to create a PyTorch tensor is to pass data directly into the tensor() function. This allows you to pass in different data, such as lists of lists.

Let’s take a look at a simple example. We’ll first load a variable, data, as a list of lists. We then pass this into the torch.tensor() function to create a PyTorch tensor.

# Creating a Tensor From Scratch
data = [[1, 2], [3, 4], [5, 6]]
tensor_data = torch.tensor(data)
print(tensor_data)

# Returns:
# tensor([[1, 2],
#         [3, 4],
#         [5, 6]])

By printing our new tensor, we can see that this returns our new data structure. We can confirm the data type of our tensor by passing it into the type() function.

# Checking the Type of Tensors
print(type(tensor_data))

# Returns: <class 'torch.Tensor'>

We can see that this easily confirms that our data type is what we expect it to be.

Creating PyTorch Tensors from NumPy Arrays

In most cases, however, you won’t be creating a tensor manually. PyTorch provides extensive functionality to work with NumPy. We can create a PyTorch tensor directly from a NumPy array using the from_numpy() function. Let’s see what this looks like:

# Creating a Tensor From NumPy
np_array = np.array([[1, 2], [3, 4], [5, 6]])
tensor_np = torch.from_numpy(np_array)

print(tensor_np)

# Returns:
# tensor([[1, 2],
#         [3, 4],
#         [5, 6]])

We first use the NumPy array function to create an array from our list of lists. We then pass this into the torch.from_numpy() function, which creates the same tensor that we had before.

Creating PyTorch Tensors with Constant and Random Values

In many cases, you’ll also want to create tensors, either with constant values or with random values. PyTorch provides a number of helpful functions for just this! Let’s take break down a sampling of the functions we have available to us:

1. torch.ones() create a tensor filled with ones,
2. torch.zeros() creates a tensor filled with zeroes,
3. torch.full() creates a tensor with a fill value,
4. torch.rand() creates a tensor with random values

Let’s see what these functions look like in practice, starting with creating a tensor filled with ones:

# Creating Tensors of 1s
ones = torch.ones(size=(3,2))
print(ones)

# Returns:
# tensor([[1., 1.],
#         [1., 1.],
#         [1., 1.]])

We can see that the function expects only a single parameter, size (even though it accepts more than just this parameter). When we pass in the size parameter, we pass in a tuple that defines the shape. In this case, we create a tensor that is three by two.

We can create a similar tensor of zeros, using the torch.zeros() function, as shown below:

# Creating Tensors of 0s
zeros = torch.zeros(size=(3,2))
print(zeros)

# Returns:
# tensor([[0., 0.],
#         [0., 0.],
#         [0., 0.]])

We can see that the functionality is the same, allowing us to pass in the size of the tensor that we want to define.

We can create a similar tensor using a constant fill value using the torch.full() function. In this case, we’ll need to pass in both a size and a fill value:

# Creating Tensors with Fill Values
filled = torch.full(size=(3, 2), fill_value=2)
print(filled)

# Returns:
# tensor([[2, 2],
#         [2, 2],
#         [2, 2]])

In the code block above, we created a tensor filled with two, being allowed to define the size of the tensor again.

Finally, let’s take a look at creating a tensor filled with random values. This can be helpful for instantiating weights of a neural network. Let’s see how we can use the torch.rand() function to create a tensor of a specific size:

# Creating Random Tensors
randoms = torch.rand(size=(3,2))
print(randoms)

# Returns:
# tensor([[0.5215, 0.2354],
#         [0.2960, 0.6190],
#         [0.1766, 0.8673]])

We can see that the tensor is created above, that we create a tensor filled with random values. If you’re running this code, you’ll get different values (since the function is, well, random).

If we wanted to create reproducible results, we could set a random seed. PyTorch random seeds a bit more complex, because of the interoperability between different libraries. In some cases, you’ll also need to create a random seed for NumPy and for Python as a whole (using the random library).

Let’s see what this looks like:

# Adding Reproducibility to Our Tensors
torch.manual_seed(42)
randoms = torch.rand(size=(3,2))
print(randoms)

Creating Like-Sized Tensors in PyTorch

In this section, we’ll take a look at creating tensors a little further by creating tensors that are sized like another tensor. Understanding these functions is useful when exploring your PyTorch data. PyTorch provides a number of helpful functions to create like-sized tensors, including:

1. torch.zeros_like(), which creates a tensor of 0s
2. torch.ones_like(), which creates a tensor of 1s
3. torch.rand_like(), which creates a tensor of random values

Let’s take a look at some practical examples of what this looks like. We’ll start by instantiating a tensor and then creating like-sized tensors:

# Creating Like-sized Tensors
data = [[1, 2], [3, 4], [5, 6]]
tensor_data = torch.tensor(data)

zeros_like = torch.zeros_like(tensor_data)
ones_like = torch.ones_like(tensor_data)
random_like = torch.rand_like(tensor_data, dtype=torch.float)

print(f'Zeros Like:\n{zeros_like}')
print(f'\nOnes Like:\n{ones_like}')
print(f'\nRandom Like:\n{random_like}')

# Returns:
# Zeros Like: 
# tensor([[0, 0],
#         [0, 0],
#         [0, 0]])

# Ones Like: 
# tensor([[1, 1],
#         [1, 1],
#         [1, 1]])

# Random Like: 
# tensor([[0.9938, 0.3513],
#         [0.6042, 0.2700],
#         [0.7633, 0.9896]])

You may have noticed that we used a new parameter in the random_like() function. Because our original tensor was an integer tensor, we needed to specify that we want to create a tensor of floating point data types. You’ll learn more about data types in PyTorch later in this tutorial.

Let’s now dive into how to explore the different attributes of PyTorch tensors.

Understanding PyTorch Tensor Attributes

PyTorch tensors are objects that have different attributes. In this section, we’ll explore some of the main attributes that you’ll use throughout your deep learning projects. While tensors have many different attributes, we’ll focus on the main ones.

Tip! If you want to explore all of the attributes and methods of the tensor, check out this guide on printing an object’s attributes.

PyTorch tensors have the following key attributes:

• .shape returns the shape of a tensor as a torch.Size object
• .dtype returns the data type of the tensor as a string
• .device returns the device that the tensor is stored on

Let’s explore these with some practical examples:

# Understanding Different Tensor Attributes
data = torch.tensor([[1, 2], [3, 4]])

print(f"Shape of tensor: {data.shape}")
print(f"Datatype of tensor: {data.dtype}")
print(f"Device tensor is stored on: {data.device}")

# Returns:
# Shape of tensor: torch.Size([2, 2])
# Datatype of tensor: torch.int64
# Device tensor is stored on: cpu

In the code block above, we illustrated how to load the different attributes of a tensor. Being able to access the data type and sizes allow you to more easily troubleshoot your model. Let’s now take a look at how to modify the data type of your tensor.

Working with PyTorch Tensors of Different Data Types

PyTorch data types are flexible and allow you to use different data types. In this section, you’ll learn about how to customize the data types that your tensors use.

Let’s first create a tensor and check its data type, similar to what you have learned so far in the previous sections:

# Checking the Data Type of a Tensor
zeros = torch.zeros((3,2))
print(zeros.dtype)

# Returns: torch.float32

In the code block above, we combined using the zeros() function to create a tensor and then check its data type attribute.

By default, PyTorch will use a float32 data type. Let’s see how we can change the data type when we create our tensor. In order to do this, we can add in an optional parameter, dtype, to specify what data type to use:

# Changing a Tensor to a Different Data Type
zeros = torch.zeros((3, 2), dtype=torch.int)
print(zeros)

# Returns:
# tensor([[0, 0],
#         [0, 0],
#         [0, 0]], dtype=torch.int32)

We can see that by specifying the data type, PyTorch will modify the resulting tensor to integers.

In some cases, you’ll need to update the data type after a tensor has been created. In these cases, you can use the .to() method, which allows you to specify the data type you want to change to. Let’s see what this looks like:

# Converting an Existing Tensor to Another Data Type
int_tensor = torch.tensor([1, 2, 3, 4, 5])
float_tensor = int_tensor.to(torch.float32)

print(f'Int tensor:\n{int_tensor}')
print(f'\nFloat tensor:\n{float_tensor}')

# Returns:
# Int tensor:
# tensor([1, 2, 3, 4, 5])

# Float tensor:
# tensor([1., 2., 3., 4., 5.])

In the code block above, we first created a tensor of an integer data type. We then used the .to() method to convert that tensor to a floating point data type.

Let’s now dive into how we can use PyTorch to perform operations on our tensors.

Performing PyTorch Tensor Operations

So far, you have learned a ton about how to create and understand tensors. Let’s now explore how you can use PyTorch tensors and perform operations, such as addition and multiplication on them.

PyTorch provides a variety of simple ways to add tensors, giving you flexibility in how you structure your code. For example, you can use the .add() method to add one tensor to another. Similarly, you can also use the + operator to add two tensors together.

Take a look at the code block below to see how to add tensors in PyTorch:

# Adding Tensors
tensor1 = torch.tensor([[1, 2], [3, 4]])
tensor2 = torch.ones((2, 2))
added = tensor1.add(tensor2)
# added = tensor1 + tensor2
print(added)

# Returns:
# tensor([[2., 3.],
#         [4., 5.]])

We can see by adding two tensors using the .add() method, that the values are added element-wise, as expected.

Similarly, if you just wanted to add a single scalar to every element in the tensor, you can pass that scalar directly into the .add() method or use the + operator to add it. Let’s see what this looks like:

# Adding a Scalar to a Tensor
tensor = torch.ones((2,2))
tensor = tensor.add(1)
# tensor = tensor + 1
print(tensor)

# Returns:
# tensor([[2., 2.],
#         [2., 2.]])

PyTorch provides similar functions and operators for other mathematical operations, such as multiplication. In order to multiply two matrices together, you can use either the .multiply() method or the * operator.

# Multipling Tensors
tensor = torch.rand((2, 3))
multiplied = tensor.multiply(tensor)
# multiplied = tensor * tensor
print(multiplied)

# Returns: 
# tensor([[0.2825, 0.0252, 0.4279],
#         [0.1075, 0.4267, 0.1567]])

By using this familiar API, your PyTorch code can be easily understood. Because both the .multiply() method and the * operator are highly understandable, you don’t need to worry about the intention of your code being unclear.

In the following section, you’ll learn how to perform some of these operations in place.

Performing In-Place PyTorch Tensor Operations

The operations that you learned about in the previous section worked by re-assigning the tensor to either itself or a new tensor altogether. PyTorch provides a simple way to make these operations happen in-place.

If you see a method that is suffixed with an underscore, such as .add_(), the operation will happen in place. This means that you don’t need to re-assign the tensor to something else:

# In Place Operations
ones = torch.ones((2, 3))
print(f'Original tensor:\n{ones}')
ones.add_(3)
print(f'\nTensor with inplace operations:\n{ones}')

# Returns:
# Original tensor:
# tensor([[1., 1., 1.],
#         [1., 1., 1.]])

# Tensor with inplace operations:
# tensor([[4., 4., 4.],
#         [4., 4., 4.]])

In the example above, we used the .add_() method to add the value 3 to each item in the tensor. This would work in the same way if you were adding a tensor to the tensor using the .add_() method.

Similarly, PyTorch supports many other in-place operators. I would recommend being consistent in how you perform operations to ensure that your code’s behavior is consistent, too!

Indexing and Slicing PyTorch Tensors

PyTorch tensors also work with general Python slicing and indexing, allow you to grab and assign data based on the position of the data. Let’s see how we can slice the first row and the first column of a two-dimensional tensor:

# Slicing a Tensor
data = torch.ones((2, 3))
print(f'Indexing the first row:\n{data[0]}')
print(f'Indexing the first column:\n{data[:, 0]}')

# Returns:
# Indexing the first row:
# tensor([1., 1., 1.])
# Indexing the first column:
# tensor([1., 1.])

In the example above, we used square bracket slicing. First, we used the [0] slice, which also represents [0, :], to select the first row in the tensor. Then, we used the [: , 0] slice to select the first column of the tensor.

Now, let’s see how we can select an individual element, namely the first element. Since our tensor has two dimensions, we need to index first the row, then an element within that selection. Let’s take a look at how this works:

# Indexing a Tensor
data = torch.ones((2, 3))
print(data[0][0])

# Returns: tensor(1.)

In the example above, we first select the first item (which the first row in that tensor), then select the first item in that row. This returns a scalar.

Setting Values in a PyTorch Tensor

Now let’s take a look at how we can use slicing to set values in a tensor. We can use slicing to set a value to a subset of a PyTorch tensor. Let’s create a tensor filled with ones and then set a subset equal to 0:

# Setting Values Using Slicing
data = torch.ones((2, 3))
data[:, 0] = 0
print(data)

# Returns:
# tensor([[0., 1., 1.],
#         [0., 1., 1.]])

In the example above, we selected the values in the first column or our tensor and then used the = operator to set those values to 0. This is a handy way to make sure that we’re able to control the way in which our tensor’s data is structured.

Reshaping PyTorch Tensors

Some very common operations you’ll need to do in deep learning is to reshape tensors, such as by transposing them. Similar to transposing Pandas DataFrames, PyTorch uses the .T attribute to transpose our matrix. Let’s see what this looks like:

# Transposing a Tensor
data = torch.ones((2, 3))
transposed = data.T

print(f'Original tensor:\n{data}')
print(f'\nTransposed tensor:\n{transposed}')

# Returns:
# Original tensor:
# tensor([[1., 1., 1.],
#         [1., 1., 1.]])
#
# Transposed tensor:
# tensor([[1., 1.],
#         [1., 1.],
#         [1., 1.]])

In the code block above, we instantiated a 2×3 tensor. We then used the .T attribute to transpose it into a 3×2 tensor.

Similarly, we can use the .view() method to reshape our tensors. The view() method is used to reshape a tensor while keeping the underlying data unchanged. It allows you to change the shape of a tensor to match the desired dimensions without modifying the actual data elements. The view operation is very efficient as it merely manipulates the metadata of the tensor, avoiding the need to create a new copy of the data.

# Reshaping a Tensor Using view
ones = torch.ones((3, 2))
one_dim = ones.view(6)

print(f'Original tensor:\n{ones}')
print(f'\nUpdated tensor:\n{one_dim}')

# Returns:
# Original tensor:
# tensor([[1., 1.],
#         [1., 1.],
#         [1., 1.]])

# Updated tensor:
# tensor([1., 1., 1., 1., 1., 1.])

In the example above, we passed in the value of 6 into the .view() method. This means that we want to flatten the tensor to a single dimension. We needed to use 6, since there were 6 elements in the overall array.

However, if you don’t know how many elements are in your tensor, you can also use the value of -1 to flatten the tensor. This allows your code to be more dynamic, allowing you to prevent errors more easily. Let’s see what this looks like:

# Reshaping a Tensor Using view
ones = torch.ones((3, 2))
one_dim = ones.view(-1)

print(f'Original tensor:\n{ones}')
print(f'\nUpdated tensor:\n{one_dim}')

# Returns:
# Original tensor:
# tensor([[1., 1.],
#         [1., 1.],
#         [1., 1.]])

# Updated tensor:
# tensor([1., 1., 1., 1., 1., 1.])

Similarly, we can use the .view() method to change the structure of our tensor with multiple dimensions. The important thing to note is that the number of values in a tensor must be the same. Let’s see how we can use the .view() method to change the dimensions of our tensor:

# Reshaping a Tensor Using view to Multiple Dimensions
ones = torch.ones((3, 2))
one_dim = ones.view(2, 3)

print(f'Original tensor:\n{ones}')
print(f'\nUpdated tensor:\n{one_dim}')

# Returns:
# Original tensor:
# tensor([[1., 1.],
#         [1., 1.],
#         [1., 1.]])

# Updated tensor:
# tensor([[1., 1., 1.],
#         [1., 1., 1.]])

The example above worked because the number of values remained unchanged between our two tensors.

Sometimes, however, you might know how many values you want in a single dimension, but don’t want to calculate the other dimension. In other cases, you may not know what a dimension may be, such as when batching data in PyTorch DataLoaders.

Let’s see how we can replicate our example above using the value of -1:

# Reshaping a Tensor Using view Using Negative Values
ones = torch.ones((3, 2))
one_dim = ones.view(-1, 3)

print(f'Original tensor:\n{ones}')
print(f'\nUpdated tensor:\n{one_dim}')

# Returns:
# Original tensor:
# tensor([[1., 1.],
#         [1., 1.],
#         [1., 1.]])

# Updated tensor:
# tensor([[1., 1., 1.],
#         [1., 1., 1.]])

In the example above, we filled in -1 as our first dimension and 3 as the second. As the size of our original tensor changes, this allows our code to be dynamic.

Let’s now move on to how to move your PyTorch tensors between devices.

Working with GPUs and CPUs on PyTorch Tensors

One of the fundamental benefits of using PyTorch is that you can move your data and models between devices, such as between CPUs and GPUs. In this section, you’ll learn how to move your data between devices. Let’s start by defining a variable that holds the current device information:

# Creating a Device Variable
device = 'cuda' if torch.cuda.is_available() else 'cpu'

The code block above is a conventional way of defining device-agnostic code. This allows you to ensure that your code runs regardless of the device runs on.

Let’s now define a tensor and check what device it’s on using the .device attribute that you learned about earlier:

# Checking the Device of a Tensor
tensor = torch.tensor([1, 2, 3, 4])
print(tensor.device)

# Returns: cpu

We can see that in the code block above we checked the device that the data is on. We can see that the data are currently stored on a CPU. We can move the data to a different device by using the .to() method, which allows us to pass in a device type that we want to move it to. Let’s see what this looks like:

# Moving a Tensor to a Device
tensor = tensor.to(device)

Because we defined our code to be device-agnostic, we can pass in our device variable ensuring that the data is passed to a GPU if it’s available.

We can also define the device as we create a tensor. This is a lot more efficient if we are manually creating a tensor. Let’s see how this works:

# Creating a Tensor with a Device
tensor = torch.tensor([1, 2, 3, 4], device=device)

In the code block above, we used the device= parameter to pass in a device. This allows us to immediately create the data on a particular device, rather than needing to move it to a device afterwards.

Converting PyTorch Tensors to NumPy Arrays

A great feature of PyTorch is the interoperability between PyTorch and NumPy. One of these features is that it allows you to convert a PyTorch tensor to a NumPy array. This is done using the .numpy() method, which converts a tensor to an array.

Let’s see what this looks like in Python:

# Converting a Tensor to a NumPy Array
tensor = torch.tensor([1, 2, 3, 4])
numpy_array = tensor.numpy()
print(numpy_array)

# Returns: [1 2 3 4]

In the code block above, we first created a PyTorch tensor. Then, we converted it to a NumPy array using the .numpy() method.

Tracking Gradients with PyTorch Tensors

In this final section, I’ll briefly demonstrate how you can enable gradient tracking on PyTorch tensors. This is an important element to be aware of when creating deep learning models. By default, gradient tracking is turned off.

You can enable gradient tracking by using the requires_grad=True argument when you define a tensor:

# Adding Gradient Tracking
tensor = torch.tensor([1, 2, 3, 4], dtype=torch.float32, requires_grad=True)
print(tensor)

# Returns:
# tensor([1., 2., 3., 4.], requires_grad=True)

We can see that by printing out the tensor, that gradient tracking has been enabled. This can also be confirmed using the .requires_grad attribute, which returns a boolean value indicating whether gradient tracking is enabled or not.

Conclusion

In conclusion, this guide has equipped you with essential skills for working with PyTorch tensors – the building blocks of deep-learning models. You’ve learned to create tensors, perform operations on them, and understand their attributes. With this knowledge, building neural networks and tackling machine learning challenges using PyTorch becomes much more accessible.

Now equipped with a strong foundation, you’re ready to explore more advanced topics, optimize your models, and contribute to the ever-evolving field of artificial intelligence. Embrace your newfound understanding of PyTorch tensors and let it propel you toward exciting possibilities in the world of deep learning. Happy coding and may your machine learning journey be filled with success!

To learn more about PyTorch tensors, check out the official documentation.