```
import torch
= torch.ones(2, 4, requires_grad=True)
x x
```

```
tensor([[1., 1., 1., 1.],
[1., 1., 1., 1.]], requires_grad=True)
```

Author

Marie-Hélène Burle

PyTorch has automatic differentiation capabilities—meaning that it can track all the operations performed on tensors during the forward pass and compute all the gradients automatically for the backpropagation—thanks to its package torch.autograd.

Let’s have a look at this.

**Derivative of a function:**

Rate of change of a function with a single variable w.r.t. its variable.

**Partial derivative:**

Rate of change of a function with multiple variables w.r.t. one variable while other variables are considered as constants.

**Gradient:**

Vector of partial derivatives of function with several variables.

**Differentiation:**

Calculation of the derivatives of a function.

**Chain rule:**

Formula to calculate the derivatives of composite functions.

**Automatic differentiation:**

Automatic computation of partial derivatives by algorithms.

First, we need to talk about backpropagation: the backward pass following each forward pass and which adjusts the model’s parameters to minimize the output of the loss function.

The last 2 videos of 3Blue1Brown neural network series explains backpropagation and its manual calculation very well.

There is one minor terminological error in this video: they call the use of mini-batches *stochastic gradient descent*. In fact, this is called *mini-batch gradient descent*. Stochastic gradient descent uses a single example at each iteration.

If we had to do all this manually, it would be absolute hell. Thankfully, many tools—including PyTorch—can do this automatically.

For the automation of the calculation of all those derivatives through chain rules, PyTorch needs to track computations during the forward pass.

PyTorch does not however track all the computations on all the tensors (this would be extremely memory intensive!). To start tracking computations on a vector, set the `requires_grad`

attribute to `True`

:

```
tensor([[1., 1., 1., 1.],
[1., 1., 1., 1.]], requires_grad=True)
```

`grad_fun`

attributeWhenever a tensor is created by an operation involving a tracked tensor, it has a `grad_fun`

attribute:

You don’t want to track more than is necessary. There are multiple ways to avoid tracking what you don’t want.

You can stop tracking computations on a tensor with the method `detach`

:

You can change its `requires_grad`

flag:

```
tensor([[0., 0., 0.],
[0., 0., 0.]], requires_grad=True)
```

Alternatively, you can wrap any code you don’t want to track under `with torch.no_grad()`

:

```
tensor([[2., 2., 2., 2.],
[2., 2., 2., 2.]])
```

Compare this with what we just did above.

Let’s calculate gradients manually, then use autograd, in a very simple case: imagine that \(x\), \(y\), and \(z\) are tensors containing the parameters of a model and that the error \(e\) could be calculated with the equation:

\[e=2x^4-y^3+3z^2\]

Let’s see how we would do this manually.

First, we need the model parameters tensors:

We calculate \(e\) following the above equation:

The gradients of the error \(e\) w.r.t. the parameters \(x\), \(y\), and \(z\) are:

\[\frac{de}{dx}=8x^3\] \[\frac{de}{dy}=-3y^2\] \[\frac{de}{dz}=6z\]

We can calculate them with:

For this method, we need to define our model parameters with `requires_grad`

set to `True`

:

\(e\) is calculated in the same fashion (except that here, all the computations on \(x\), \(y\), and \(z\) are tracked):

The backward propagation is done automatically with:

And we have our 3 partial derivatives:

The result is the same, as can be tested with:

Of course, calculating the gradients manually here was extremely easy, but imagine how tedious and lengthy it would be to write the chain rules to calculate the gradients of all the composite functions in a neural network manually…