Tensors

Note

To solve these exercises, consulting the torch function reference can be helpful.

Question 1: Tensor creation and manipulation

Recreate this torch tensor:

torch_tensor
 1  2  3
 4  5  6
[ CPULongType{2,3} ]
Hint First create an R matrix and then convert it using torch_tensor().

Next, create a view of the tensor so it looks like this:

torch_tensor
 1  2
 3  4
 5  6
[ CPULongType{3,2} ]
Hint Use the $view() method and pass the desired shape as a vector.

Check programmatically that you successfully created a view, and not a copy.

Hint See what happens when you modify one of the tensors.

Solution

We start by creating the tensor:

x <- torch_tensor(matrix(1:6, byrow = TRUE, nrow = 2))
x
torch_tensor
 1  2  3
 4  5  6
[ CPULongType{2,3} ]

Then, we create a view of the tensor:

y <- x$view(c(3, 2))

To check that we created a view, we can modify one of the tensors and see if the other one changes:

x[1, 1] <- 100
y
torch_tensor
 100    2
   3    4
   5    6
[ CPULongType{3,2} ]

Question 2: More complex reshaping

Consider the following tensor:

x <- torch_tensor(1:6)
x
torch_tensor
 1
 2
 3
 4
 5
 6
[ CPULongType{6} ]

Reshape it so it looks like this.

torch_tensor
 1  3  5
 2  4  6
[ CPULongType{2,3} ]
Hint First reshape to (2, 3) and then $permute() the two dimensions.

Solution We therefore first reshape to (3, 2) and then permute the two dimensions to get the desired shape (2, 3).

x <- x$reshape(c(3, 2))
x
torch_tensor
 1  2
 3  4
 5  6
[ CPULongType{3,2} ]
x$permute(c(2, 1))
torch_tensor
 1  3  5
 2  4  6
[ CPULongType{2,3} ]

Question 3: Broadcasting

Consider the following vectors:

x1 <- torch_tensor(c(1, 2))
x1
torch_tensor
 1
 2
[ CPUFloatType{2} ]
x2 <- torch_tensor(c(3, 7))
x2
torch_tensor
 3
 7
[ CPUFloatType{2} ]

Predict the result (shape and values) of the following operation by applying the broadcasting rules.

x1 + x2$reshape(c(2, 1))

Solution

The result is the following tensor:

torch_tensor
 4  5
 8  9
[ CPUFloatType{2,2} ]

We will now show how to arrive at this step by step. According to the broadcasting rules, we start by adding a singleton dimension to the first tensor:

x1 <- x1$reshape(c(1, 2))

Now, we have a tensor of shape (1, 2) and a tensor of shape (2, 1). Next, we extend the first tensor along the first dimension to match the second tensor:

x1 <- x1$expand(c(2, 2))

We do this analogously for the second (reshaped) tensor:

x2 <- x2$reshape(c(2, 1))$expand(c(2, 2))

Now they both have the same shape (2, 2), so we can add them:

x1 + x2
torch_tensor
 4  5
 8  9
[ CPUFloatType{2,2} ]

Question 4: Handling Singleton dimensions

A common operation in deep learning is to add or get rid of singleton dimensions, i.e., dimensions of size 1. As this is so common, torch offers a $squeeze() and $unsqueeze() method to add and remove singleton dimensions.

Use these two functions to first remove the second dimension and then add one in the first position.

x <- torch_randn(2, 1)
x
torch_tensor
-0.1115
 0.1204
[ CPUFloatType{2,1} ]

Solution

x$squeeze(2)$unsqueeze(1)
torch_tensor
-0.1115  0.1204
[ CPUFloatType{1,2} ]

Question 5: Matrix multiplication

Generate a random matrix \(A\) of shape (10, 5) and a random matrix \(B\) of shape (10, 5) by sampling from a standard normal distribution.

Hint Use torch_randn(nrow, ncol) to generate random matrices.

Can you multiply these two matrices with each other and if so, in which order? If not, generate two random matrices with compatible shapes and multiply them.

Solution

We can only multiply a matrix of shape (n, k) with a matrix of shape (k, m), i.e., the the number of columns in the first matrix matches the number of rows in the second matrix.

We can therefore not multiply the two matrices with each other in either order. To generate two random matrices with compatible shapes, we can generate two random matrices with shape (10, 5) and (5, 10).

A <- torch_randn(10, 5)
B <- torch_randn(5, 10)
A$matmul(B)
torch_tensor
-1.4311  0.6090 -1.4795 -0.6977  2.4857 -0.7402  0.4060 -0.4299  2.9035  0.1459
-4.0841  3.8794 -1.5376 -3.5270  4.8175 -0.7630  0.1188  3.0368  1.0634  0.0011
-0.3880 -1.4639 -1.3191 -0.0589  3.1754 -3.1779  1.7006  0.0521  5.0765  0.0552
 1.6030 -2.2295  1.1606  3.3083  3.3677  1.5567 -2.3565 -5.1759 -1.9122  5.1734
 4.0126 -4.3978  0.5547  1.9958 -3.4347 -2.2880  2.1990  0.2017  2.6702 -1.7145
 0.8548  3.0118 -2.0971 -3.3564 -8.1899  3.3494  1.5969  4.4134  0.4593 -6.8904
 0.0597 -0.1650 -2.5737 -1.1190  6.1582 -0.6400  0.8576  0.2152  5.0070  1.6070
 0.2675  2.4575 -2.6582 -3.1801 -3.0074  2.0887  1.4936  3.5447  2.3877 -4.3110
-3.7894  1.8938  0.0528 -0.9525  0.3706 -1.8813  0.0365  0.2768  0.2025 -0.8839
 2.7060 -2.1856  1.0679  2.6758 -6.8991  1.6866 -0.2875 -2.8479 -1.4630 -1.6319
[ CPUFloatType{10,10} ]

Question 6: Uniform sampling

Generate 10 random variables from a uniform distribution (using only torch functions) in the interval \([10, 20]\). Use torch_rand() for this (which does not allow for min and max parameters).

Hint Add the lower bound and multiply with the width of the interval.

Then, calculate the mean of the values that are larger than 15.

Solution Because the uniform distribution of torch has no min and max parameters like runif(), we instead sample from a standard uniform distribution and then scale and shift it to the desired interval.

n <- 10
a <- 10
b <- 20
x <- torch_rand(n) * (b - a) + a
x
torch_tensor
 17.2108
 15.4495
 15.4898
 13.4831
 15.0240
 13.4448
 16.4367
 19.8558
 15.7574
 12.7854
[ CPUFloatType{10} ]
mean(x[x > 15])
torch_tensor
16.4606
[ CPUFloatType{} ]

Question 7: Don’t touch this

Consider the code below:

f <- function(x) {
  x[1] <- torch_tensor(-99)
  return(x)
}
x <- torch_tensor(1:3)
y <- f(x)
x
torch_tensor
-99
  2
  3
[ CPULongType{3} ]

Implement a new different version of this function that returns the same tensor but does not change the value of the input tensor in-place.

Hint The $clone() method might be helpful.

Solution

We need to $clone() the tensor before we modify it.

g <- function(x) {
  x <- x$clone()
  x[1] <- torch_tensor(-99)
  x
}
x <- torch_tensor(1:3)
y <- g(x)
x
torch_tensor
 1
 2
 3
[ CPULongType{3} ]