Tensors are multidimensional arrays with a uniform type. (Wikipedia)

PyTorch provides a core data structure, the Tensor, which is a multidimensional array that has many similarities with NumPy arrays, but in addition to Numpy, has extra features to accelerate mathematical operations, and an extensive library of common deep learning functions. With PyTorch, both tensors and related operations can run on the CPU or GPU.

One of the key things to understand about tensors it’s that they require all their elements to have a single data type (also called dtype). They require also their data to be in a regular shape, it means regularity has to be maintained. Simply explained tensors is a container for data, which can be a number, a vector or an n-dimensional array. In Machine Learning Tensors are an important data structure and can be thought of generalisation of scalars, vectors and matrices elements and can be easily understood as a multidimensional array.

“In the general case, an array of numbers arranged on a regular grid with a variable number of axes is known as a tensor.” - “Deep Learning” (Goodfellow et al.)

A scalar is considered to be a zero-order tensor or a rank zero tensor.
A vector is a one-dimensional or first-order tensor, and a matrix is a two-dimensional or second-order tensor.

Shape is a property of the tensors, it gives us the length along each dimension. A tensor with a scalar will have a shape of zero! One array of 5 elements will have as shape one element like [1].
A matrix will have rows and columns as shape property like [2, 3].

What is the difference between a tensor and a matrix? A matrix has to have two dimensions. We can say a matrix is a special type of tensor but a tensor is not a matrix.

Consider tensor shapes as the number of lists that a dimension holds. For instance, a tensor shaped (1, 4, 4, 2) will have 1 list containing 4 elements of 4 elements of 2 elements. The first dimension can hold 1 element. The second can hold 4 elements. The third can hold 4 elements. The fourth dimension can hold 2 elements.

What is the difference between python arrays and tensors?

  • Python lists store numbers as full-fledged objects. A floating-point number takes only 32 bits to be represented on a computer but Python creates objects with overheads like reference counting etc. This is inefficient for large arrays.
  • Python lists are not optimized for operations like the dot product of two vectors or summing two vectors. Also, the layout of their content in memory is not optimized. PyTorch tensors or NumPy arrays are contiguous memory blocks containing unboxed C numeric types, not Python objects. A tensor is a view of such contiguous storage instance, which is always one-dimensional, and can have different shapes but at the core is pointing to the same data allocation.
  • The python interpreter is slower than the optimized compiled versions of libraries like Numpy or PyTorch which are written in C.

Some resources

This is a really good resource about PyTorch. They offer a free download of some of the chapters: https://pytorch.org/deep-learning-with-pytorch
“Deep Learning” Goodfellow et al. has a free html version here : https://www.deeplearningbook.org
You can see some of my notebooks here at https://jovian.ml/pymultitudes.