Linear Algebra for Deep Learning

From Deep Learning Course Wiki
Jump to: navigation, search


Linear Algebra

Linear Algebra is a branch of mathematics that seeks to describe lines and planes using structures like vectors and matrices. Khan Academy has awesome Linear Algebra tutorials.


Vectors in geometry are 1-dimensional arrays of numbers or functions used to operate on points on a line or plane. Vectors store the magnitude and direction of a potential change to a point. A vector with more than one dimension is called a matrix.

Vector Notation

There are a variety of ways to represent vectors:

Vector notation.png

Or more simply:

How Vectors Work

Vectors typically represent movement from a point. They store both the magnitude and direction of potential changes to a point. This vector says move left 2 units and up 5 units.

A vector can be applied to any point on a plane:

Vector field scalar.png

The vector’s direction equals the slope created by moving up 5 and left 2. Its magnitude equals the length of the hypotenuse (the long side in a right angle triangle).

Vector magnitude direction.png


Why are Vectors Useful?

Vectors can be used to manipulate gradients. Vector operations on gradients help us find the slope of a point in any direction.


The output of vector addition is another vector.


y = np.array([1,2,3])
x = np.array([2,3,4])
y + x = [3, 5, 7]


The output of vector subtraction is another vector.


y = np.array([1,2,3])
x = np.array([2,3,4])
y - x = [-1, -1, -1]


The output of vector division is another vector.


y = np.array([1,2,3]).astype(float)
x = np.array([2,3,4]).astype(float)
y / x = [.5, .67, .75]


Deep learning relies on two different techniques for multiplying vectors/matrices: Dot Product and Hadamard Product. In both cases the inputs must be of equal dimensions.

Dot Product

The output of dot product is a scalar number.


y = np.array([1,2,3])
x = np.array([2,3,4]),x) = 20


Hadamard product

Element-wise multiplication of two vectors or matrices. The output of Hadamard product is another vector.


y = np.array([1,2,3])
x = np.array([2,3,4])
y * x = [2, 6, 12]

You can take the Hadamard product of a matrix and a vector as long as they have at least one dimension in common: either the same number of rows or the same number of columns. This rule extends to all the above operations except for dot product.

a = np.array([
b = np.array([
a * b
[[ 3,  4],
 [10, 12]]

Unit Vectors

A unit vector is a vector with a magnitude of 1. Unit vectors can have any slope (move in any direction), but the magnitude (length of vector) must equal 1. They are useful when you care only about the direction of the change, and not the magnitude. Unit vectors are used by directional derivatives.


Given the vector (3, 4), the magnitude is 5 (hypotenuse). This is not a unit vector. To find the unit vector of this vector, we divide each value by the magnitude of the vector. The new vector (3/5, 4/5) points in the same direction and has a magnitude of 1.

Vector Fields

A vector field is a diagram that shows how for a given point in space (a,b), where that point would move in your if you apply a vector function to it. Given a point, the vector field shows the “power” and “direction” of our vector function. Here is an example vector field:

Vector field functions.png Source

Wait, why does the vector field point in different directions? Shouldn’t it look like this:

Vector field scalar.png Source

The difference is because the second vector contains only scalar numbers. So from any point we always move over 2 and up 5. The first vector on the other hand contains functions. So for each point, we derive the direction by inputing the coordinates into a function. For non-linear functions, things can become very fancy indeed.


A matrix is a rectangular grid of numbers or terms (like an Excel spreadsheet) with special rules for addition, subtraction, and multiplication.


We describe the dimensions of a matrix as rows x columns.


a = np.array([
a.shape == (2,3)

Addition and Subtraction

In order to add or subtract, the dimensions must be the same. We then add/subtract the corresponding values in each matrix.

Numpy add and subtract.

a = np.array([
b = np.array([

a + b
[[2, 4],
 [6, 8]]

a - b
[[0, 0],
 [0, 0]]


Matrix multiplication specifies a set of rules for multiplying matrices to produce a new matrix.

Why is it Useful?

It turns complicated problems into simple, more efficiently calculated problems. It’s used in a number of fields including machine learning, computer graphics, and population ecology. Source


Not all matrices are eligible for multiplication. In addition, there is a requirement on the dimensions of the matrix product. Source.

  1. The number of columns in the first matrix must equal the number of rows in the second
  2. The product of an M x N matrix and an N x K matrix is an M x K matrix. The new matrix takes the rows of the first and columns of the second.


Matrix multiplication uses Dot Product to multiply various combinations of rows and columns to derive its product. In the image below, each entry in Matrix C is the dot product of a row in matrix A and a column in matrix B.

Matrix product steps.png Source.

The operation means we take the dot product of the 1st row in matrix A and the 1st column in matrix B .

Here's another way to look at it:


Q1: What are the dimensions of the matrix product?

Q2: What are the dimension of the matrix product?

Q3: What is the matrix product?

Q4: What is the matrix product?

Q5: What is the matrix product?

Why Does It Work This Way?

It’s an arbitrary human construct. There is no mathematical law underlying why it's done this way. Mathematicians decided on this approach because it turned out to be very useful in real life.


Matrix multiplication with Numpy.

a = np.array([
        [1, 2]
a.shape == (1,2)

b = np.array([
        [3, 4],
        [5, 6]
b.shape == (2,2)

#Matrix Multiply
mm =,b)
mm == [13, 16]
mm.shape == (1,2)

Identity and Inverse

The Identity Matrix in an x square matrix where if multiplied by any other matrix of the same dimensions equals that matrix. It's like multiplying by 1.

The identity matrix consists of all zeros except for a diagonal line of 1s from top left to the bottom right. Wikipedia.

The inverse matrix of another square matrix of equal dimensions is the matrix where when multiplied together produces the identity matrix.


To transpose a matrix there are two steps:

  1. Rotate the matrix 90°
  2. Reverse the order of elements in each row (e.g. [a b c] becomes [c b a])

Transpose M into T:

Matrix transpose with numpy.

a = np.array([
     [1, 2], 
     [3, 4]

[[1, 3],
 [2, 4]]

Linear Algebra in Deep Learning

Explanation of how linear algebra is used in Deep Learning.


I recommend starting here. These tutorials explain things intuitively with very little math: