What is the Determinant?
What is the Cross Product?
What is the Dot Product

This is the fourth in a series of Linear Algebra. I suggest you read Linear Algebra 1-3 before this, as they are prerequisites to this post.


Determinant

Consider our basis vectors $\hat{i} = \begin{bmatrix}1 \\ 0 \end{bmatrix}$ and $\hat{j}  = \begin{bmatrix}0 \\ 1 \end{bmatrix}$.
They have an area (or volume in 3D space), which we can indicate like so:

That area indicated in white, is the sum of the determinant of $\hat{i}$ and $\hat{j}$. Without doing the calculation nor telling you the formula, the area would be 1.

The sum of the determinant is especially used with Linear Transformation (read Linear Algebra 3). Assuming the standard basis vectors, we can find out just how much space has been squished or stretched after a transformation by calculating the determinant of the transformed $\hat{i}$ and $\hat{j}$ — essentially it tells us how much space has been scaled.

There are 3 outcomes:

  1. If the determinant is negative (e.g. $sum < 0$), geometrically we have flipped space. Imagine turning a paper to the other side, that is what is happening.
  2. If the determinant has a $sum = 0$, then it tells us that all of space has been squished into a smaller dimension, e.g. all of 2D space being squished onto a line, or all of 3D space being squished onto a plane.
  3. If the determinant is positive (e.g. $sum > 0$), then space is not flipped, but just scaled.

If the determinant is 0, this could be the case:

Show me an example!

Let's start out by laying out the formular for finding the determinant of a matrix in 2-dimensional space (different from other dimensions):

$$ det\left( \begin{bmatrix} a & b \\ c & d \end{bmatrix}\right) = ad-bc $$

So let's apply a transformation to space and find the determinant.
Given a matrix $A=\begin{bmatrix}3 & 0 \\ 0 & 4 \end{bmatrix}$ let's apply the transformation to our basis vectors:

$$ \begin{bmatrix} 3 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} $$

Which we can split up into two equations and add at the end (read Linear Algebra 3):

$$ \begin{bmatrix} 3 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} 1 \\ 0 \end{bmatrix} = 1\begin{bmatrix} 3 \\ 0 \end{bmatrix} + 0\begin{bmatrix} 0 \\ 4 \end{bmatrix} = \begin{bmatrix} 3 \times 1 + 0 \times 0\\ 0 \times 1 + 4 \times 0 \end{bmatrix} $$
$$ \begin{bmatrix} 3 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} 0 \\ 1 \end{bmatrix} = 0\begin{bmatrix} 3 \\ 0 \end{bmatrix} + 1\begin{bmatrix} 0 \\ 4 \end{bmatrix} = \begin{bmatrix} 3 \times 0 + 0 \times 1\\ 0 \times 0 + 4 \times 1 \end{bmatrix} $$

Then we add the results:

$$ \begin{bmatrix} 3 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} 1 & 0\\ 0 & 1 \end{bmatrix} = \begin{bmatrix} 3 \times 1 + 0 \times 0 & 3 \times 0 + 0 \times 1\\ 0 \times 1 + 4 \times 0 & 0 \times 0 + 4 \times 1 \end{bmatrix} = \begin{bmatrix} 3 & 0\\ 0 & 4 \end{bmatrix} $$

The result is that $\hat{i} = \begin{bmatrix}3 \\ 0 \end{bmatrix}$ and $\hat{j}  = \begin{bmatrix}0 \\ 4 \end{bmatrix}$. This means our new area looks like this:

Now we can easily find the determinant, but I just want you to remember how this is visualized and what we are doing when finding the determinant, namely the area of these basis vectors. Take a moment to notice such transformations can squish space together or stretch it out, hereby that the area we get can be many shapes if we were to apply certain transformation.


Let me clarify why the determinant can be important in Linear Algebra, specifically when $det=0$. When that is the case, all of space is squished together on a straight line — then all areas are equal to 0. This is what we call Linear Dependence (read Linear Algebra 2), where all our vectors are stuck on a line. It means a number of things, but here is just a few with examples when the determinant is 0:

  • Matrices cannot be reversed (covered in a future post)
  • A system of equations (equations involving the same set of variables), does not have a unique solution

There is also a definition for the determinant 3-dimensional vectors:

$$ det\left( \begin{bmatrix} a & b & c \\ d & e & f \\ g & g & i \end{bmatrix}\right) = a \times det\left( \begin{bmatrix} e & f \\ h & i \end{bmatrix}\right) -b \times det\left( \begin{bmatrix} d & f \\ g & i \end{bmatrix}\right) +c \times det\left( \begin{bmatrix} d & e \\ g & h \end{bmatrix}\right) = aei+bfg+cdh-ceg-bdi-afh $$

Cross Product

The Cross Product has a nice graphic interpretation and is quite similar to that of the determinant, as covered above.

Just like we find the determinant after a transformation, we also find the cross product after a transformation. This means orientation could be flipped, just like how I mentioned above – turning a paper over is the equivalent of flipped orientation.

Remember how I told you that a vector has a tail and a head in Linear Algebra 1? Anytime we have two vectors $\vec{v}$ and $\vec{w}$, taking the cross product $\vec{v}\times\vec{w}$ results in placing a copy of $\vec{v}$'s tail on $\vec{w}$'s head and a copy of $\vec{w}$'s tail on $\vec{v}$'s. Let me just give you the image:

So every time we are taking the cross product, we can imagine, at least in 2D, that we have this geometric interpretation — and have a visual sense of what we are actually computing. If it wasn't obvious to you by now, the cross product computation is much like the determinant, as we are finding the area of this area.

And the computation? Just as easy as the determinant:

$$ \vec{v}\times\vec{w} = det\left( \begin{bmatrix} v_1 & w_1 \\ v_2 & w_2 \end{bmatrix}\right) = v_xw_y-w_xv_y $$

So we take $\vec{v}$'s first coordinate times $\vec{w}$'s second coordinate minus $\vec{w}$'s first coordinate times $\vec{v}$'s second coordinate. The sum of that is the sum of the cross product.

Very important knowledge about the Cross Product

1st BUT! In which direction the second vector points matters. If the second vector $\vec{w}$ is on the right side of the first vector $\vec{v}$, the cross product should be negative. And the reverse is also true, that is if $\vec{w}$ is on the left side of $\vec{v}$. So the actual area would be $sum$, but the cross product would be $\pm sum$.

2nd BUT! All of the above information is a correct way to conceptualize the idea of cross products... but, cross products are only done on 2 vectors in 3-dimensional space noted by $\mathbb{R}^{3}$.
And... the sum of the cross product is actually not just a sum of the area.

The sum of the cross product is the length of a new vector perpendicular to the area, which forms a plane in 3-dimensional space, of $\vec{v}$ and $\vec{w}$.

So let's explore what exactly this means? You know how I lied about the sum of the cross product is the area? Yes? Ok, so that part makes it easier to explain.

That area that we create in 2D, that area is also created in 3D, but as a plane. So the definition says that we get a new vector perpendicular to the area. That means the vector points straight away from that plane. If we imagine dragging a vector from the plane (0 degrees) to the perpendicular position (90 degrees), we would have the new vector.

The below image is in the case of the sum being positive, otherwise the arrow would have pointed the other way. Here you can see the area of the two vectors and the result of the cross product — a new vector perpendicular to the area, at 90°:

A Final Formula for Cross Product in 3-Dimensional Space

Up until now, I have tried to show you the intuition part of cross products and determinants, now let me give you the most intuitive way of calculating the determinant of 3-Dimensional space. As you probably saw further up, just below the Cross Product title, there was a (messy, but correct) formula for calculating the 3-Dimensional determinant.

Do not let this formula confuse you, it is merely just a notational trick, where it is suggested that you take the number of $\hat{i}$ in the x-direction, $\hat{j}$ in the y-direction and $\hat{k}$ in the z-direction.

$$ \begin{bmatrix} v_1 \\ v_2 \\ v_3 \end{bmatrix} \times \begin{bmatrix} w_1 \\ w_2 \\ w_3 \end{bmatrix} = det\left( \begin{bmatrix} \hat{i} & v_1 & w_1 \\ \hat{j} & v_2 & w_2 \\ \hat{k} & v_3 & w_3 \end{bmatrix}\right) $$

Which is equivalent to

$$ \hat{i}(v_2w_3-v_3w_2)+\hat{j}(v_3w_1-v_1w_3)+\hat{k}(v_1w_2-v_2w_1) $$

This is by far not something to memorize, but something to look up once you need it in Machine Learning. I would always suggest taken notes by hand on paper.

Dot Product

The last important piece of calculation for this post is the dot product. And it is much simpler than the two previous computations.

The dot product is simply a projection of one vector onto another vector. It means that we geometrically move one vector onto another vector.

So $\vec{v}$ could be projected onto $\vec{u}$:

Just like the determinant, we can identify where the vector $\vec{v}$ gets projected onto that white line through $\vec{u}$:

  1. If the dot product is positive (e.g. $sum > 0$), then the vector points in the same direction as that vector it was projected onto.
  2. If the dot product is negative (e.g. $sum < 0$), then the vector points in the opposite direction as the vector it was projected onto.
  3. If the dot product is has the $sum = 0$, then the vector is perpendicular (as explained above in the cross product). So if we project one vector onto another, then we would end up at the origin $(0,0)$. This is what we call null space, which means that all vectors are stuck at the origin (usually after a transformation).

The formula for the dot product is quite a simple one to remember:

$$ \begin{bmatrix} a\\ b\\ c \end{bmatrix} \boldsymbol{\cdot} \begin{bmatrix} d\\ e\\ f \end{bmatrix} = \begin{bmatrix} a \times d\\ b \times e\\ c \times f \end{bmatrix} = ad+be+cf $$

  1. What is the Determinant?
    The determinant is the area or volume of two or more vectors. The sum of the determinant tells us what happened after a transformation, namely if space has been flipped (like flipping a sheet of paper), squished into a smaller dimension or scaled. It is calculated by this formula in 2D (3D formula can be found further up)
    $$ det\left(\begin{bmatrix} a & b \\c & d \end{bmatrix}\right) = ad-bc$$
  2. What is the Cross Product?
    The result of a cross product is a new vector with some length. The sum of the cross product is the length, and the vector points perpendicular to the plane of 2 vectors in 3D. The formula for the cross product is as follows
    $$\begin{bmatrix}v_1 \\v_2 \\v_3\end{bmatrix}\times\begin{bmatrix}w_1 \\w_2 \\w_3\end{bmatrix}=det\left(\begin{bmatrix}\hat{i} & v_1 & w_1 \\\hat{j} & v_2 & w_2 \\\hat{k} & v_3 & w_3\end{bmatrix}\right)=\hat{i}(v_2w_3-v_3w_2)+\hat{j}(v_3w_1-v_1w_3)+\hat{k}(v_1w_2-v_2w_1)
    $$
  3. What is the Dot Product
    The result of the dot product tells us where a vector is projected onto another vector, e.g. where it lands if we move it onto another vector. The sum of the dot product tells us (much like the determinant) in which way the projected vector is pointing; same direction, opposite direction or stuck at origin after projection. The formula for the dot product is
    $$\begin{bmatrix}a\\b\\c\end{bmatrix}\boldsymbol{\cdot}\begin{bmatrix}d\\e\\f\end{bmatrix}=\begin{bmatrix}a\times d\\b\times e\\c\times f\end{bmatrix}=ad+be+cf$$