[Linear Algebra] 6. Inverse matrices, column space and null space

Let's think about the usefulness of linear algebra. One of the main reasons that linear algebra is broadly applicable is that it can solve system of linear equations.

 

 

"System of linear equations":

$$
\begin{matrix}
ax + by + cz = l \\
dx + ey + fz = m \\
gx + hy + iz = n \\
\end{matrix}
$$

 

Let's package "system of linear equations" into single vector equation, which consists of 1). matrix ($A$) which contain all of the constant coefficient, 2). vector ($\mathbf{\vec{x}}$) which contain all of the variables, and constant vector ($\mathbf{\vec{v}}$), result of matrix-vector multiplication.

$$
\begin{matrix}
ax + by + cz = l \\
dx + ey + fz = m \\
gx + hy + iz = n \\
\end{matrix}
\longrightarrow
\underset{A}{\begin{bmatrix}a & b & c \\ d & e & f \\ g & h & i \end{bmatrix}}
\underset{\mathbf{\vec{x}}}{\begin{bmatrix}x \\ y \\ z\end{bmatrix}}
=
\underset{\mathbf{\vec{v}}}{\begin{bmatrix}l \\ m \\ n\end{bmatrix}}
$$

 

There is pretty cool geometric interpretation for this problem. The matrix $A$ corresponds with some linear transformation, so solving $A\mathbf{\vec{x}} = \mathbf{\vec{v}}$ means we're looking a vector $\mathbf{\vec{x}}$ which, after applying the transformation, lands on $\mathbf{\vec{v}}$.

 

Now, let's think about how to solve these equations. It depends on whether the transformation associated with $A$ squishes where determinant is $0$, or not.

 

Non-zero determinant

 

In this case, transformation $A$ does not squish the space. So there always be one and only one vector that lands on $\mathbf{\vec{v}}$. Therefore, we can solve these equations by playing the transformation $A$ in inverse.

 

The transformation $A$ in inverse is also another linear transformation, commonly called "the inverse of $A$" denoting $A^{-1}$. The core property of this transformation is that if we first apply $A$, then follow it with the transformation $A^{-1}$, we end up back where we started. So $A^{-1}A$ equals the matrix known as "identity transformation" which does nothing.

 

Therefore, once we find $A$ inverse, we can solve these equations by multiplying $A^{-1}$ by $\mathbf{\vec{v}}$. If determinant of $A$ is a non-zero, then there is a unique solution. This idea also makes sense in higher dimensions.

 

Zero determinant

 

In this case, transformation $A$ squishes the space. So there is no inverse of $A$. Because we cannot restore a line to plane. Moreover, there is no function that transform individual vector into a whole line.

 

It's still possible that a solution exists even when there is no inverse. If transformation squishes space onto a line, there will be a solution when the vector $\mathbf{\vec{v}}$ is sitting somewhere on that line.

 

Pseudo-inverse matrix

 

If there is no inverse matrix, we can solve equations by pesudo-inverse matrix $A^+$.

$$\begin{matrix} \text{if } n \ge m \quad A^+ = (A^\top A)^{-1} A^\top \\ \text{else if } n \le m \quad A^+ = A^\top (AA^\top)^{-1} \end{matrix}$$

---

Rank

There is a difference in degree among the cases where the determinant is $0$. Given a $3 \times 3$ matrix, for example, it seems a lot harder for a solution to exist, when it squishes space onto a line compared to when it squishes onto plane, even though both of those are zero determinant.

 

So there is terminology that's a bit more specific than "zero determinant". When the output of a transformation is a line, meaning it's one-dimensional, we say the transformation has a "rank" of $1$. If all the vectors land on some two-dimensional plane, we say the transformation has a rank of $2$. So the word "rank" means the number of dimensions of the output of a transformation.

Column space

column space of matrix is the set of all possible outputs (ex. line, plane, or 3-D space) for matrix. The columns of matrix tell us where the basis vector land, and the span of those transformed basis vectors gives us all possible outputs. In other words, the column space is the span of the columns of matrix. So, a more precise definition of rank would be "the number of dimensions in the column space". When rank equals the number of columns, we call the matrix "full rank".

Null space

If matrices that aren't full rank, squish to a smaller dimension, here are bunch of vectors that land on zero. This set of vectors that lands on the origin is called the "null space" or the "kernel" of matrix. When $\mathbf{\vec{v}}$ is a zero vector in system of linear equations, the null space represents all of the possible solutions.

---

Summary

In this section, we overview of "system of linear equations" at very high-level. 1). Each "system of linear equations" matches with some kind of linear transformation. And when transformation has an inverse, we can solve equations by that inverse. Otherwise, 2). the concept of column space helps us to understand whether a solution exists or not. and 3). the concept of null space helps us to understand what the set of possible solutions is.