The first step is to create an augmented matrix having a column of zeros.

\(\displaystyle \left[\begin{matrix} 1 & 1& 1& -1 \\ 2 & 4&5 &6 \\ 3& 9& 5& 4 \end{matrix}\right|\left.\begin{matrix} 0\\ 0\\ 0 \end{matrix}\right]\)

The next step is to get this into RREF.

\(\displaystyle 2R_1-R_2\rightarrow R_2\)

\(\displaystyle \left[\begin{matrix} 1 & 1& 1& -1 \\ 0 & -2&-3 &-8 \\ 3& 9& 5& 4 \end{matrix}\right|\left.\begin{matrix} 0\\ 0\\ 0 \end{matrix}\right]\)

\(\displaystyle 3R_1-R_3\rightarrow R_3\)

\(\displaystyle \left[\begin{matrix} 1 & 1& 1& -1 \\ 0 & -2&-3 &-8 \\ 0& -6& -2& -7 \end{matrix}\right|\left.\begin{matrix} 0\\ 0\\ 0 \end{matrix}\right]\)

\(\displaystyle -3R_2+R_3\rightarrow R_3\)

\(\displaystyle \left[\begin{matrix} 1 & 1& 1& -1 \\ 0 & -2&-3 &-8 \\ 0& 0& 7& 17 \end{matrix}\right|\left.\begin{matrix} 0\\ 0\\ 0 \end{matrix}\right]\)

\(\displaystyle \frac{1}{2}R_2+R_1\rightarrow R_1\)

\(\displaystyle \left[\begin{matrix} 1 & 0& -\frac{1}{2}& -5 \\ 0 & -2&-3 &-8 \\ 0& 0& 7& 17 \end{matrix}\right|\left.\begin{matrix} 0\\ 0\\ 0 \end{matrix}\right]\)

We can simplify to

\(\displaystyle \left[\begin{matrix} 1 & 0& -\frac{1}{2}& -5 \\ 0 & 1&\frac{3}{2} &4 \\ 0& 0& 1& \frac{17}{7} \end{matrix}\right|\left.\begin{matrix} 0\\ 0\\ 0 \end{matrix}\right]\)

\(\displaystyle -\frac{3}{2}R_3+R_2\rightarrow R_2\)

\(\displaystyle \left[\begin{matrix} 1 & 0& -\frac{1}{2}& -5 \\ 0 & 1&0 & \ \frac{5}{14} \\ 0& 0& 1& \frac{17}{7} \end{matrix}\right|\left.\begin{matrix} 0\\ 0\\ 0 \end{matrix}\right]\)

\(\displaystyle \frac{1}{2}R_3+R_1\rightarrow R_1\)

\(\displaystyle \left[\begin{matrix} 1 & 0& 0& -\frac{53}{14} \\ 0 & 1&0 & \ \frac{5}{14} \\ 0& 0& 1& \frac{17}{7} \end{matrix}\right|\left.\begin{matrix} 0\\ 0\\ 0 \end{matrix}\right]\)

This tells us the following.

\(\displaystyle \\x_1-\frac{53}{14}x_4=0 \\ \\ x_2+\frac{5}{14}x_4=0 \\ \\ x_3+\frac{17}{7}x_4=0\)

\(\displaystyle \\x_1=\frac{53}{14}x_4 \\ \\ x_2=-\frac{5}{14}x_4 \\ \\ x_3=-\frac{17}{7}x_4\)

Now we need to write this as a linear combination.

\(\displaystyle \begin{bmatrix} x_1\\ x_2\\ x_3\\ x_4 \end{bmatrix} =x_4\begin{bmatrix} \frac{53}{14}\\ \\ -\frac{5}{14}\\ \\ -\frac{17}{7} \\ 1 \end{bmatrix}\)

The null space is then

\(\displaystyle N(A)=Span\begin{bmatrix} \frac{53}{14}\\ \\ -\frac{5}{14}\\ \\ -\frac{17}{7} \\ 1 \end{bmatrix}\)

We can find a basis for \(\displaystyle A\)'s range space first by finding a basis for the column space of its reduced row echelon form.

Using a calculator or row reduction, we obtain 

\(\displaystyle \begin{bmatrix} 1& 0& 0&-3 \\ 0& 1&0 & 2\\ 0&0 &1 & 1 \end{bmatrix}\)

for the reduced row echelon form.

The fourth column in this matrix can be seen by inspection to be a linear combination of the other three columns, so it is not included in our basis. Hence the first three columns form a basis for the column space of the reduced row echelon form of \(\displaystyle A\), and therefore the first three columns of \(\displaystyle A\) form a basis for its range space.

\(\displaystyle \begin{Bmatrix} \begin{bmatrix} 1\\ -1\\ 0\\ \end{bmatrix} _, \begin{bmatrix} 2\\ -3\\ 1\\ \end{bmatrix} _,\begin{bmatrix} 3\\ -2\\ 1\\ \end{bmatrix} \end{Bmatrix}\).

We can find a basis for \(\displaystyle A\)'s range space first by finding a basis for the column space of its reduced row echelon form.

Using a calculator or row reduction, we obtain 

\(\displaystyle R=\begin{bmatrix} 1& 2& 0 \\ 0& 0&1\\ 0&0&0 \end{bmatrix}\)

for the reduced row echelon form.

The second column in this matrix can be seen by inspection to be a linear combination of the first column, so it is not included in our basis for \(\displaystyle R\). Hence the first and the third columns form a basis for the column space of \(\displaystyle R\), and therefore the first and the third columns of \(\displaystyle A\) form a basis for the range space of \(\displaystyle A\).

\(\displaystyle \begin{Bmatrix} \begin{bmatrix} 1\\ 0\\ 0\\ \end{bmatrix} _, \begin{bmatrix} 3\\ 2\\ -1\\ \end{bmatrix}\end{Bmatrix}\)

We can find a basis for \(\displaystyle A\)'s range space first by finding a basis for the column space of its reduced row echelon form.

Using a calculator or row reduction, we obtain 

\(\displaystyle R=\begin{bmatrix} 1& 0& 0&1\\ 0& 1&0&-1\\ 0&0&1&1\\0&0&0&0 \end{bmatrix}\)

for the reduced row echelon form.

The fourth column in this matrix can be seen by inspection to be a linear combination of the first three columns, so it is not included in our basis for \(\displaystyle R\). Hence the first three columns form a basis for the column space of \(\displaystyle R\), and therefore the first three columns of \(\displaystyle A\) form a basis for the range space of \(\displaystyle A\).

\(\displaystyle \begin{Bmatrix} \begin{bmatrix} 0\\ 0\\ 1\\ 1\\ \end{bmatrix}_,\begin{bmatrix} 1\\ 0\\ 0\\ 1\\ \end{bmatrix}_, \begin{bmatrix} 1\\ 1\\ 0\\ 0\\ \end{bmatrix} \end{Bmatrix}\)

The null space of the matrix \(\displaystyle A\) is the set of solutions to the equation

\(\displaystyle \begin{bmatrix} 5 &2\\ 1& 0 \end{bmatrix} \begin{bmatrix} x_1\\x_2\end{bmatrix} = \begin{bmatrix} 0\\0 \end{bmatrix}\).

We can solve the above system by row reducing \(\displaystyle A\) using either row reduction, or a calculator to find its reduced row echelon form. After that, our system becomes

\(\displaystyle \begin{bmatrix} 1 &0\\0 & 1 \end{bmatrix}\begin{bmatrix} x_1\\x_2 \end{bmatrix}= \begin{bmatrix} 0\\0 \end{bmatrix} \Longrightarrow \begin{align*} x_1 &=0 \\x_2 &=0 \end{align*} \Longrightarrow\)

\(\displaystyle X= \begin{bmatrix} 0\\0 \end{bmatrix}\)

Hence a basis for the null space is just the zero vector;

\(\displaystyle \begin{Bmatrix} \begin{bmatrix} 0\\ 0\\ \end{bmatrix} \end{Bmatrix}\).

The null space of the operator \(\displaystyle T_M\) is the set of solutions to the equation

\(\displaystyle \begin{bmatrix} 1 &1 & 0\\ 0& -1& 1\\ 9& 0& 1\end{bmatrix} \begin{bmatrix} x_1\\x_2\\x_3 \end{bmatrix} = \begin{bmatrix} 0\\0 \\0 \end{bmatrix}\).

We can solve the above system by row reducing our \(\displaystyle 3 \times 3\) matrix \(\displaystyle M\) using either row reduction, or a calculator to find its reduced row echelon form. After that, our system becomes

\(\displaystyle \begin{bmatrix} 1 &0 & 0\\0 & 1& 0\\ 0& 0&1 \end{bmatrix}\begin{bmatrix} x_1\\x_2 \\x_3 \end{bmatrix}= \begin{bmatrix} 0\\0 \\0 \end{bmatrix} \Longrightarrow \begin{align*} x_1 &=0 \\x_2 &=0 \\ x_3&=0 \end{align*}\)

Hence the null space consists of only the zero vector. \(\displaystyle \begin{Bmatrix} \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix} \end{Bmatrix}\)

The null space of the matrix \(\displaystyle A\) is the set of solutions to the equation

\(\displaystyle \begin{bmatrix} 2 &0 & 0\\ 1& 0& 0\\ 9& 8& 2\end{bmatrix} \begin{bmatrix} x_1\\x_2\\x_3 \end{bmatrix} = \begin{bmatrix} 0\\0 \\0 \end{bmatrix}\).

We can solve the above system by row reducing \(\displaystyle A\) using either row reduction, or a calculator to find its reduced row echelon form. After that, our system becomes

\(\displaystyle \begin{bmatrix} 1 &0 & 0\\0 & 1& 1/4\\ 0& 0&0 \end{bmatrix}\begin{bmatrix} x_1\\x_2 \\x_3 \end{bmatrix}= \begin{bmatrix} 0\\0 \\0 \end{bmatrix} \Longrightarrow \begin{align*} x_1 &=0 \\x_2 &=-(1/4) x_3 \\ x_3&=x_3 \end{align*} \Longrightarrow\)

\(\displaystyle X= \begin{bmatrix} 0\\-1/4 \\1 \end{bmatrix}t\)

Multiplying this vector by \(\displaystyle 4\) gets rid of the fraction, and does not affect our answer, since there is an arbitrary constant behind it.

Hence the null space consists of all vectors spanned by \(\displaystyle \begin{bmatrix} 0\\-1 \\4 \end{bmatrix}\);

\(\displaystyle Span\begin{Bmatrix} \begin{bmatrix} 0\\ -1\\4 \end{bmatrix} \end{Bmatrix}\).

The rank is equal to the dimension of the row space and the column space (both spaces always have the same dimension). This matrix has three rows and five columns, which means the largest possible number of vectors in a basis for the row space of a \(\displaystyle 3 \times 5\) matrix is \(\displaystyle 3\), so this is the largest possible rank.

According to the Rank + Nullity Theorem, 

\(\displaystyle Rank(A)+Nullity(A)= columns\)

Since the matrix has \(\displaystyle 5\) columns, we can rearrange the equation to get

\(\displaystyle Nullity(A) = 5- Rank(A)\)

So to make the nullity as small as possible, we need to make the rank as large as possible.

The rank is equal to the dimension of the row space and the column space (both spaces always have the same dimension). This matrix has three rows and five columns, which means the largest possible number of vectors in a basis for the row space of a \(\displaystyle 3 \times 5\) matrix is \(\displaystyle 3\), so this is the largest possible rank.

Hence the smallest possible nullity is \(\displaystyle 5-(3) =2\).