The transpose of a column matrix is the row matrix with the same entries, so

$\displaystyle A = \begin{bmatrix} \cos \frac{4\pi }{13}\\ \\i \sin \frac{4\pi }{13}\end{bmatrix}$, so $\displaystyle A^{T} = \begin{bmatrix} \cos \frac{4\pi }{13} & i \sin \frac{4\pi }{13}\end{bmatrix}$

Find $\displaystyle A^{T}A$ by simply adding the products of corresponding entries:

$\displaystyle A^{T}A= \begin{bmatrix} \cos \frac{4\pi }{13} & i \sin \frac{4\pi }{13}\end{bmatrix} \begin{bmatrix} \cos \frac{4\pi }{13}\\ \\i \sin \frac{4\pi }{13}\end{bmatrix}$

$\displaystyle \begin{bmatrix} \cos \frac{4\pi }{13} \cdot \cos \frac{4\pi }{13} + i \sin \frac{4\pi }{13} \cdot i \sin \frac{4\pi }{13} \end{bmatrix}$

So 

$\displaystyle a= \cos \frac{4\pi }{13} \cdot \cos \frac{4\pi }{13} + i \sin \frac{4\pi }{13} \cdot i \sin \frac{4\pi }{13}$

$\displaystyle = \cos ^{2}\frac{4\pi }{13} + i^{2} \sin^{2} \frac{4\pi }{13}$

$\displaystyle = \cos ^{2}\frac{4\pi }{13} + (-1) \sin^{2} \frac{4\pi }{13}$

$\displaystyle = \cos ^{2}\frac{4\pi }{13} -\sin^{2} \frac{4\pi }{13}$

Apply the trigonometric identity

$\displaystyle \cos ^{2} \theta -\sin^{2} \theta = \cos 2\theta$, 

setting $\displaystyle \theta =\frac{4\pi }{13}$:

$\displaystyle a= \cos \frac{8 \pi}{13}$

$\displaystyle A$ and $\displaystyle B$, being nonsquare matrices, cannot have inverses, so $\displaystyle AB + B^{-1}A^{-1}$ can be eliminated as a choice.

The product of a $\displaystyle m \times n$ matrix and a $\displaystyle n \times p$ matrix is a $\displaystyle m \times p$ matrix. Therefore:

$\displaystyle AB$ is a $\displaystyle 2 \times 2$ matrix; $\displaystyle (AB)^{T}$ and $\displaystyle (AB)^{-1}$ are also $\displaystyle 2 \times 2$ matrices.

Similarly, 

$\displaystyle BA$, $\displaystyle (BA)^{T}$, and $\displaystyle (BA)^{-1}$ are $\displaystyle 3 \times 3$ matrices.

Matrices of different dimensions cannot be added, so  $\displaystyle (AB )^{-1}+BA$, $\displaystyle AB + (BA)^{T}$, and $\displaystyle (AB )^{T}+ (BA)^{-1}$ can be eliminated.

$\displaystyle B^{T}$ and $\displaystyle A^{T}$ are  $\displaystyle 3 \times 2$ and $\displaystyle 2 \times 3$ matrices, respectively. Therefore, $\displaystyle B^{T}A^{T}$ is a $\displaystyle 3 \times 3$ matrix; $\displaystyle AB + B^{T}A^{T}$, the sum of $\displaystyle 3 \times 3$ matrices, is a defined expression.

The transpose of a column matrix is the row matrix with the same entries, so if

$\displaystyle A = \begin{bmatrix} \sqrt{2 }\cos \frac{6\pi }{11}\\ \\i \end{bmatrix}$,

then

$\displaystyle A^{T} = \begin{bmatrix} \sqrt{2 }\cos\cos \frac{6\pi }{11} & i \end{bmatrix}$

Find $\displaystyle A^{T}A$ by simply adding the products of corresponding entries:

$\displaystyle A^{T} A = \begin{bmatrix} \sqrt{2 }\cos \frac{6\pi }{11} \cdot \sqrt{2 } \cos \frac{6\pi }{11} + i \cdot i \end{bmatrix}$

$\displaystyle a = \sqrt{2 }\cos \frac{6\pi }{11} \cdot \sqrt{2 } \cos \frac{6\pi }{11} + i \cdot i$

$\displaystyle =2 \cos ^{2 } \frac{6\pi }{11} + i ^{2}$

$\displaystyle =2 \cos ^{2 } \frac{6\pi }{11} -1$

Apply the trigonometric identity

$\displaystyle 2 \cos^{2} \theta -1 = \cos 2\theta$

Setting $\displaystyle \theta = \frac{6\pi }{11}$,

$\displaystyle a = \cos \frac{12\pi}{11}$.

$\displaystyle A$ is a diagonal matrix. The product of two diagonal matrices can be found by multiplying elements in corresponding diagonal positions; the idea can be extended to powers of matrices, so 

$\displaystyle A^{6}= \begin{bmatrix} \left (\cos \frac{\pi }{36} + i \sin \frac{\pi }{36} \right ) ^{6}& 0 \\ 0 & \left (\cos \frac{\pi }{18} + i \sin \frac{\pi }{18} \right )^{6} \end{bmatrix}$

By DeMoivre's Theorem,

$\displaystyle \left ( \cos \theta + i \sin \theta \right )^{n} = \cos n \theta + i \sin n \theta$

Set $\displaystyle n = 6$ and $\displaystyle \theta = \frac{\pi }{36 }$ in the upper left element:

$\displaystyle \left (\cos \frac{\pi }{36} + i \sin \frac{\pi }{36} \right ) ^{6}$

$\displaystyle = \cos \left ( 6 \cdot \frac{\pi}{36}\right )+ i \sin\left ( 6 \cdot \frac{\pi}{36}\right )$

$\displaystyle = \cos \frac{\pi}{6} + i \sin \frac{\pi}{6}$

$\displaystyle =\frac{ \sqrt{3}}{2}+\frac{1}{2} i$

Set $\displaystyle n = 6$ and $\displaystyle \theta = \frac{\pi }{18}$ in the lower right element:

$\displaystyle \left (\cos \frac{\pi }{18} + i \sin \frac{\pi }{18} \right ) ^{6}$

$\displaystyle = \cos \left ( 6 \cdot \frac{\pi}{18}\right )+ i \sin\left ( 6 \cdot \frac{\pi}{18}\right )$

$\displaystyle = \cos \frac{\pi}{3} + i \sin \frac{\pi}{3}$

$\displaystyle =\frac{1}{2} +\frac{ \sqrt{3}}{2} i$

Therefore,

$\displaystyle A^{6}= \begin{bmatrix} \frac{ \sqrt{3}}{2}+\frac{1}{2} i& 0 \\ 0 &\frac{1}{2} +\frac{ \sqrt{3}}{2} i \end{bmatrix}$,

which is not among the choices.

An easy way to find this is to note that $\displaystyle A^{4} = (A^{2} )^{2}$; therefore, we can find $\displaystyle A^{4}$ by squaring $\displaystyle A$ and squaring the result. 

Matrix multiplication is worked row by column - each row in the former matrix is multiplied by each column in the latter by adding the products of elements in corresponding positions, as follows:

$\displaystyle A^{2} = \begin{bmatrix} 2 & 3 \\ -2 & 1 \end{bmatrix} \begin{bmatrix} 2 & 3 \\ -2 & 1 \end{bmatrix}$

$\displaystyle = \begin{bmatrix} 2(2)+3(-2) & 2(3)+3(1) \\ -2(2)+1(-2) & -2(3) +1(1)\end{bmatrix}$

$\displaystyle = \begin{bmatrix}-2 & 9 \\-6& -5\end{bmatrix}$

Now square this:

$\displaystyle A^{4} = (A^{2} )^{2}= \begin{bmatrix}-2 & 9 \\-6& -5\end{bmatrix} \begin{bmatrix}-2 & 9 \\-6& -5\end{bmatrix}$

$\displaystyle = \begin{bmatrix}-2(-2) + 9(-6) & -2(9) + 9(-5) \\ -6(-2) + (-5)(-6) & -6(9) + (-5)(-5) \end{bmatrix}$

$\displaystyle = \begin{bmatrix}-50 & -63 \\ 42& -29\end{bmatrix}$

$\displaystyle A$ is a diagonal matrix, so it can be raised to a power by raising the individual entries to that power:

$\displaystyle A^{35} = \begin{bmatrix} \left (\cos \frac{ \pi}{7} - i \sin \frac{ \pi}{7} \right ) ^{35}& 0 \\ 0 & \left (\cos \frac{ \pi}{5} - i \sin \frac{ \pi}{5} \right ) ^{35}\end{bmatrix}$

By DeMoivre's Theorem, 

$\displaystyle (\cos \theta + i \sin \theta )^{n} = \cos n\theta + i \sin n \theta$.

We will need to rewrite the entries in the matrix as sums, not differences. We can do this by noting that the cosine and sine functions are even and odd, respectively, so $\displaystyle A^{35}$ can be rewritten as

$\displaystyle A^{35} = \begin{bmatrix} \left (\cos\left ( -\frac{ \pi}{7} \right )+ i \sin\left ( -\frac{ \pi}{7} \right )\right ) ^{35}& 0 \\ 0 & \left (\cos\left ( -\frac{ \pi}{5} \right )+ i \sin\left ( -\frac{ \pi}{5} \right )\right ) ^{35}\end{bmatrix}$

Applying DeMoivre's Theorem,

$\displaystyle A^{35} = \begin{bmatrix} \cos \left (35\cdot \left ( -\frac{ \pi}{7} \right ) \right )+ i \sin \left (35\cdot \left ( -\frac{ \pi}{7} \right ) \right ) & 0 \\ 0 & \cos \left (35\cdot \left ( -\frac{ \pi}{5} \right ) \right )+ i \sin \left (35\cdot \left ( -\frac{ \pi}{5} \right ) \right ) \end{bmatrix}$

$\displaystyle A^{35} = \begin{bmatrix} \cos \left (-5 \pi \right )+ i \sin \left (-5 \pi \right ) & 0 \\ 0 &\cos \left (-7\pi \right )+ i \sin \left (-7 \pi \right ) \end{bmatrix}$

The coterminal angle for both $\displaystyle -5 \pi$ and $\displaystyle -7\pi$ is $\displaystyle \pi$, so

$\displaystyle A^{35} = \begin{bmatrix} \cos \pi + i \sin \pi & 0 \\ 0 & \cos \pi + i \sin \pi \end{bmatrix}$

$\displaystyle A^{35} = \begin{bmatrix}-1 + i (0)& 0 \\ 0 & -1 + i (0) \end{bmatrix}$

$\displaystyle A^{35} = \begin{bmatrix}-1 & 0 \\ 0 & -1 \end{bmatrix}$

$\displaystyle A$, a column matrix with ten entries, is a $\displaystyle 7 \times 1$ matrix; $\displaystyle A^{T}$, its transpose, is a $\displaystyle 1 \times 7$ matrix. 

If $\displaystyle X$ and $\displaystyle Y$ are $\displaystyle m \times n$ and $\displaystyle n \times p$ matrices, respectively, then the product $\displaystyle XY$ is a $\displaystyle m \times p$ matrix. Therefore, $\displaystyle A^{T}A$ is a $\displaystyle 1 \times 1$ matrix - a matrix with a single entry.

$\displaystyle A$, a column matrix with ten entries, is a $\displaystyle 10 \times 1$ matrix; $\displaystyle A^{T}$, its transpose, is a $\displaystyle 1 \times 10$ matrix. 

If $\displaystyle X$ and $\displaystyle Y$ are $\displaystyle m \times n$ and $\displaystyle n \times p$ matrices, respectively, then the product $\displaystyle XY$ is a $\displaystyle m \times p$ matrix. Therefore, $\displaystyle AA^{T}$ is a $\displaystyle 10 \times 10$ matrix.

$\displaystyle A$ and $\displaystyle B$ are both elementary matrices, in that each can be formed from the (four-by-four) identity matrix $\displaystyle I$  by a single row operation.  

Since $\displaystyle A$ differs from $\displaystyle I$ in that the entry $\displaystyle -3$ is in Row 2, Column 2, the row operation is $\displaystyle -3R2 \rightarrow R2$. Since $\displaystyle B$ differs from $\displaystyle I$ in that entry $\displaystyle -2$ is in Row 3, Column 2, the row operation is $\displaystyle -2R2+R3 \rightarrow R3$.

Premultiplying a matrix by an elementary matrix has the effect of performing that row operation on the matrix. Looking at $\displaystyle ABC$ as $\displaystyle A(BC)$:

$\displaystyle C = \begin{bmatrix} 1 & 1 & -1 & -1 \\ 2 & 1 & 3 & -3 \\ 1 & -1 & -2 & 2 \\ 1 & 2 & 1 & 2 \end{bmatrix}$

Premultiply $\displaystyle C$ by $\displaystyle B$ by performing the operation $\displaystyle -2R2+R3 \rightarrow R3$:

$\displaystyle BC = \begin{bmatrix} 1 & 1 & -1 & -1 \\ 2 & 1 & 3 & -3 \\ -3 & -3 & -8 &8 \\ 1 & 2 & 1 & 2 \end{bmatrix}$

Premultiply $\displaystyle BC$ by $\displaystyle A$ by performing the operation $\displaystyle -3R2 \rightarrow R2$:

$\displaystyle ABC = A (BC )= \begin{bmatrix} 1 & 1 & -1 & -1 \\ -6 & -3 & -9 & 9 \\ -3 & -3 & -8 &8 \\ 1 & 2 & 1 & 2 \end{bmatrix}$

This is the correct product.

All of the matrices are diagonal, so the seventh power of each can be determined by simply taking the seventh power of the individual entries in the main diagonal. Also, note that each entry in each choice is of the form 

$\displaystyle \cos \theta + i \sin \theta$.

By DeMoivre's Theorem, for any real $\displaystyle n$,

$\displaystyle (\cos \theta + i \sin \theta ) ^{n} = \cos n \theta + i \sin n \theta$

Combining these ideas, we can take the seventh power of each matrix and determine which exponentiation yields the identity.

If 

$\displaystyle A = \begin{bmatrix} \cos \frac{5\pi}{14} + i \sin \frac{5 \pi}{14} & 0 \\ 0 & \cos \frac{9\pi}{14} + i \sin \frac{9 \pi }{14} \end{bmatrix}$,

then 

$\displaystyle A^{7} = \begin{bmatrix}( \cos \frac{5\pi}{14} + i \sin \frac{5 \pi}{14} )^{7}& 0 \\ 0 & (\cos \frac{9\pi}{14} + i \sin \frac{9 \pi }{14} )^{7}\end{bmatrix}$

$\displaystyle = \begin{bmatrix}\cos \frac{5\pi}{2} + i \sin \frac{5 \pi}{2} & 0 \\ 0 & \cos \frac{9\pi}{2} + i \sin \frac{9 \pi }{2} \end{bmatrix}$

$\displaystyle = \begin{bmatrix}\cos \frac{\pi}{2} + i \sin \frac{ \pi}{2} & 0 \\ 0 & \cos \frac{\pi}{2} + i \sin \frac{ \pi }{2} \end{bmatrix}$

$\displaystyle = \begin{bmatrix}0 + i (1)& 0 \\ 0 & 0 + i (1) \end{bmatrix}$

$\displaystyle = \begin{bmatrix}i & 0 \\ 0 & i \end{bmatrix}$

$\displaystyle \ne I$

If 

$\displaystyle A = \begin{bmatrix} \cos \frac{\pi}{14} + i \sin \frac{ \pi}{14} & 0 \\ 0 & \cos \frac{13\pi}{14} + i \sin \frac{13 \pi }{14} \end{bmatrix}$,

then 

$\displaystyle A^{7} = \begin{bmatrix}( \cos \frac{\pi}{14} + i \sin \frac{ \pi}{14} )^{7}& 0 \\ 0 &( \cos \frac{13\pi}{14} + i \sin \frac{13 \pi }{14} )^{7}\end{bmatrix}$

$\displaystyle = \begin{bmatrix} \cos \frac{\pi}{2} + i \sin \frac{ \pi}{2} & 0 \\ 0 & \cos \frac{13\pi}{2} + i \sin \frac{13 \pi }{2} \end{bmatrix}$

$\displaystyle = \begin{bmatrix} \cos \frac{\pi}{2} + i \sin \frac{ \pi}{2} & 0 \\ 0 & \cos \frac{ \pi}{2} + i \sin \frac{ \pi }{2} \end{bmatrix}$

$\displaystyle = \begin{bmatrix}0 + i (1)& 0 \\ 0 & 0 + i (1) \end{bmatrix}$

$\displaystyle = \begin{bmatrix}i & 0 \\ 0 & i \end{bmatrix}$

$\displaystyle \ne I$

If 

$\displaystyle A = \begin{bmatrix} \cos \frac{ \pi}{7} + i \sin \frac{ \pi}{7} & 0 \\ 0 & \cos \frac{6\pi}{7} + i \sin \frac{6\pi}{7} \end{bmatrix}$,

then 

$\displaystyle A^{7} = \begin{bmatrix} (\cos \frac{ \pi}{7} + i \sin \frac{ \pi}{7} )^{7} & 0 \\ 0 &( \cos \frac{6\pi}{7} + i \sin \frac{6\pi}{7} )^{7} \end{bmatrix}$

$\displaystyle = \begin{bmatrix} \cos \pi + i \sin \pi & 0 \\ 0 & \cos 6 \pi + i \sin 6 \pi \end{bmatrix}$

$\displaystyle = \begin{bmatrix} \cos \pi + i \sin \pi & 0 \\ 0 & \cos 0 + i \sin 0 \end{bmatrix}$

$\displaystyle = \begin{bmatrix} -1 + i (0)& 0 \\ 0 & 1 + i (0) \end{bmatrix}$

$\displaystyle = \begin{bmatrix} -1 & 0 \\ 0 & 1 \end{bmatrix}$

$\displaystyle \ne I$

If 

$\displaystyle A = \begin{bmatrix} \cos \frac{2\pi}{7} + i \sin \frac{2\pi}{7} & 0 \\ 0 & \cos \frac{4\pi}{7} + i \sin \frac{4\pi}{7} \end{bmatrix}$,

then 

$\displaystyle A^{7} = \begin{bmatrix} (\cos \frac{2\pi}{7} + i \sin \frac{2\pi}{7})^{7} & 0 \\ 0 &( \cos \frac{4\pi}{7} + i \sin \frac{4\pi}{7} )^{7} \end{bmatrix}$

$\displaystyle A^{7} = \begin{bmatrix} \cos 2 \pi + i \sin 2 \pi & 0 \\ 0 & \cos 4\pi + i \sin4\pi \end{bmatrix}$

$\displaystyle = \begin{bmatrix} \cos 0 + i \sin 0 & 0 \\ 0 & \cos 0 + i \sin 0 \end{bmatrix}$

$\displaystyle = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$

$\displaystyle = I$

$\displaystyle \begin{bmatrix} \cos \frac{2\pi}{7} + i \sin \frac{2\pi}{7} & 0 \\ 0 & \cos \frac{4\pi}{7} + i \sin \frac{4\pi}{7} \end{bmatrix}$ is the only possible matrix value of $\displaystyle A$ among the choices.