14) Vectors and Linear Transformations¶

Related references:

Vectors¶

Overview¶

Vectors This and other comics from The Manga Guide to Linear Algebra

Vectors

Linear Dependence and Bases¶

Vectors

Find the constants $c_1, c_2, c_3$, and $c_4$ that satisfy the equation:

\[\begin{split}\left( \begin{array}{ccc} 0 \\ 0 \end{array} \right) = c_1 \left( \begin{array}{ccc} 1 \\ 0 \end{array} \right) + c_2 \left( \begin{array}{ccc} 1 \\ 0 \end{array} \right) + c_3 \left( \begin{array}{ccc} 3 \\ 1 \end{array} \right) + c_4 \left( \begin{array}{ccc} 1 \\ 2 \end{array} \right)\end{split}\]

Vectors

Dimension¶

Vectors

What is a subspace?¶

Let $c$ be an arbitrary real number and $W$ be a nonempty subset of $R^m$ satisfying these two conditions:

An element in $W$ multiplied by $c$ is still an element in $W$. (Closed under scalar multiplication.)

If $:raw-latex:left`( \begin{array}{c} a_{1i} \ a_{2i} \ :raw-latex:vdots \ a_{mi} \end{array} :raw-latex:right`) :raw-latex:`\in `W $, then $c :raw-latex:`left`( \begin{array}{c} a_{1i} \ a_{2i} \ :raw-latex:`\vdots `\\ a_{mi} \\end{array} :raw-latex:`right`) :raw-latex:`in `W $
The sum of two arbitrary elements in W is still an element in W. (Closed under addition.)

If $:raw-latex:left`( \begin{array}{c} a_{1i} \ a_{2i} \ :raw-latex:vdots \ a_{mi} \end{array} :raw-latex:right`) :raw-latex:`\in `W $ and $:raw-latex:`left`( \begin{array}{c} a_{1j} \ a_{2j} \ :raw-latex:`\vdots `\\ a_{mj} \\end{array} :raw-latex:`right`) :raw-latex:`\in `W $, then $ :raw-latex:`left`( \begin{array}{c} a_{1i} \ a_{2i} \ :raw-latex:`\vdots `\\ a_{mi} \\end{array} :raw-latex:`right`) + :raw-latex:`\left`( \\begin{array}{c} a_{1j} \\ a_{2j} \\ :raw-latex:`vdots \ a_{mj} \end{array} :raw-latex:right`) :raw-latex:`in `W $

If both of these conditions hold, then $W$ is a subspace of $R^m$.

Vectors

Another, more concrete way to look at one-dimensional subspaces is as lines through the origin. Two-dimensional subspaces are similarly planes through the origin. Other subspaces can also be visualized, but not as easily.

Vectors

Linear Transformations¶

Vectors

Image¶

The image of $f$ (written Im $f$) is equal to the set of vectors that is made up of all of the possible output values of f, as you can see in the following relation:

Vectors

Rank¶

The number of linearly independent vectors among the columns of the matrix $M$ (which is also the dimension of the $R^m$ subspace Im $f$) is called the rank of $M$, and it is written: rank $M$.

What is the rank of $ :raw-latex:`left`(

\begin{array}{ccc} 3 & 1 \\ 1 & 2 \end{array}

:raw-latex:`\right`) $?

In [1]:

import numpy as np

a = np.array([[3, 1], [1, 2]])
np.linalg.matrix_rank(a)

Out[1]:

Of $ :raw-latex:`left`(

\begin{array}{ccc} 3 & 6 \\ 1 & 2 \end{array}

:raw-latex:`\right`) $?

In [2]:

b = np.array([[3, 6], [1, 2]])
np.linalg.matrix_rank(b)

Out[2]:

Vectors

Back to the cautions about matrix multiplication:¶

Recall that:

$:raw-latex:mathbf{AC} = :raw-latex:`\mathbf{AD}` $ does not necessarily imply that $:raw-latex:mathbf{C} = :raw-latex:`\mathbf{D}` $, even when $:raw-latex:mathbf{A} :raw-latex:`neq `0 $
$:raw-latex:mathbf{AB} = 0 $ does not necessarily imply that $:raw-latex:mathbf{A} = 0 $ or $:raw-latex:mathbf{B} = 0 $ or $:raw-latex:mathbf{BA} = 0 $

We can be specific about when implications are true using the concept of rank.

Let A, B, C be $n \times n$ matrices. Then:

If rank A $= n$ and AB = AC, then B = C.
If rank A $= n$, then AB = 0 implies B = 0.
If AB = 0 but A $:raw-latex:neq `$ **0** and **B** $:raw-latex:neq `$ 0, then rank A $ < n $ and rank B $ < n $.
If A is singular, so are AB and BA.

Solutions of Linear Equations¶

Fundamental theorem for linear systems¶

Existence. A linear system of $m$ equations and $n$ unknowns $x_1, \cdots, x_n$:

\[\begin{split} \begin{array}{ccc} a_{11}x_1 &+& a_{12}x_2 &+& \cdots &+& a_{1n}x_n = b_1 \\ a_{21}x_1 &+& a_{22}x_2 &+& \cdots &+& a_{2n}x_n = b_2 \\ \vdots &+& \vdots &+& \ddots &+& \vdots \\ a_{m1}x_1 &+& a_{m2}x_2 &+& \cdots &+& a_{mn}x_n = b_m \end{array}\end{split}\]

has solutions if and only if the coefficient matrix A and the augmented matrix $\mathbf{\tilde{A}}$:

\[\begin{split} \mathbf{A} = \left [\begin{array}{ccc} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{array} \right] \quad and \quad \mathbf{\tilde{A}} = \left [\begin{array}{ccc} a_{11} & a_{12} & \cdots & a_{1n} & b_1 \\ a_{21} & a_{22} & \cdots & a_{2n} & b_2 \\ \vdots & \vdots & \ddots & \vdots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} & b_m \end{array} \right]\end{split}\]

Uniqueness. The system has precisely one solution if and only if the common rank $r$ of A and $\mathbf{\tilde{A}}$ equals $n$.
Infinitely many solutions. If the common rank $r < n$, the system has infinitely many solutions (underspecified). All of these are obtained by determining $r$ suitable unknowns (whose submatrix of coefficients must have rank $r$) in terms of the remaining $n-r$ unknowns, to which arbitrary values can be assigned.
Gaussian elimination. If solutions exist, they can all be obtained by Gaussian elimination. It may be started without first determining ranks, as it will automatically reveal if solutions exist or not.

Example:¶

Solve the linear system of equations:

\[\begin{split} \begin{array}{ccc} 3x_0 &+& x_1 &=& 9 \\ x_0 &+& 2x_1 &=& 8 \end{array}\end{split}\]

In [3]:

a = np.array([[3, 1], [1, 2]])
b = np.array([[9], [8]])
a_aug = np.hstack([a, b])
print(a_aug)

[[3 1 9]
 [1 2 8]]

In [4]:

print(np.linalg.matrix_rank(a))
print(np.linalg.matrix_rank(a_aug))

2
2

In [5]:

x = np.linalg.solve(a, b)
print(x)

[[ 2.]
 [ 3.]]

In [6]:

# Check solution
np.allclose(np.dot(a, x), b)

Out[6]:

True

The homogeneous linear system¶

The linear system of $m$ equations and $n$ unknowns $x_1, \cdots, x_n$ is called homogeneous if all the $b_j$’s on the right side equal zero. Otherwise, the system is called nonhomogeneous.

The homogenous linear system:

\[\begin{split} \begin{array}{ccc} a_{11}x_1 &+& a_{12}x_2 &+& \cdots &+& a_{1n}x_n = 0 \\ a_{21}x_1 &+& a_{22}x_2 &+& \cdots &+& a_{2n}x_n = 0 \\ \vdots &+& \vdots &+& \ddots &+& \vdots \\ a_{m1}x_1 &+& a_{m2}x_2 &+& \cdots &+& a_{mn}x_n = 0 \end{array}\end{split}\]

always has the trivial solution $x_j = 0$. Nontrivial solutions exist if and only if rank A < n. If rank A < n, these solutions, together with x = 0 form the solution space of the system of equations.

If x$_1$ and x$_2$ are both solution vectors of a homogenous system of equations, then x = $c_1$x$_1$ + $c_2$x$_2$ is a solution vector of the system of equations (this does not hold for nonhomogenous systems).

Homogeneous equations with fewer equations than unknowns always has nontrivial solutions.

The nonhomogeneous linear system¶

If a nonhomogeneous linear system of equations has solutions, then all these solutions are of the form:

\[\mathbf{x} = \mathbf{x}_0 + \mathbf{x}_h\]

where $\mathbf{x}_0$ is any solution of the nonhomogenous system of equations, and $\mathbf{x}_h$ runs through all the solutions of the corresponding homogeneous system.

Cramer’s Theorem¶

If a linear system of $n$ equations and the same number of unknowns $x_1, \cdots, x_n$, has a nonzero coefficient determinant D = det A, then the system has precisely one solution.

If the system is homogeneous and D $\neq$ 0, it has only the trivial solution x = 0. If D = 0, the homogeneous system also has nontrivial solutions.

Next up: Eigenvectors and eigenvalues!