← Back to Math Roadmap

Linear Algebra

Vectors, matrices, transformations, and eigenvalues — the mathematics of multi-dimensional spaces.

Vectors: The Atoms of Linear Algebra

▼
A vector is an ordered list of numbers capturing both magnitude and direction. In 2D, a pair (x,y). In 3D, a triple (x,y,z). Vectors add component-wise — geometrically, the parallelogram law. The dot product multiplies corresponding components and sums them, producing a scalar. Geometrically, it equals the product of lengths times the cosine of the angle between them — it measures how much two vectors point the same way. Two vectors are orthogonal (perpendicular) if their dot product is zero. Vectors generalize to any dimension n, and in machine learning they can reach millions of dimensions. A basis is a set of linearly independent vectors that span the space — every vector can be uniquely written as a linear combination of basis vectors.
Dot product: algebraic (sum of products) equals geometric (lengths cosine of angle). Zero = orthogonal. Positive = acute angle. Negative = obtuse.

Matrices: Packaging Transformations

▼
A matrix is a rectangular array of numbers encoding a linear transformation — a rule mapping input vectors to output vectors. Key transformations: rotation (spin without changing shape), scaling (stretch along axes), reflection (mirror), shear (tilt), and projection (flatten). Matrix multiplication AB composes transformations B then A. Crucially, AB ≠ BA in general: rotate-then-scale is different from scale-then-rotate. The identity matrix I (1s on diagonal, 0s elsewhere) is the do-nothing transformation. The inverse A^{-1} undoes A. A matrix is invertible (non-singular) if its determinant is not zero. The transpose A^T flips rows and columns. A matrix is symmetric if A = A^T. This is tensor theory at rank 2.

Matrix Transform Visualizer

▼
Adjust the 4 matrix entries. The dashed square is the original unit square. The blue parallelogram shows how the matrix transforms it. The red arrow is where (1,0) maps to (column 1). The green arrow is where (0,1) maps to (column 2). The determinant is the area of the parallelogram.
Solving linear systems. If A is invertible, the unique solution is A^{-1}b. Otherwise, zero or infinitely many solutions exist.

Eigenvalues and Eigenvectors: The Soul of a Matrix

▼
For a square matrix A, an eigenvector v is a special direction where applying A simply scales v by a factor λ. The equation Av = λv is elegantly simple but immensely deep. For an n×n matrix, there are n eigenvalues (counting multiplicities), found by solving det(A - λI) = 0. The trace tr(A) equals the sum of eigenvalues. The determinant det(A) equals the product of eigenvalues. A symmetric matrix always has real eigenvalues and orthogonal eigenvectors — a mathematical miracle. This powers PCA, quantum mechanics, and vibration analysis. The Singular Value Decomposition extends eigendecomposition to any matrix, rectangular or singular.
Eigenvalue equation. The transformation A only stretches eigenvector v by factor λ. The direction is preserved; only the length changes.

The Four Fundamental Subspaces of a Matrix

▼
  • Column space (range): all possible outputs Ax. Dimension = rank. Contains all vectors the matrix can produce.
  • Nullspace (kernel): all x where Ax = 0. Dimension = n minus rank. The directions the matrix annihilates.
  • Row space: column space of A^T. Also dimension = rank. Orthogonal complement of the nullspace.
  • Left nullspace: nullspace of A^T. Dimension = m minus rank. Orthogonal complement of the column space.

Applications Where Linear Algebra Rules

▼
Computer Graphics
Every 3D game engine multiplies millions of 4×4 matrices per frame for the vertex transformation pipeline: model → world → view → projection. Translation, rotation, scaling — all are matrix operations on homogeneous coordinates.
Machine Learning
Neural networks are nested matrix multiplications with nonlinearities: a^{(l+1)} = σ(W^{(l)}a^{(l)} + b^{(l)}). Backpropagation is the chain rule applied to these matrix operations. Weight matrices transform activations from layer to layer.
Google PageRank
The web is a giant adjacency matrix. PageRank is the principal eigenvector — the stationary distribution of a random web surfer — revealing the most important pages. This eigenvector centrality measures influence in any network.
Quantum Mechanics
States are vectors in Hilbert space. Observables are Hermitian matrices (H = H^†). Measurement probabilities come from inner products. Schrödinger's equation is essentially a matrix ODE: iħ d/dt |ψ⟩ = H|ψ⟩.

Worked Example: Eigenvalues

▼
Worked Example: Eigenvalues
Matrix A = [[3,1],[0,2]].
Characteristic equation: det(A-λI) = (3-λ)(2-λ) - 0 = 0 → λ = 3, 2.
For λ=3: (A-3I)v=0 → [0,1;0,-1][x;y]=0. So y=0, x free. v₁=[1,0].
For λ=2: [1,1;0,0][x;y]=0. So x+y=0. v₂=[1,-1].
det(A)=3×2=6. tr(A)=3+2=5. ✓ Eigenvalues sum to trace, multiply to determinant.

Essential Matrix Facts

▼
Essential Matrix Facts
det(AB)=det(A)det(B). tr(A)=sum of eigenvalues. An orthogonal matrix Q satisfies Q^T Q = I (preserves lengths and angles). A symmetric matrix always has real eigenvalues and orthogonal eigenvectors. The SVD decomposes any matrix into rotation-scaling-rotation: the ultimate matrix factorization.