← Back to Math Roadmap

Numerical Methods

Computational algorithms for solving mathematical problems — when exact solutions are impossible.

Why Numerical Methods Exist

▼
Most real-world mathematical problems cannot be solved exactly. Numerical methods provide approximate solutions using iteration, discretization, and clever algorithms. They trade exactness for feasibility. Root finding algorithms locate where a function equals zero. Numerical integration approximates definite integrals. Numerical ODE solvers compute solutions step by step. The field sits at the intersection of mathematics and computer science — it requires understanding both the underlying math and the computational limitations of floating-point arithmetic.
Newton-Raphson method. Start with a guess, follow the tangent line to the x-axis. Converges quadratically when close to the root.

ODE Solvers: Runge-Kutta Methods

▼
The most widely used algorithm for solving ODEs numerically is the Runge-Kutta Runge-Kutta (used in digital PID) family, especially the fourth-order method known as RK4. It computes four intermediate slope estimates at the beginning, midpoint, and end of each step, then combines them with weighted averaging to produce an estimate accurate to the fourth power of the step size. Adaptive step size methods like Runge-Kutta-Fehlberg (RK45) automatically adjust the step size to control error — taking smaller steps where the solution changes rapidly and larger steps on smooth regions. For stiff equations (where components evolve at vastly different rates), implicit methods are essential because explicit methods become unstable.
Fourth-order Runge-Kutta (RK4). Four slope estimates (k1 through k4) are weighted to produce an O(h⁴) accurate step.

Stability and Error Analysis

▼
Numerical methods introduce two types of error. Truncation error is the error from approximating a continuous process with discrete steps — it decreases as the step size gets smaller. Round-off error comes from finite-precision arithmetic in computers — it increases as you do more steps because each operation adds a tiny rounding error. The art of numerical analysis is finding the sweet spot between these competing sources of error. The condition number of a problem measures how sensitive the output is to small changes in the input. An ill-conditioned problem amplifies small errors into large output errors, regardless of the numerical method used.

Key Methods by Problem Type

▼
Key Methods by Problem Type
Root finding: Newton-Raphson (fast, needs derivative), Secant (no derivative needed), Bisection (guaranteed but slow). Linear systems: Gaussian elimination for small systems, iterative methods (Jacobi, Gauss-Seidel, CG) for large sparse systems. Optimization: Gradient descent, Newton, quasi-Newton (BFGS). Integration: Simpson's rule, Gaussian quadrature, adaptive quadrature. ODEs: Euler (simple, inaccurate), RK4 (workhorse), implicit methods for stiff problems.

Worked Example

▼
Worked Example
Use Newton's method to find the square root of 2.
Let f(x) = x² − 2, so f'(x) = 2x. Start with x0 = 1.
x1 = 1 − (1−2)/(2) = 1.5.
x2 = 1.5 − (2.25−2)/(3) = 1.5 − 0.0833... = 1.4167.
x3 = 1.4167 − (0.0069)/(2.833) = 1.4142. Already accurate to 4 decimal places!

Newton's Method Visualizer

▼
Watch Newton's method converge to a root. Each iteration follows the tangent line to where it crosses the x-axis.