Proof Final

Updated 4 Oct 2026

CSS322 - Scientific Computing: Proofs Cheat Sheet

Final Exam Reference (Lectures 8-17)


Lecture 8: Numerical Integration

Proof: Method of Undetermined Coefficients = Integrating the Interpolant

Claim: If we use the same set of nodes, the method of undetermined coefficients gives the same weights as integrating the interpolant.

Proof:

  • Let w1,…,wnw_1, \ldots, w_n denote the weights obtained by the method of undetermined coefficients
  • Let Qn(f)=∑i=1nwif(xi)Q_n(f) = \sum_{i=1}^n w_i f(x_i) denote the resulting quadrature rule
  • Let pn−1(x)=c1+c2x+…+cnxn−1p_{n-1}(x) = c_1 + c_2x + \ldots + c_nx^{n-1} denote the polynomial of degree n−1n-1 that interpolates ff at x1,…,xnx_1, \ldots, x_n

∫abpn−1(x)dx=∫ab(c1+c2x+⋯+cnxn−1)dx\int_a^b p_{n-1}(x)dx = \int_a^b (c_1 + c_2x + \cdots + c_nx^{n-1})dx

This equals:
c1∫abdx+c2∫abxdx+⋯+cn∫abxn−1dxc_1\int_a^b dx + c_2\int_a^b x dx + \cdots + c_n\int_a^b x^{n-1} dx

By the method of undetermined coefficients, these integrals equal:
c1(w1+⋯+wn)+c2(w1x1+⋯+wnxn)+⋯+cn(w1x1n−1+⋯+wnxnn−1)c_1(w_1 + \cdots + w_n) + c_2(w_1x_1 + \cdots + w_nx_n) + \cdots + c_n(w_1x_1^{n-1} + \cdots + w_nx_n^{n-1})

Rearranging:
w1(c1+c2x1+⋯+cnx1n−1)+w2(c1+c2x2+⋯+cnx2n−1)+⋯w_1(c_1 + c_2x_1 + \cdots + c_nx_1^{n-1}) + w_2(c_1 + c_2x_2 + \cdots + c_nx_2^{n-1}) + \cdots

This simplifies to:
w1pn−1(x1)+w2pn−1(x2)+⋯+wnpn−1(xn)=w1f(x1)+w2f(x2)+⋯+wnf(xn)=Qn(f)w_1p_{n-1}(x_1) + w_2p_{n-1}(x_2) + \cdots + w_np_{n-1}(x_n) = w_1f(x_1) + w_2f(x_2) + \cdots + w_nf(x_n) = Q_n(f)

The last line follows because pn−1p_{n-1} interpolates ff at x1,…,xnx_1, \ldots, x_n, so pn−1(xi)=f(xi)p_{n-1}(x_i) = f(x_i).

This shows that the quadrature rule obtained by the method of undetermined coefficients is exactly the integral of the interpolant. ∎


Proof: Midpoint Rule is Degree 1

Claim: The midpoint rule M(f)=(b−a)f(a+b2)M(f) = (b-a)f\left(\frac{a+b}{2}\right) is of degree 1.

Proof:

  • By construction, midpoint rule is exact for degree 0 (constants)
  • For degree 1, any polynomial has form p(x)=cx+ep(x) = cx + e

Exact integral:
∫ab(cx+e)dx=c[x22]ab+e[x]ab=c2(b2−a2)+e(b−a)\int_a^b (cx + e)dx = c\left[\frac{x^2}{2}\right]_a^b + e[x]_a^b = \frac{c}{2}(b^2 - a^2) + e(b - a)

Midpoint rule gives:
(b−a)p(a+b2)=(b−a)(ca+b2+e)(b - a)p\left(\frac{a + b}{2}\right) = (b - a)\left(c\frac{a + b}{2} + e\right)
=c(b−a)a+b2+e(b−a)=c2(b2−a2)+e(b−a)= c(b-a)\frac{a+b}{2} + e(b-a) = \frac{c}{2}(b^2 - a^2) + e(b - a)

These match! ✓

Counterexample for degree 2:
∫023x2dx=[x3]02=8\int_0^2 3x^2 dx = [x^3]_0^2 = 8
But:
M(f)=2(3)(1)2=6≠8M(f) = 2(3)(1)^2 = 6 \neq 8

Therefore, midpoint rule is NOT exact for all degree 2 polynomials. ∎


Proof: Midpoint is Twice as Accurate as Trapezoid

Setup: Let m=a+b2m = \frac{a + b}{2} (midpoint)

Taylor expansion of f(x)f(x) about mm:
f(x)=f(m)+f′(m)(x−m)+f′′(m)2(x−m)2+f(3)(m)6(x−m)3+f(4)(m)24(x−m)4+…f(x) = f(m) + f'(m)(x - m) + \frac{f''(m)}{2}(x - m)^2 + \frac{f^{(3)}(m)}{6}(x - m)^3 + \frac{f^{(4)}(m)}{24}(x - m)^4 + \ldots

For Midpoint Rule:

Integrate both sides from aa to bb. Odd powers integrate to zero due to symmetry about mm:

∫abf(x)dx=f(m)(b−a)+0+f′′(m)24(b−a)3+0+f(4)(m)1920(b−a)5+…\int_a^b f(x)dx = f(m)(b - a) + 0 + \frac{f''(m)}{24}(b - a)^3 + 0 + \frac{f^{(4)}(m)}{1920}(b - a)^5 + \ldots

Therefore:
I(f)=M(f)+E(f)+F(f)+…I(f) = M(f) + E(f) + F(f) + \ldots

where:
E(f)=f′′(m)24(b−a)3\boxed{E(f) = \frac{f''(m)}{24}(b - a)^3}
F(f)=f(4)(m)1920(b−a)5\boxed{F(f) = \frac{f^{(4)}(m)}{1920}(b - a)^5}

For Trapezoid Rule:

Starting from Taylor expansions at aa and bb, then adding them and multiplying by (b−a)/2(b-a)/2:

T(f)=M(f)+3E(f)+5F(f)+…T(f) = M(f) + 3E(f) + 5F(f) + \ldots

Rearranging:
I(f)=T(f)−2E(f)−4F(f)+…I(f) = T(f) - 2E(f) - 4F(f) + \ldots

Comparison:

  • Midpoint error: E(f)+F(f)+…E(f) + F(f) + \ldots
  • Trapezoid error: 2E(f)+4F(f)+…2E(f) + 4F(f) + \ldots

The trapezoid rule has twice the error of the midpoint rule (when (b−a)(b - a) is not too large and f(4)f^{(4)} is well-behaved). ∎

Error Estimation Formula:

From T(f)=M(f)+3E(f)+5F(f)+…T(f) = M(f) + 3E(f) + 5F(f) + \ldots:

E(f)≈T(f)−M(f)3\boxed{E(f) \approx \frac{T(f) - M(f)}{3}}


Derivation: Simpson's Rule from Midpoint and Trapezoid

Setup: From the expansions:

  • I(f)=M(f)+E(f)+F(f)+…I(f) = M(f) + E(f) + F(f) + \ldots
  • I(f)=T(f)−2E(f)−4F(f)+…I(f) = T(f) - 2E(f) - 4F(f) + \ldots

Multiply midpoint by 2/3:
23I(f)=23M(f)+23E(f)+23F(f)+…\frac{2}{3}I(f) = \frac{2}{3}M(f) + \frac{2}{3}E(f) + \frac{2}{3}F(f) + \ldots

Multiply trapezoid by 1/3:
13I(f)=13T(f)−23E(f)−43F(f)+…\frac{1}{3}I(f) = \frac{1}{3}T(f) - \frac{2}{3}E(f) - \frac{4}{3}F(f) + \ldots

Add them:
I(f)=(23M(f)+13T(f))−23F(f)+…I(f) = \left(\frac{2}{3}M(f) + \frac{1}{3}T(f)\right) - \frac{2}{3}F(f) + \ldots

But:
23M(f)+13T(f)=23(b−a)f(m)+13b−a2(f(a)+f(b))\frac{2}{3}M(f) + \frac{1}{3}T(f) = \frac{2}{3}(b - a)f(m) + \frac{1}{3}\frac{b - a}{2}(f(a) + f(b))
=b−a6(f(a)+4f(a+b2)+f(b))=S(f)= \frac{b - a}{6}\left(f(a) + 4f\left(\frac{a + b}{2}\right) + f(b)\right) = S(f)

Therefore:
I(f)=S(f)−23F(f)+…\boxed{I(f) = S(f) - \frac{2}{3}F(f) + \ldots}

This shows the dominant error term of Simpson's rule is O((b−a)5)O((b-a)^5), making it much more accurate! ∎


Lecture 10: Linear Least Squares

Lemma 1: ATAA^T A is Symmetric Positive Definite

Theorem: Suppose A∈Rm×nA \in \mathbb{R}^{m \times n} has rank nn. Then ATAA^T A is symmetric positive definite.

Proof:

Part 1 - Symmetry:
(ATA)T=AT(AT)T=ATA(A^T A)^T = A^T (A^T)^T = A^T A ✓

Part 2 - Positive Definiteness:

Suppose x∈Rnx \in \mathbb{R}^n is nonzero. Then:
xTATAx=(Ax)TAx=∥Ax∥22≥0x^T A^T Ax = (Ax)^T Ax = \|Ax\|_2^2 \geq 0

But AxAx is a nontrivial linear combination of columns of AA (since x≠0x \neq 0).

Since columns of AA are linearly independent (because rank of AA is nn), we have Ax≠0Ax \neq 0.

Therefore: xTATAx=∥Ax∥22>0x^T A^T Ax = \|Ax\|_2^2 > 0 ✓

This proves ATAA^T A is symmetric positive definite. ∎


Lemma 2: Minimizer of Quadratic Forms

Theorem: Suppose f(x)=xTCx−2hTx+df(x) = x^T Cx - 2h^T x + d, where CC is symmetric positive definite. Then ff has a unique minimizer at x∗=C−1h\boxed{x^* = C^{-1}h}.

Proof:

Step 1: Evaluate f(x∗)f(x^*)

f(x∗)=f(C−1h)=(C−1h)TCC−1h−2hTC−1h+df(x^*) = f(C^{-1}h) = (C^{-1}h)^T CC^{-1}h - 2h^T C^{-1}h + d
=hTC−Th−2hTC−1h+d= h^T C^{-T}h - 2h^T C^{-1}h + d

Since CC is symmetric: C−T=(CT)−1=C−1C^{-T} = (C^T)^{-1} = C^{-1}

Therefore:
f(x∗)=hTC−1h−2hTC−1h+d=−hTC−1h+df(x^*) = h^T C^{-1}h - 2h^T C^{-1}h + d = -h^T C^{-1}h + d

Step 2: Evaluate f(x∗+y)f(x^* + y) for any nonzero vector yy

Let y∈Rny \in \mathbb{R}^n be any arbitrary nonzero vector:

f(x∗+y)=(C−1h+y)TC(C−1h+y)−2hT(C−1h+y)+df(x^* + y) = (C^{-1}h + y)^T C(C^{-1}h + y) - 2h^T(C^{-1}h + y) + d
=(hTC−1+yT)(h+Cy)−2hTC−1h−2hTy+d= (h^T C^{-1} + y^T)(h + Cy) - 2h^T C^{-1}h - 2h^T y + d
=hTC−1h+hTy+yTh+yTCy−2hTC−1h−2hTy+d= h^T C^{-1}h + h^T y + y^T h + y^T Cy - 2h^T C^{-1}h - 2h^T y + d

Since hTy=yThh^T y = y^T h (scalars are symmetric):
f(x∗+y)=−hTC−1h+d+yTCy=f(x∗)+yTCyf(x^* + y) = -h^T C^{-1}h + d + y^T Cy = f(x^*) + y^T Cy

Step 3: Show x∗x^* is the unique minimizer

But CC is symmetric positive definite, so: yTCy>0y^T Cy > 0 for all nonzero yy

Therefore: f(x∗+y)=f(x∗)+yTCy>f(x∗)f(x^* + y) = f(x^*) + y^T Cy > f(x^*) for all nonzero yy

So x∗x^* is the unique minimizer. ∎


Main Theorem: Normal Equations Solution

Theorem:
x=(ATA)−1ATb\boxed{x = (A^T A)^{-1}A^T b}
is the unique solution to the linear least squares problem when AA has rank nn.

Proof: This follows directly from Lemma 1 and Lemma 2 by setting:

  • f(x)=∥Ax−b∥22=xTATAx−2bTAx+bTbf(x) = \|Ax - b\|_2^2 = x^T A^T Ax - 2b^T Ax + b^T b
  • C=ATAC = A^T A (symmetric positive definite by Lemma 1)
  • h=ATbh = A^T b
  • d=bTbd = b^T b

By Lemma 2: x∗=C−1h=(ATA)−1ATbx^* = C^{-1}h = (A^T A)^{-1}A^T b ∎


QR Factorization Theorems

Theorem 1: Any A∈Rm×nA \in \mathbb{R}^{m \times n} (m≥nm \geq n) can be factored A=QRA = QR where QQ is an m×mm \times m orthogonal matrix and RR is an m×nm \times n upper triangular matrix.

Theorem 2: If A∈Rm×nA \in \mathbb{R}^{m \times n}, m≥nm \geq n, has rank nn and A=QRA = QR is the QR factorization of AA, then RR has rank nn and R1R_1 is nonsingular.

Main Result: Let A=QRA = QR be the QR factorization of AA. Assume A∈Rm×nA \in \mathbb{R}^{m \times n}, m≥nm \geq n, rank(A)=n\text{rank}(A) = n.

Let c=QTbc = Q^T b. Let c1c_1 be the top nn entries of cc. Then xx minimizes ∥Ax−b∥2\|Ax - b\|_2 if:
R1x=c1\boxed{R_1 x = c_1}

Proof:
∥Ax−b∥2=∥QRx−b∥2=∥Q(Rx−QTb)∥2=∥Rx−c∥2\|Ax - b\|_2 = \|QRx - b\|_2 = \|Q(Rx - Q^T b)\|_2 = \|Rx - c\|_2
=∥[R10]x−[c1c2]∥2=∥[R1x−c1−c2]∥2= \left\|\begin{bmatrix} R_1 \\ 0 \end{bmatrix} x - \begin{bmatrix} c_1 \\ c_2 \end{bmatrix}\right\|_2 = \left\|\begin{bmatrix} R_1 x - c_1 \\ -c_2 \end{bmatrix}\right\|_2
=∥R1x−c1∥22+∥c2∥22= \sqrt{\|R_1 x - c_1\|_2^2 + \|c_2\|_2^2}

Since c2c_2 is constant, minimizing this is equivalent to minimizing ∥R1x−c1∥2\|R_1 x - c_1\|_2.

If R1R_1 is nonsingular (which follows from Theorem 2), then R1x=c1R_1 x = c_1 has a unique solution that minimizes the norm. ∎


Householder Transformation Properties

Theorem: Let v∈Rmv \in \mathbb{R}^m be a nonzero vector. Then H=I−2vvTvTvH = I - 2\frac{vv^T}{v^T v} is symmetric and orthogonal.

Proof:

Symmetry:
HT=(I−2vvTvTv)T=IT−2(vvT)TvTv=I−2vvTvTv=HH^T = \left(I - 2\frac{vv^T}{v^T v}\right)^T = I^T - 2\frac{(vv^T)^T}{v^T v} = I - 2\frac{vv^T}{v^T v} = H ✓

Orthogonality:
HHT=HH=(I−2vvTvTv)(I−2vvTvTv)HH^T = HH = \left(I - 2\frac{vv^T}{v^T v}\right)\left(I - 2\frac{vv^T}{v^T v}\right)
=I−4vvTvTv+4vvTvvT(vTv)2= I - 4\frac{vv^T}{v^T v} + 4\frac{vv^T vv^T}{(v^T v)^2}
=I−4vvTvTv+4v(vTv)vT(vTv)2= I - 4\frac{vv^T}{v^T v} + 4\frac{v(v^T v)v^T}{(v^T v)^2}
=I−4vvTvTv+4vvTvTv=I= I - 4\frac{vv^T}{v^T v} + 4\frac{vv^T}{v^T v} = I ✓ ∎


Lecture 14: Singular Value Decomposition

Lemma 3: Norm of Diagonal Matrix

Lemma: Let D∈Rm×nD \in \mathbb{R}^{m \times n} be diagonal. Then ∥D∥p=max⁡∣dii∣\|D\|_p = \max |d_{ii}| for any pp.

Proof: [Exercise for student]

Key steps:

  • For any vector xx, DxDx has entries diixid_{ii}x_i
  • ∥Dx∥p≤(max⁡∣dii∣)∥x∥p\|Dx\|_p \leq (\max |d_{ii}|) \|x\|_p
  • Equality achieved when xx is the unit vector in direction of largest ∣dii∣|d_{ii}|
  • Therefore ∥D∥p=max⁡∣dii∣\|D\|_p = \max |d_{ii}| ∎

Theorem: Matrix 2-Norm from SVD

Theorem: If A=UΣVTA = U\Sigma V^T (SVD of AA), then ∥A∥2=σ1\|A\|_2 = \sigma_1 (largest singular value).

Proof:

From Lemma 1 and Lemma 2 (from lecture notes):

  • ∥QA∥2=∥A∥2\|QA\|_2 = \|A\|_2 for orthogonal QQ
  • ∥AQ∥2=∥A∥2\|AQ\|_2 = \|A\|_2 for orthogonal QQ

Therefore:
∥A∥2=∥UΣVT∥2=∥ΣVT∥2=∥Σ∥2\|A\|_2 = \|U\Sigma V^T\|_2 = \|\Sigma V^T\|_2 = \|\Sigma\|_2

By Lemma 3: ∥Σ∥2=σ1\|\Sigma\|_2 = \sigma_1 (largest diagonal entry). ∎


Theorem: Condition Number from SVD

Theorem: If σ1,…,σn\sigma_1, \ldots, \sigma_n are singular values of A∈Rn×nA \in \mathbb{R}^{n \times n} and AA is invertible, then:
∥A−1∥2=1σn\boxed{\|A^{-1}\|_2 = \frac{1}{\sigma_n}}

Proof:

From SVD A=UΣVTA = U\Sigma V^T:
A−1=(UΣVT)−1=V−TΣ−1U−1=VΣ−1UTA^{-1} = (U\Sigma V^T)^{-1} = V^{-T} \Sigma^{-1} U^{-1} = V\Sigma^{-1} U^T

Note that:
Σ−1=[1σ10⋯001σ2⋯0⋮⋮⋱⋮00⋯1σn]\Sigma^{-1} = \begin{bmatrix} \frac{1}{\sigma_1} & 0 & \cdots & 0 \\ 0 & \frac{1}{\sigma_2} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \frac{1}{\sigma_n} \end{bmatrix}

However, this is not an SVD of A−1A^{-1} because diagonal entries are not in non-increasing order.

Consider permutation matrix P=[1⋱1]P = \begin{bmatrix} & & 1 \\ & \ddots & \\ 1 & & \end{bmatrix}

Then:
PΣ−1P=[1σn⋯0⋮⋱⋮0⋯1σ1]P\Sigma^{-1} P = \begin{bmatrix} \frac{1}{\sigma_n} & \cdots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \cdots & \frac{1}{\sigma_1} \end{bmatrix}

has diagonal entries in non-increasing order.

Since PP=IPP = I, VPVP is orthogonal, and PUTPU^T is orthogonal:
A−1=VΣ−1UT=(VP)(PΣ−1P)(PUT)A^{-1} = V\Sigma^{-1} U^T = (VP)(P\Sigma^{-1} P)(PU^T)

is the SVD of A−1A^{-1}.

Thus: ∥A−1∥2=1σn\|A^{-1}\|_2 = \frac{1}{\sigma_n} (largest singular value of A−1A^{-1}). ∎

Corollary:
cond2(A)=∥A∥2∥A−1∥2=σ1σn\boxed{\text{cond}_2(A) = \|A\|_2 \|A^{-1}\|_2 = \frac{\sigma_1}{\sigma_n}}


Theorem: Condition Number of ATAA^T A

Theorem: Suppose A∈Rm×nA \in \mathbb{R}^{m \times n} with rank(A)=n\text{rank}(A) = n. Then:
cond2(ATA)=cond2(A)2\boxed{\text{cond}_2(A^T A) = \text{cond}_2(A)^2}

Proof: [Exercise for student]

Key steps:

  • If A=UΣVTA = U\Sigma V^T, then ATA=VΣTUTUΣVT=VΣTΣVTA^T A = V\Sigma^T U^T U\Sigma V^T = V\Sigma^T\Sigma V^T
  • ΣTΣ\Sigma^T\Sigma has diagonal entries σi2\sigma_i^2
  • cond2(ATA)=σ12σn2=(σ1σn)2=cond2(A)2\text{cond}_2(A^T A) = \frac{\sigma_1^2}{\sigma_n^2} = \left(\frac{\sigma_1}{\sigma_n}\right)^2 = \text{cond}_2(A)^2 ∎

Theorem: Rank Determination

Theorem: If A=UΣVTA = U\Sigma V^T, then:
rank(A)=number of nonzero singular values\boxed{\text{rank}(A) = \text{number of nonzero singular values}}

Proof: [Standard linear algebra result - relies on orthogonal transformations preserving rank]


Lecture 15: Eigenvalues and Eigenvectors

Theorem: Eigenvalues and Eigenvectors of Inverse Matrix

Theorem: If A∈Rn×nA \in \mathbb{R}^{n \times n} is invertible, then AA and A−1A^{-1} have the same eigenvectors, and if λ\lambda is an eigenvalue of AA, then 1/λ1/\lambda is an eigenvalue of A−1A^{-1}.

Proof:

Suppose (λ,x)(\lambda, x) is an eigenpair of AA:
Ax=λxAx = \lambda x

Multiply both sides by A−1A^{-1}:
A−1(Ax)=A−1(λx)A^{-1}(Ax) = A^{-1}(\lambda x)
x=λA−1xx = \lambda A^{-1}x

Divide both sides by λ\lambda (assuming λ≠0\lambda \neq 0):
(1λ)x=A−1x\left(\frac{1}{\lambda}\right)x = A^{-1}x

Therefore, (1/λ,x)(1/\lambda, x) is an eigenpair of A−1A^{-1}. ∎


Theorem: Shifted Matrix Properties

Theorem: Suppose (A−σI)(A - \sigma I) is invertible. Then:

  1. AA and (A−σI)−1(A - \sigma I)^{-1} have the same eigenvectors
  2. λ\lambda is an eigenvalue of AA if and only if 1λ−σ\frac{1}{\lambda - \sigma} is an eigenvalue of (A−σI)−1(A - \sigma I)^{-1}

Proof:

Suppose (λ,x)(\lambda, x) is an eigenpair of AA:
Ax=λxAx = \lambda x

Subtract σx\sigma x from both sides:
(A−σI)x=(λ−σ)x(A - \sigma I)x = (\lambda - \sigma)x

Multiply both sides by (A−σI)−1(A - \sigma I)^{-1}:
x=(λ−σ)(A−σI)−1xx = (\lambda - \sigma)(A - \sigma I)^{-1}x

Divide by (λ−σ)(\lambda - \sigma) (assuming λ≠σ\lambda \neq \sigma):
(1λ−σ)x=(A−σI)−1x\left(\frac{1}{\lambda - \sigma}\right)x = (A - \sigma I)^{-1}x

Therefore, (1/(λ−σ),x)(1/(\lambda - \sigma), x) is an eigenpair of (A−σI)−1(A - \sigma I)^{-1}. ∎


Lecture 13: Initial Value Problems for ODEs

Theorem: Symmetry of Mixed Partials

Theorem: If both ∂2f∂xj∂xi\frac{\partial^2 f}{\partial x_j \partial x_i} and ∂2f∂xi∂xj\frac{\partial^2 f}{\partial x_i \partial x_j} are well-defined and continuous, then:
∂2f∂xj∂xi=∂2f∂xi∂xj\boxed{\frac{\partial^2 f}{\partial x_j \partial x_i} = \frac{\partial^2 f}{\partial x_i \partial x_j}}

Proof: [Standard multivariable calculus result - Schwarz's theorem/Clairaut's theorem]


Lecture 16: Boundary Value Problems

Integration by Parts Formula

Formula:
∫abv′′(t)ϕ(t) dt=v′(t)ϕ(t)∣ab−∫abv′(t)ϕ′(t) dt\boxed{\int_a^b v''(t) \phi(t) \, dt = v'(t)\phi(t)\Big|_a^b - \int_a^b v'(t) \phi'(t) \, dt}

When ϕ(a)=ϕ(b)=0\phi(a) = \phi(b) = 0:
∫abv′′(t)ϕ(t) dt=−∫abv′(t)ϕ′(t) dt\boxed{\int_a^b v''(t) \phi(t) \, dt = -\int_a^b v'(t) \phi'(t) \, dt}

Proof:

Standard integration by parts with u=ϕ(t)u = \phi(t), dv=v′′(t)dtdv = v''(t)dt:
∫abv′′(t)ϕ(t) dt=v′(t)ϕ(t)∣ab−∫abv′(t)ϕ′(t) dt\int_a^b v''(t) \phi(t) \, dt = v'(t)\phi(t)\Big|_a^b - \int_a^b v'(t) \phi'(t) \, dt

If ϕ(a)=ϕ(b)=0\phi(a) = \phi(b) = 0:
v′(b)ϕ(b)−v′(a)ϕ(a)=0v'(b)\phi(b) - v'(a)\phi(a) = 0

Therefore:
∫abv′′(t)ϕ(t) dt=−∫abv′(t)ϕ′(t) dt\int_a^b v''(t) \phi(t) \, dt = -\int_a^b v'(t) \phi'(t) \, dt ∎


Important Relationships and Formulas

Finite Difference Approximations

First derivative (forward): O(h)O(h)
u′(t)≈u(t+h)−u(t)h\boxed{u'(t) \approx \frac{u(t+h) - u(t)}{h}}

First derivative (backward): O(h)O(h)
u′(t)≈u(t)−u(t−h)h\boxed{u'(t) \approx \frac{u(t) - u(t-h)}{h}}

First derivative (centered): O(h2)O(h^2)
u′(t)≈u(t+h)−u(t−h)2h\boxed{u'(t) \approx \frac{u(t+h) - u(t-h)}{2h}}

Second derivative (centered): O(h2)O(h^2)
u′′(t)≈u(t+h)−2u(t)+u(t−h)h2\boxed{u''(t) \approx \frac{u(t+h) - 2u(t) + u(t-h)}{h^2}}


Richardson Extrapolation

Formula:
a0=F(h)+F(h)−F(h/q)q−p−1\boxed{a_0 = F(h) + \frac{F(h) - F(h/q)}{q^{-p} - 1}}

where:

  • F(h)F(h) is the approximation with step size hh
  • F(h/q)F(h/q) is the approximation with step size h/qh/q
  • pp is the order of the method
  • The extrapolated value has error O(hr)O(h^r) where r>pr > p

Power Method Convergence

For AA with eigenvalues λ1,…,λn\lambda_1, \ldots, \lambda_n where ∣λ1∣>∣λj∣|\lambda_1| > |\lambda_j| for all j>1j > 1:

If x(0)=∑j=1nαjvjx^{(0)} = \sum_{j=1}^{n} \alpha_j v_j (linear combination of eigenvectors), then:

x(k)=∑j=1nλjkαjvj=λ1k(α1v1+∑j=2n(λjλ1)kαjvj)\boxed{x^{(k)} = \sum_{j=1}^{n} \lambda_j^k \alpha_j v_j = \lambda_1^k \left(\alpha_1 v_1 + \sum_{j=2}^{n} \left(\frac{\lambda_j}{\lambda_1}\right)^k \alpha_j v_j\right)}

As k→∞k \to \infty:
x(k)→λ1kα1v1\boxed{x^{(k)} \to \lambda_1^k \alpha_1 v_1} (multiple of dominant eigenvector)


Rayleigh Quotient

λ=(x(k))TAx(k)(x(k))Tx(k)\boxed{\lambda = \frac{(x^{(k)})^T Ax^{(k)}}{(x^{(k)})^T x^{(k)}}}

(Denominator may be omitted if x(k)x^{(k)} is normalized)


Error Analysis for ODEs

Local Truncation Error: lk=yk−y(tk)l_k = y_k - y(t_k) assuming yi=y(ti)y_i = y(t_i) for i<ki < k

Global Truncation Error: ek=yk−y(tk)e_k = y_k - y(t_k) (accumulated error)

Relationship: For a wide class of ODEs and methods:

  • If local truncation error is O(hp+1)O(h^{p+1})
  • Then global truncation error is O(hp)O(h^p)

Stability Conditions

Explicit Euler for Heat Equation:
Δt≤(Δx)22c\boxed{\Delta t \leq \frac{(\Delta x)^2}{2c}}

General Form for Parabolic PDEs: Explicit methods require Δt=O((Δx)2)\Delta t = O((\Delta x)^2)

Implicit Methods: Typically unconditionally stable (no restriction on Δt\Delta t)


Quick Reference Tables

Numerical Integration Methods

MethodOrderNodesFormula
Midpoint11(b−a)f(a+b2)(b-a)f\left(\frac{a+b}{2}\right)
Trapezoid12b−a2(f(a)+f(b))\frac{b-a}{2}(f(a) + f(b))
Simpson33b−a6(f(a)+4f(a+b2)+f(b))\frac{b-a}{6}\left(f(a) + 4f\left(\frac{a+b}{2}\right) + f(b)\right)

ODE Methods

MethodTypeOrderStability
EulerExplicit1Conditional
Backward EulerImplicit1Unconditional
Implicit TrapezoidImplicit2Unconditional
RK4Explicit4Conditional

PDE Methods (Heat Equation)

MethodTypeTime AccuracyStability
Forward EulerExplicit1Δt≤(Δx)22c\Delta t \leq \frac{(\Delta x)^2}{2c}
Backward EulerImplicit1Unconditional
Crank-NicolsonImplicit2Unconditional

Key Theorems for Exam

1. Normal Equations

  • Solution exists and is unique when rank(A)=n\text{rank}(A) = n
  • Solution: x=(ATA)−1ATbx = (A^T A)^{-1}A^T b
  • ATAA^T A is symmetric positive definite

2. QR Factorization

  • Every matrix has QR factorization
  • If rank(A)=n\text{rank}(A) = n, then R1R_1 is nonsingular
  • Least squares solution: solve R1x=c1R_1 x = c_1 where c1c_1 are first nn entries of QTbQ^T b

3. SVD Properties

  • ∥A∥2=σ1\|A\|_2 = \sigma_1 (largest singular value)
  • cond2(A)=σ1σn\text{cond}_2(A) = \frac{\sigma_1}{\sigma_n}
  • rank(A)=\text{rank}(A) = number of nonzero singular values
  • Pseudoinverse: A+=VΣ+UTA^+ = V\Sigma^+ U^T

4. Eigenvalue Methods

  • Power method finds largest ∣λ∣|\lambda|
  • Inverse power method finds smallest ∣λ∣|\lambda|
  • Shifted inverse power method finds λ\lambda closest to shift σ\sigma
  • All require unique target eigenvalue for convergence

5. Stability vs Accuracy

  • Stability: errors don't grow unboundedly
  • Accuracy: truncation error order
  • Backward stable: computes exact solution to nearby problem
  • Conditionally stable: requires relationship between parameters

Common Mistakes to Avoid

  1. Forgetting to normalize in power methods
  2. Using wrong order formulas together (e.g., O(h)O(h) with O(h2)O(h^2))
  3. Not checking rank conditions before using normal equations
  4. Confusing local and global truncation error
  5. Forgetting boundary conditions in PDE problems
  6. Not checking stability conditions for explicit methods
  7. Computing matrix inverses explicitly (use factorizations instead)
  8. Mixing up conditioning and stability

Final Exam Tips

  1. Proofs: Focus on understanding the logic flow, not memorizing word-for-word
  2. Formulas: Know when each method applies and its limitations
  3. Order of accuracy: Always check consistency of approximations
  4. Stability: Know which methods are stable and under what conditions
  5. Relationships: Understand how different concepts connect (e.g., SVD → condition number → stability)
  6. Special cases: Remember homogeneous vs non-homogeneous boundary conditions
  7. Computational cost: Know the big-O complexity of major algorithms

Last-Minute Review Checklist

  • Can derive normal equations from minimization
  • Understand QR vs Cholesky for least squares
  • Know SVD properties and applications
  • Understand power method variants
  • Know stability conditions for PDEs
  • Can set up finite difference schemes
  • Understand explicit vs implicit methods
  • Know integration by parts for Galerkin method
  • Understand relationship between local and global error
  • Can identify method type and order

Good luck on your final exam! 🎓