CSS322 - Scientific Computing: Proofs Cheat Sheet
Final Exam Reference (Lectures 8-17)
Lecture 8: Numerical Integration
Proof: Method of Undetermined Coefficients = Integrating the Interpolant
Claim: If we use the same set of nodes, the method of undetermined coefficients gives the same weights as integrating the interpolant.
Proof:
- Let w1,…,wn denote the weights obtained by the method of undetermined coefficients
- Let Qn(f)=∑i=1nwif(xi) denote the resulting quadrature rule
- Let pn−1(x)=c1+c2x+…+cnxn−1 denote the polynomial of degree n−1 that interpolates f at x1,…,xn
∫abpn−1(x)dx=∫ab(c1+c2x+⋯+cnxn−1)dx
This equals:
c1∫abdx+c2∫abxdx+⋯+cn∫abxn−1dx
By the method of undetermined coefficients, these integrals equal:
c1(w1+⋯+wn)+c2(w1x1+⋯+wnxn)+⋯+cn(w1x1n−1+⋯+wnxnn−1)
Rearranging:
w1(c1+c2x1+⋯+cnx1n−1)+w2(c1+c2x2+⋯+cnx2n−1)+⋯
This simplifies to:
w1pn−1(x1)+w2pn−1(x2)+⋯+wnpn−1(xn)=w1f(x1)+w2f(x2)+⋯+wnf(xn)=Qn(f)
The last line follows because pn−1 interpolates f at x1,…,xn, so pn−1(xi)=f(xi).
This shows that the quadrature rule obtained by the method of undetermined coefficients is exactly the integral of the interpolant. ∎
Proof: Midpoint Rule is Degree 1
Claim: The midpoint rule M(f)=(b−a)f(2a+b) is of degree 1.
Proof:
- By construction, midpoint rule is exact for degree 0 (constants)
- For degree 1, any polynomial has form p(x)=cx+e
Exact integral:
∫ab(cx+e)dx=c[2x2]ab+e[x]ab=2c(b2−a2)+e(b−a)
Midpoint rule gives:
(b−a)p(2a+b)=(b−a)(c2a+b+e)
=c(b−a)2a+b+e(b−a)=2c(b2−a2)+e(b−a)
These match! ✓
Counterexample for degree 2:
∫023x2dx=[x3]02=8
But:
M(f)=2(3)(1)2=6=8
Therefore, midpoint rule is NOT exact for all degree 2 polynomials. ∎
Proof: Midpoint is Twice as Accurate as Trapezoid
Setup: Let m=2a+b (midpoint)
Taylor expansion of f(x) about m:
f(x)=f(m)+f′(m)(x−m)+2f′′(m)(x−m)2+6f(3)(m)(x−m)3+24f(4)(m)(x−m)4+…
For Midpoint Rule:
Integrate both sides from a to b. Odd powers integrate to zero due to symmetry about m:
∫abf(x)dx=f(m)(b−a)+0+24f′′(m)(b−a)3+0+1920f(4)(m)(b−a)5+…
Therefore:
I(f)=M(f)+E(f)+F(f)+…
where:
E(f)=24f′′(m)(b−a)3
F(f)=1920f(4)(m)(b−a)5
For Trapezoid Rule:
Starting from Taylor expansions at a and b, then adding them and multiplying by (b−a)/2:
T(f)=M(f)+3E(f)+5F(f)+…
Rearranging:
I(f)=T(f)−2E(f)−4F(f)+…
Comparison:
- Midpoint error: E(f)+F(f)+…
- Trapezoid error: 2E(f)+4F(f)+…
The trapezoid rule has twice the error of the midpoint rule (when (b−a) is not too large and f(4) is well-behaved). ∎
Error Estimation Formula:
From T(f)=M(f)+3E(f)+5F(f)+…:
E(f)≈3T(f)−M(f)
Derivation: Simpson's Rule from Midpoint and Trapezoid
Setup: From the expansions:
- I(f)=M(f)+E(f)+F(f)+…
- I(f)=T(f)−2E(f)−4F(f)+…
Multiply midpoint by 2/3:
32I(f)=32M(f)+32E(f)+32F(f)+…
Multiply trapezoid by 1/3:
31I(f)=31T(f)−32E(f)−34F(f)+…
Add them:
I(f)=(32M(f)+31T(f))−32F(f)+…
But:
32M(f)+31T(f)=32(b−a)f(m)+312b−a(f(a)+f(b))
=6b−a(f(a)+4f(2a+b)+f(b))=S(f)
Therefore:
I(f)=S(f)−32F(f)+…
This shows the dominant error term of Simpson's rule is O((b−a)5), making it much more accurate! ∎
Lecture 10: Linear Least Squares
Lemma 1: ATA is Symmetric Positive Definite
Theorem: Suppose A∈Rm×n has rank n. Then ATA is symmetric positive definite.
Proof:
Part 1 - Symmetry:
(ATA)T=AT(AT)T=ATA ✓
Part 2 - Positive Definiteness:
Suppose x∈Rn is nonzero. Then:
xTATAx=(Ax)TAx=∥Ax∥22≥0
But Ax is a nontrivial linear combination of columns of A (since x=0).
Since columns of A are linearly independent (because rank of A is n), we have Ax=0.
Therefore: xTATAx=∥Ax∥22>0 ✓
This proves ATA is symmetric positive definite. ∎
Theorem: Suppose f(x)=xTCx−2hTx+d, where C is symmetric positive definite. Then f has a unique minimizer at x∗=C−1h.
Proof:
Step 1: Evaluate f(x∗)
f(x∗)=f(C−1h)=(C−1h)TCC−1h−2hTC−1h+d
=hTC−Th−2hTC−1h+d
Since C is symmetric: C−T=(CT)−1=C−1
Therefore:
f(x∗)=hTC−1h−2hTC−1h+d=−hTC−1h+d
Step 2: Evaluate f(x∗+y) for any nonzero vector y
Let y∈Rn be any arbitrary nonzero vector:
f(x∗+y)=(C−1h+y)TC(C−1h+y)−2hT(C−1h+y)+d
=(hTC−1+yT)(h+Cy)−2hTC−1h−2hTy+d
=hTC−1h+hTy+yTh+yTCy−2hTC−1h−2hTy+d
Since hTy=yTh (scalars are symmetric):
f(x∗+y)=−hTC−1h+d+yTCy=f(x∗)+yTCy
Step 3: Show x∗ is the unique minimizer
But C is symmetric positive definite, so: yTCy>0 for all nonzero y
Therefore: f(x∗+y)=f(x∗)+yTCy>f(x∗) for all nonzero y
So x∗ is the unique minimizer. ∎
Main Theorem: Normal Equations Solution
Theorem:
x=(ATA)−1ATb
is the unique solution to the linear least squares problem when A has rank n.
Proof: This follows directly from Lemma 1 and Lemma 2 by setting:
- f(x)=∥Ax−b∥22=xTATAx−2bTAx+bTb
- C=ATA (symmetric positive definite by Lemma 1)
- h=ATb
- d=bTb
By Lemma 2: x∗=C−1h=(ATA)−1ATb ∎
QR Factorization Theorems
Theorem 1: Any A∈Rm×n (m≥n) can be factored A=QR where Q is an m×m orthogonal matrix and R is an m×n upper triangular matrix.
Theorem 2: If A∈Rm×n, m≥n, has rank n and A=QR is the QR factorization of A, then R has rank n and R1 is nonsingular.
Main Result: Let A=QR be the QR factorization of A. Assume A∈Rm×n, m≥n, rank(A)=n.
Let c=QTb. Let c1 be the top n entries of c. Then x minimizes ∥Ax−b∥2 if:
R1x=c1
Proof:
∥Ax−b∥2=∥QRx−b∥2=∥Q(Rx−QTb)∥2=∥Rx−c∥2
=[R10]x−[c1c2]2=[R1x−c1−c2]2
=∥R1x−c1∥22+∥c2∥22
Since c2 is constant, minimizing this is equivalent to minimizing ∥R1x−c1∥2.
If R1 is nonsingular (which follows from Theorem 2), then R1x=c1 has a unique solution that minimizes the norm. ∎
Theorem: Let v∈Rm be a nonzero vector. Then H=I−2vTvvvT is symmetric and orthogonal.
Proof:
Symmetry:
HT=(I−2vTvvvT)T=IT−2vTv(vvT)T=I−2vTvvvT=H ✓
Orthogonality:
HHT=HH=(I−2vTvvvT)(I−2vTvvvT)
=I−4vTvvvT+4(vTv)2vvTvvT
=I−4vTvvvT+4(vTv)2v(vTv)vT
=I−4vTvvvT+4vTvvvT=I ✓ ∎
Lecture 14: Singular Value Decomposition
Lemma 3: Norm of Diagonal Matrix
Lemma: Let D∈Rm×n be diagonal. Then ∥D∥p=max∣dii∣ for any p.
Proof: [Exercise for student]
Key steps:
- For any vector x, Dx has entries diixi
- ∥Dx∥p≤(max∣dii∣)∥x∥p
- Equality achieved when x is the unit vector in direction of largest ∣dii∣
- Therefore ∥D∥p=max∣dii∣ ∎
Theorem: Matrix 2-Norm from SVD
Theorem: If A=UΣVT (SVD of A), then ∥A∥2=σ1 (largest singular value).
Proof:
From Lemma 1 and Lemma 2 (from lecture notes):
- ∥QA∥2=∥A∥2 for orthogonal Q
- ∥AQ∥2=∥A∥2 for orthogonal Q
Therefore:
∥A∥2=∥UΣVT∥2=∥ΣVT∥2=∥Σ∥2
By Lemma 3: ∥Σ∥2=σ1 (largest diagonal entry). ∎
Theorem: Condition Number from SVD
Theorem: If σ1,…,σn are singular values of A∈Rn×n and A is invertible, then:
∥A−1∥2=σn1
Proof:
From SVD A=UΣVT:
A−1=(UΣVT)−1=V−TΣ−1U−1=VΣ−1UT
Note that:
Σ−1=σ110⋮00σ21⋮0⋯⋯⋱⋯00⋮σn1
However, this is not an SVD of A−1 because diagonal entries are not in non-increasing order.
Consider permutation matrix P=1⋱1
Then:
PΣ−1P=σn1⋮0⋯⋱⋯0⋮σ11
has diagonal entries in non-increasing order.
Since PP=I, VP is orthogonal, and PUT is orthogonal:
A−1=VΣ−1UT=(VP)(PΣ−1P)(PUT)
is the SVD of A−1.
Thus: ∥A−1∥2=σn1 (largest singular value of A−1). ∎
Corollary:
cond2(A)=∥A∥2∥A−1∥2=σnσ1
Theorem: Condition Number of ATA
Theorem: Suppose A∈Rm×n with rank(A)=n. Then:
cond2(ATA)=cond2(A)2
Proof: [Exercise for student]
Key steps:
- If A=UΣVT, then ATA=VΣTUTUΣVT=VΣTΣVT
- ΣTΣ has diagonal entries σi2
- cond2(ATA)=σn2σ12=(σnσ1)2=cond2(A)2 ∎
Theorem: Rank Determination
Theorem: If A=UΣVT, then:
rank(A)=number of nonzero singular values
Proof: [Standard linear algebra result - relies on orthogonal transformations preserving rank]
Lecture 15: Eigenvalues and Eigenvectors
Theorem: Eigenvalues and Eigenvectors of Inverse Matrix
Theorem: If A∈Rn×n is invertible, then A and A−1 have the same eigenvectors, and if λ is an eigenvalue of A, then 1/λ is an eigenvalue of A−1.
Proof:
Suppose (λ,x) is an eigenpair of A:
Ax=λx
Multiply both sides by A−1:
A−1(Ax)=A−1(λx)
x=λA−1x
Divide both sides by λ (assuming λ=0):
(λ1)x=A−1x
Therefore, (1/λ,x) is an eigenpair of A−1. ∎
Theorem: Shifted Matrix Properties
Theorem: Suppose (A−σI) is invertible. Then:
- A and (A−σI)−1 have the same eigenvectors
- λ is an eigenvalue of A if and only if λ−σ1 is an eigenvalue of (A−σI)−1
Proof:
Suppose (λ,x) is an eigenpair of A:
Ax=λx
Subtract σx from both sides:
(A−σI)x=(λ−σ)x
Multiply both sides by (A−σI)−1:
x=(λ−σ)(A−σI)−1x
Divide by (λ−σ) (assuming λ=σ):
(λ−σ1)x=(A−σI)−1x
Therefore, (1/(λ−σ),x) is an eigenpair of (A−σI)−1. ∎
Lecture 13: Initial Value Problems for ODEs
Theorem: Symmetry of Mixed Partials
Theorem: If both ∂xj∂xi∂2f and ∂xi∂xj∂2f are well-defined and continuous, then:
∂xj∂xi∂2f=∂xi∂xj∂2f
Proof: [Standard multivariable calculus result - Schwarz's theorem/Clairaut's theorem]
Lecture 16: Boundary Value Problems
Formula:
∫abv′′(t)ϕ(t)dt=v′(t)ϕ(t)ab−∫abv′(t)ϕ′(t)dt
When ϕ(a)=ϕ(b)=0:
∫abv′′(t)ϕ(t)dt=−∫abv′(t)ϕ′(t)dt
Proof:
Standard integration by parts with u=ϕ(t), dv=v′′(t)dt:
∫abv′′(t)ϕ(t)dt=v′(t)ϕ(t)ab−∫abv′(t)ϕ′(t)dt
If ϕ(a)=ϕ(b)=0:
v′(b)ϕ(b)−v′(a)ϕ(a)=0
Therefore:
∫abv′′(t)ϕ(t)dt=−∫abv′(t)ϕ′(t)dt ∎
Finite Difference Approximations
First derivative (forward): O(h)
u′(t)≈hu(t+h)−u(t)
First derivative (backward): O(h)
u′(t)≈hu(t)−u(t−h)
First derivative (centered): O(h2)
u′(t)≈2hu(t+h)−u(t−h)
Second derivative (centered): O(h2)
u′′(t)≈h2u(t+h)−2u(t)+u(t−h)
Formula:
a0=F(h)+q−p−1F(h)−F(h/q)
where:
- F(h) is the approximation with step size h
- F(h/q) is the approximation with step size h/q
- p is the order of the method
- The extrapolated value has error O(hr) where r>p
Power Method Convergence
For A with eigenvalues λ1,…,λn where ∣λ1∣>∣λj∣ for all j>1:
If x(0)=∑j=1nαjvj (linear combination of eigenvectors), then:
x(k)=j=1∑nλjkαjvj=λ1k(α1v1+j=2∑n(λ1λj)kαjvj)
As k→∞:
x(k)→λ1kα1v1 (multiple of dominant eigenvector)
Rayleigh Quotient
λ=(x(k))Tx(k)(x(k))TAx(k)
(Denominator may be omitted if x(k) is normalized)
Error Analysis for ODEs
Local Truncation Error: lk=yk−y(tk) assuming yi=y(ti) for i<k
Global Truncation Error: ek=yk−y(tk) (accumulated error)
Relationship: For a wide class of ODEs and methods:
- If local truncation error is O(hp+1)
- Then global truncation error is O(hp)
Stability Conditions
Explicit Euler for Heat Equation:
Δt≤2c(Δx)2
General Form for Parabolic PDEs: Explicit methods require Δt=O((Δx)2)
Implicit Methods: Typically unconditionally stable (no restriction on Δt)
Quick Reference Tables
Numerical Integration Methods
| Method | Order | Nodes | Formula |
|---|
| Midpoint | 1 | 1 | (b−a)f(2a+b) |
| Trapezoid | 1 | 2 | 2b−a(f(a)+f(b)) |
| Simpson | 3 | 3 | 6b−a(f(a)+4f(2a+b)+f(b)) |
ODE Methods
| Method | Type | Order | Stability |
|---|
| Euler | Explicit | 1 | Conditional |
| Backward Euler | Implicit | 1 | Unconditional |
| Implicit Trapezoid | Implicit | 2 | Unconditional |
| RK4 | Explicit | 4 | Conditional |
PDE Methods (Heat Equation)
| Method | Type | Time Accuracy | Stability |
|---|
| Forward Euler | Explicit | 1 | Δt≤2c(Δx)2 |
| Backward Euler | Implicit | 1 | Unconditional |
| Crank-Nicolson | Implicit | 2 | Unconditional |
Key Theorems for Exam
1. Normal Equations
- Solution exists and is unique when rank(A)=n
- Solution: x=(ATA)−1ATb
- ATA is symmetric positive definite
2. QR Factorization
- Every matrix has QR factorization
- If rank(A)=n, then R1 is nonsingular
- Least squares solution: solve R1x=c1 where c1 are first n entries of QTb
3. SVD Properties
- ∥A∥2=σ1 (largest singular value)
- cond2(A)=σnσ1
- rank(A)= number of nonzero singular values
- Pseudoinverse: A+=VΣ+UT
4. Eigenvalue Methods
- Power method finds largest ∣λ∣
- Inverse power method finds smallest ∣λ∣
- Shifted inverse power method finds λ closest to shift σ
- All require unique target eigenvalue for convergence
5. Stability vs Accuracy
- Stability: errors don't grow unboundedly
- Accuracy: truncation error order
- Backward stable: computes exact solution to nearby problem
- Conditionally stable: requires relationship between parameters
Common Mistakes to Avoid
- Forgetting to normalize in power methods
- Using wrong order formulas together (e.g., O(h) with O(h2))
- Not checking rank conditions before using normal equations
- Confusing local and global truncation error
- Forgetting boundary conditions in PDE problems
- Not checking stability conditions for explicit methods
- Computing matrix inverses explicitly (use factorizations instead)
- Mixing up conditioning and stability
Final Exam Tips
- Proofs: Focus on understanding the logic flow, not memorizing word-for-word
- Formulas: Know when each method applies and its limitations
- Order of accuracy: Always check consistency of approximations
- Stability: Know which methods are stable and under what conditions
- Relationships: Understand how different concepts connect (e.g., SVD → condition number → stability)
- Special cases: Remember homogeneous vs non-homogeneous boundary conditions
- Computational cost: Know the big-O complexity of major algorithms
Last-Minute Review Checklist
Good luck on your final exam! 🎓