Math Model Reduction Using SVD

Math 2984H Term Project
Implicit Solvers and SVD with an Application
in Model Reduction
Mark Brandao and Susanna Mostaghim
May 10, 2013
Abstract
The goal of this project is to employ our Linear Algebra, Differential Equations and
Matlab skills for a specific application in the area of Model Reduction. The main goal at
the end is to reduce a large number of differential equations into a much smaller one in such
a way that the smaller number of ODEs gives almost the same information as the original
large set. The main tools in achieving this goal would be numerical solution of ODEs and an
important matrix decomposition, called the Singular Value Decomposition (SVD).
1 Part I: Implicit Solvers for Differential Equations
1.1 Problem 1.1
There are numerous numerical solutions techniques, including implicit solvers, for the systems of
differential equations of the form y = Ay + bu(t) , y(t0) = y0, but one of the most fundamental
methods is Euler’s method, which is a direct method solving differential equations. Applying
Euler’s method to an equation of this form yields
yk+1 = yk + h(Ayk + buk), for k = 0, 1, 2, ...
However, when given a scalar equation, as follows (let us assume λ < 0), applying Euler’s
method leads to an expression for yk+1 that is independent of yk.
y = λy, y(0) = y0 (1.1)
y1 = y0 + hλy0
= y0(1 + hλ)

y2 = y1 + hλy1
= (y0 + hλy0) + hλ(y0 + hλy0)
= y0 + hλy0 + hλy0 + h2
λ2
y0
= y0(1 + 2hλ + (hλ)2
)
= y0(1 + hλ)2
y3 = y2 + hλy2
= (y0(1 + hλ)2
) + hλ(y0(1 + hλ)2
)
= y0(1 + hλ)2
+ hλy0(1 + hλ)2
= y0((1 + hλ)2
+ hλ(1 + hλ)2
)
= y0((1 + hλ)2
(1 + hλ))
= y0(1 + hλ)3
So for each yk in the interative Euler method, the result seems to be
yk = y0(1 + hλ)k
.
Indeed, it can be shown that
y1 = y0(1 + hλ)1
= y0(1 + hλ)
= y0 + y0hλ
Because this is what we defined our original y1 point to be, this formula holds for the base case
(the first step approximation is the zeroth step times the h-step times the differential equation at
the zeroth step).
Assume that the equation holds for yn:
yn = y0(1 + hλ)n
We will show that, assuming this is true, the equation holds for yn+1 as well.
yn+1 = yn + ynhλ
= (y0(1 + hλ)n
) + (y0(1 + hλ)n
) hλ
= y0(1 + hλ)n
+ y0hλ(1 + hλ)n
= y0((1 + hλ)n
+ hλ(1 + hλ)n
)
= y0((1 + hλ)(1 + hλ)n
)
= y0(1 + hλ)n+1
Because this holds as well, by induction, we can say that this equation (yk = y0(1 + hλ)k
) holds
for all k ≥ 1.

Because the actual solution to y = λy is y(t) = y0eλt
and λ < 0, we know that the solution
decays to zero. So it is essential, if Euler’s method yields an accurate approximation, for the iterate
yk to decay to zero (yk → 0 ) as k → ∞ .
lim
k→∞
y0(1 + hλ)k+1
y0(1 + hλ)k
= lim
k→∞
(1 + hλ)k
(1 + hλ)
(1 + hλ)k
= lim
k→∞
|(1 + hλ)|
= |(1 + hλ)|
Thus, if yk = y0(1 + hλ)k
→ 0,
|1 + hλ| < 1
−1 <1 + hλ < 1
−2 < hλ < 0
Since λ < 0, we can infer that, for the sequence to converge,
0 < h|λ| < 2.
So if yk = y0(1 + hλ)k
→ 0, then 0 < h|λ| < 2.
However, if 0 < h|λ| < 2,
0 < h|λ| < 2
−2 < hλ < 0
−1 < 1 + hλ < 1
|1 + hλ| < 1
|1 + hλ| = lim
k→∞
y0(1 + hλ)k+1
y0(1 + hλ)k
Since |1 + hλ| < 1, the iterate y0(1 + hλ)k
→ 0. With this condition in place, Euler’s method
yields an approximation of the original function which is usable. However, if h|λ| > 2, then the
sequence diverges, and the Euler’s method approximation is defective.
1.2 Problem 1.2
Let us consider a system of the form
y (t) = Ay y(0) = y0 , where A =
−8003 1999
23988 −9004
, y(0) =
1
4
The solution of this equation is of the form
y(t) = c1eλ1t
v1 + c2eλ2t
v2

So the ﬁrst step of solving this equation is ﬁnding the eigenvalues and associated eigenvectors of
the matrix A. For the matrix A,
λ1 = −14000 v1 =
1
−3
λ2 = −7 v2 =
1
4
So the general solution is
y(t) = c1e−14000t 1
−3
+ c2e−7t 1
4
And since y(0) =
1
4
,
y(0) =
1
4
= c1e−14000(0) 1
−3
+ c2e−7(0) 1
4
= c1e0 1
−3
+ c2e0 1
4
= c1
1
−3
+ c2
1
4
=
c1 + c2
−3c1 + 4c2
Breaking each row up into individual equations, the result is
c1 + c2 = 1 ⇒ c1 = 1 − c2
−3c1 + 4c2 = 4 ⇒ − 3(1 − c2) + 4c2 = 4
The solution for these two equations is
c1 = 0 , c2 = 1
Which yields for the solution equation
y(t) = (0)e−14000t 1
−3
+ (1)e−7t 1
4
= e−7t 1
4
=
e−7t
4e−7t
Because both components of the vector solution have a negative exponent, they are both decaying
to zero as t → ∞. Since we want to model this equation, we will be using Euler’s method on the
interval t ∈ [0, 0.02]. See Appendix A for Matlab code. The true solution,
e−7t
4e−7t , is as follows
for t ∈ [0, 0.02]:

Using the Euler’s method approximation with a step size (h) of .001, the result is as follows:
This is not a particularly good approximation, since h|λ| > 2 (the second value of λ was -14000,
so h|λ| = 14). For the last two step sizes, the values of y go to
−300
1000
rather than decaying to
0
0
. If we use a smaller step size such that h|λ| < 2, for example, 1.3571 × 10−4
, we should get
a better approximation.
When we use 1.3571 × 10−4
for h, the result is as follows:

This is a better approximation, since this graph decays to
0
0
. When h = 1.3571 × 10−4
,
h|λ2| ≈ 1.9 < 2, and h|λ1| ≈ 9.5 × 10−4
< 2 so this approximation converges and is not defective.
Indeed, this approximation seems to model the original function very accurately.
1.3 Problem 1.3
So, as seen in the previous problem, there are some difficulties when dealing with a stiff equation
(the eigenvalue of -14000 made the prospect of getting an accurate approximation somewhat diffi-
cult). An equation is said to be stiff if all the eigenvalues of the matrix A have negative real parts
and the ratio of the largest of absolute values of the real parts to the smallest one is large. In the
case of the last problem, the ratio was 2000, which is certainly large. As such, Euler’s method, as
well as other direct methods for solving differential equations, can run into some trouble. These
issues can be addressed by implicit methods for solving differential equations.
One such method is the Trapezoidal Rule. The Trapezoidal Rule Method is:
yk+1 = yk +
h
2
(f(yk, tk) + f(yk+1, tk+1)) , for k = 0, 1, 2, ...
This method relies on the k step as well as the k + 1 step (their average), whereas Euler’s method
relies only on the k step; as such, this method gives a better approximation of the function.
However, in this form, yk+1 is on both sides of the equation, so it is not an easy task to solve;
indeed, because of this, it is said to be implicitly defined.
In the case where f(y, t) = Ay + bu(t), the Trapezoidal Method yields
yk+1 = yk +
h
2
(Ayk + buk) +
h
2
(Ayk+1 + buk+1)
Which can be rearranged so that all instances of yk+1 are on the left side:
I −
h
2
A yk+1 = yk +
h
2
(Ayk + buk + buk+1)

With this equation in place, one can develop an algorithm to solve the equation for the Trapezoidal
Method. The algorithm is as follows:
Algorithm 1.1 Trapezoidal Method to solve y = Ay + bu(t), y(0) = y0
1. Choose h.
2. Set M = I − h
2
A.
3. for k = 0, 1, 2, 3, . . .
(a) Set zk = yk + h
2
(Ayk + buk + buk+1)
(b) Solve the linear system Myk+1 = zk for yk+1
One disadvantage of this algorithm, however, is the final step; having to do Gaussian elimination
to solve the linear system over and over (for every time step) is expensive. One way to make the
system easier is the find the LU decomposition of M and then solve two simpler systems of
equations.
For the linear system Myk+1 = zk, let M = LU be the LU-decomposition of M. This can be
solved in two steps.
(LU)yk+1 = zk
L(Uyk+1) = zk
Set Uyk+1 = vk+1 and solve the linear system.
yk+1 = U−1
vk+1
And substitute this result back into the original equation.
L(UU−1
vk+1) = zk
Lvk+1 = zk
Thus, there are only two systems to solve: Uyk+1 = vk+1 and Lvk+1 = zk. This is very easy,
because L and U are lower- and upper-triangular matrices.
This leads to the following, more efficient algorithm.
Algorithm 1.2 Trapezoidal Method with LU-decomposition
1. Choose h.
2. Set M = I − h
2
A
3. Compute the LU-decomposition for M

4. for k = 0, 1, 2, 3, . . .
(a) Set zk = yk + h
2
(Ayk + buk + buk+1)
(b) Solve the linear system Lvk+1 = zk+1 for vk+1.
(c) Solve the linear system Uyk+1 = vk+1 for yk+1.
1.4 Problem 1.4
Using this algorithm on the same function which was earlier approximated by Euler’s method,
y (t) = Ay y(0) = y0 , where A =
−8003 1999
23988 −9004
, y(0) =
1
4
Using a Matlab code (see Appendix B), one can approximate this system with the Trapezoidal
Method. Below is the approximation for h = .001 and t ∈ [0, 0.02].
This approximation is very close to the original function; indeed, it converges and gives a better
approximation for this step size than Euler’s method, which diverged for h = .001. To test the
extent of this implicit method’s accuracy, we shall investigate its approximation for h = .1 , t ∈
[0, 2].

So the Trapezoidal Method still converges with a larger step size, and it is even accurate on a
much larger interval as well. It is easy to see that this implicit solver is superior to Euler’s direct
method of approximating functions.
1.5 Problem 1.5
This comparison between the Trapezoidal Method and Euler’s method shows that the Trapezoidal
Method converges even for larger step sizes. So the question becomes, for what step sizes will the
solution yk converge?
To answer that question, however, we must ﬁrst apply the Trapezoidal Method symbolically to
determine the form of the iterate which we will analyze.
Let us consider the scalar equation
y = λy, y(0) = y0 , where λ < 0
The Trapezoidal Method, once more, is
yk+1 = yk +
h
2
(f(yk, tk) + f(yk+1, tk+1)) , for k = 0, 1, 2, ...
Applying the Trapezoidal Method to the scalar equation y = λy yields:

y1 = y0 +
h
2
(λy0 + λy1)
= y0 +
h
2
λy0 +
h
2
λy1
y1 −
h
2
λy1 = y0 +
h
2
λy0
(1 −
h
2
λ)y1 = (1 +
h
2
λ)y0
y1 =
1 + h
2
λ
1 − h
2
λ
y0
=
2 + hλ
2 − hλ
y0
y2 = y1 +
h
2
(λy1 + λy2)
=
2 + hλ
2 − hλ
y0 +
h
2
λ
2 + hλ
2 − hλ
y0 +
h
2
λy2
y2 −
h
2
λy2 = 1 +
h
2
λ
2 + hλ
2 − hλ
y0
1 −
h
2
λ y2 = 1 +
h
2
λ
2 + hλ
2 − hλ
y0
y2 =
1 + h
2
λ 2+hλ
2−hλ
y0
1 − h
2
λ
=
(2 + hλ) 2+hλ
2−hλ
y0
2 − hλ
=
2 + hλ
2 − hλ
2 + hλ
2 − hλ
y0
=
2 + hλ
2 − hλ
2
y0
From this pattern, one can infer that yk = 2+hλ
2−hλ
k
y0, for k = 1, 2, .... To prove that this is the
case, let us examine the base case, y1:
y1 =
2 + hλ
2 − hλ
1
y0
(y1)(2 − hλ) = (2 + hλ)y0
2y1 − hλy1 = 2y0 + hλy0
2y1 = 2y0 + hλy0 + hλy1
y1 = y0 +
h
2
λy0 +
h
2
λy1
= y0 +
h
2
(λy0 + λy1)

Indeed, the equation held for our base case. Now let us assume that it holds for yn:
yn =
2 + hλ
2 − hλ
n
y0
We will now show that the equation holds for the yn+1 case as well:
yn+1 = yn +
h
2
(λyn + λyn+1)
=
2 + hλ
2 − hλ
n
y0 +
h
2
(λ
2 + hλ
2 − hλ
n
y0 + λyn+1)
=
2 + hλ
2 − hλ
n
y0 +
hλ
2
2 + hλ
2 − hλ
n
y0 +
hλ
2
yn+1
yn+1 −
hλ
2
yn+1 = 1 +
hλ
2
2 + hλ
2 − hλ
n
y0
yn+1 1 −
hλ
2
= 1 +
hλ
2
2 + hλ
2 − hλ
n
y0
yn+1 =
1 + hλ
2
1 − hλ
2
2 + hλ
2 − hλ
n
y0
=
2 + hλ
2 − hλ
2 + hλ
2 − hλ
n
y0
=
2 + hλ
2 − hλ
n+1
y0
Since this holds for the n + 1 case if assumed true for the nth case, and it has been shown that
this equation holds for the ﬁrst case, yk = 2+hλ
2−hλ
k
y0 is true for all k ≥ 1.
Since λ < 0 and h > 0, one can see that 2+hλ
2−hλ
< 1, since the fraction is essentially the same as
2−h|λ|
2+h|λ|
. The numerator is decreasing from 2, and the denominator is increasing from 2; thus, the
fraction is less than one.
Since the fraction is less than one, and for any fraction |f| < 1 raised to the power n,
limn→∞ fn
= 0,
lim
k→∞
yk = lim
k→∞
2 + hλ
2 − hλ
k
y0 = 0
for any h > 0, λ < 0. This is a much more favorable constraint than that for Euler’s Method,
(h|λ| < 2) and it indeed demonstrates the Trapezoidal Method’s superiority.
2 PART II: The Singular Value Decomposition
The Singular Value Decomposition of matrix is a method used to break down any original matrix,
A, into three component matrices, U, S, and V. Using the SVD of a matrix, it is possible to
create a lower-rank approximation of the original matrix (an approximation with less complexity),
but we want to reduce the error between A and its approximation as much as possible. Because

this is the case, we will analyze the singular value decomposition and its properties, as well as the
properties of lower-rank approximations with the singular value decomposition.
2.1 Problem 2.1
Given a matrix A ∈ Rn×m
, Let A = USVT
be the SVD of A, where
S = diag(σ1, ..., σm), U = [u1, ..., um] , V = [v1, ..., vm] , and UT
U = VT
V = In
A = u1 u2 ... um





σ1
σ2
...
σm





v1 v2 ... vm
T
= u1 u2 ... um





σ1
σ2
...
σm










vT
1
vT
2
...
vT
m





= u1 u2 ... um





σ1vT
1
σ2vT
2
...
σmvT
m





= σ1u1vT
1 + σ2u2vT
2 + ... + σmumvT
m
=
m
i=1
σiuivT
i
Since we are looking for a rank-k approximation of A, however, we will say that
Ak = UkSkVT
k
Or, equivalently,
Ak =
k
i=1
σiuivT
i
Thus, to estimate the error, we will evaluate the diﬀerence between the original matrix and its
approximation:
A − Ak 2 =
m
i=1
σiuivT
i −
k
i=1
σiuivT
i
2
=
m
i=k+1
σiuivT
i
2

Since the 2-norm of a matrix A is the largest singular value, and the singular values are equal to
the square root of the eigenvalues of AT
A,
m
i=k+1
σiuivT
i
2
= ˜σ1 =
√
λ
where
√
λ are the eigenvalues (˜λ) of


m
i=k+1
σiuivT
i
T m
i=k+1
σiuivT
i


=
m
i=k+1
σiviuT
i
m
i=k+1
σiuivT
i
= σk+1vk+1uT
k+1 + · · · + σmvmuT
m σk+1uk+1vT
k+1 + · · · + σmumvT
m
= σ2
k+1vk+1uT
k+1uk+1vT
k+1 + · · · + σ2
mvmuT
mumvT
m
Since UT
U = In, uT
j uk = 0 when j = k,
=





σ2
k+1
σ2
k+2
...
σ2
m





√
λ = ˜λ = σk+1, σk+2, . . . , σm
And since σk+1 > σk+2 > · · · > σm, the largest value is σk+1. Hence, this is the value of A−Ak 2:
the first (largest) singular value not included in the rank-k approximation of A.
2.2 Problem 2.2
Now we know how a singular value decomposition works, and we know about the error between
the rank-k approximation and the original matrix, we want to find out how to build the SVD of a
matrix: how do we find the singular values, as well as right singular vectors and the left singular
vectors? To understand this, we will examine AT
A and AAT
.
Given An×n
, let A = USVT
be the SVD of A where
S = diag(σ1, ..., σm), U = [u1, ..., um] , V = [v1, ..., vm] , and UT
U = VT
V = In

A = USVT
AAT
= USVT
USVT T
= USVT
SVT T
UT
= USVT
VST
UT
= USST
UT
Because S is a diagonal matrix, SST
= S2
= US2
UT
Because UT
U = I, UT
= U−1
= US2
U−1
This matrix is of the form PΛP−1
= Q, a form of matrix diagonalization (Λ is a diagonal matrix),
where the the column vectors of P are the eigenvectors of the matrix Q, and the values along the
diagonal of Λ are the corresponding eigenvalues. It then follows that the column vectors of U (ui)
are the eigenvectors of AAT
, and the values along the diagonal of S2
(σ2
i ) are the corresponding
eigenvalues.
A = USVT
AT
A = USVT T
USVT
= VST
UT
USVT
= VST
SVT
Because S is a diagonal matrix, ST
S = S2
= VS2
VT
Because VT
V = I, VT
= V−1
= VS2
V−1
This matrix is of the form PΛP−1
= Q, a form of matrix diagonalization (Λ is a diagonal matrix),
where the the column vectors of P are the eigenvectors of the matrix Q, and the values along the
diagonal of Λ are the corresponding eigenvalues. It then follows that the column vectors of V (vi)
are the eigenvectors of AT
A, and the values along the diagonal of S2
(σ2
i ) are the corresponding
eigenvalues.
If we want to be able to ﬁnd the eigenvectors of AAT
without calculating all the properties of that
matrix, the question arises if there is a way to ﬁnd those left singular vectors. In order to do that,

we will consider the problem Avi, where vi is a right singular vector.
A = u1 u2 ... um





σ1
σ2
...
σm





v1 v2 ... vm
T
Avi = u1 u2 ... um





σ1
σ2
...
σm





v1 v2 ... vm
T
vi
Since VT
V = I,
VT
vi =





x1
x2
...
xn





: when k = i, xk = 0; when k = i, xk = 1
= ii (Let ii denote the i-th column of the identity matrix.)
Avi = u1 u2 ... um





σ1
σ2
...
σm





ii
= u1 u2 ... um









0
0
...
σi
...
0









= σiui
Thus, it is possible to ﬁnd the left singular vectors of the matrix if one has the right singular
vectors.
2.3 Problem 2.3
Using what we now know about the components of the SVD of a matrix (the left and the right
singular vectors as well as the singular values), we will use this information to compute the SVD
of the following matrix:

A =
2 3
0 2
First, we will find the right singular vectors (and the square of the singular values) by looking at
AT
A, since we know it is of the form VS2
V−1
.
AT
A =
2 0
3 2
2 3
0 2
=
4 6
6 13
To find the eigenvalues of this matrix, we will examine det AT
A − λI .
det AT
A − λI =
4 − λ 6
6 13 − λ
= (4 − λ) (13 − λ) − 62
= 52 − 17λ + λ2
− 36
= λ2
− 17λ + 16
= (λ − 16) (λ − 1)
Thus, the eigenvalues of this matrix (AT
A) are λ = 16 and λ = 1. Using this information to
compute the eigenvectors of the matrix, we must evaluate two linear systems with the matrix
AT
A − λI and the two different values for λ:
−12 6
6 −3
v = 0
3 6
6 12
u = 0
First, we will solve the first system using an augmented matrix:
−12 6
6 −3
v = 0
=
−12 6 0
6 −3 0
=
−12 6 0
0 0 0 R2 − (−1
2
)R1
−12v1 + 6v2 = 0
6v2 = 12v1
v2 = 2v1
v =
v1
2v1
= v1
1
2

Thus,
1
2
is the eigenvector corresponding to λ = 16. Now, we will find the eigenvector corre-
sponding to λ = 1.
3 6
6 12
u = 0
=
3 6 0
6 12 0
=
3 6 0
0 0 0 R2 − (2)R1
3u1 + 6u2 = 0
3u1 = −6u2
u2 = −2u2
u =
−2u2
u2
= u2
−2
1
So a matrix diagonalization (of the form VΛV−1
) for AT
A is:
AT
A =
1 −2
2 1
16 0
0 1
1 −2
2 1
−1
Now it is known to us that the singular values of A are the square root of the eigenvalues along the
diagonal: the first singular value of A is 4, and the second singular value is 1. This diagonalization
is not completely sufficient, however. One property that is necessary of the matrix V is that
V−1
= VT
. Since V ∈ R2×2
,
v11 v12
v21 v22
v11 v21
v12 v22
=
1 0
0 1
v1 · v1 = 1 = v2 · v2 v1 · v2 = 0 = v2 · v1
For the vectors v1 =
1
2
and v2 =
−2
1
, it is the case that v1 · v2 = 0 = v2 · v1, but
v1 · v1 = 5 = 1 and v2 · v2 = 5 = 1, so the eigenvectors must be scaled so that this is the case. To
make this the case, the vectors must be normalized:
˜v1 =
v1
v1
˜v2 =
v2
v2
˜v1 =
v1
√
5
˜v2 =
v2
√
5
˜v1 =
1√
5
2√
5
˜v2 =
− 2√
5
1√
5

Thus, the matrix diagonalization with the new V matrix is:
AT
A =
1√
5
− 2√
5
2√
5
1√
5
16 0
0 1
1√
5
− 2√
5
2√
5
1√
5
T
Since it is known to use that Avi = σiui, we can use these new right singular vectors to determine
the left singular vectors.
Av1 =
2 3
0 2
1√
5
2√
5
=
8√
5
4√
5
= 4
2√
5
1√
5
= σ1u1
Thus, the left singular vector associated with the ﬁrst singular value σ1 = 4 is u1 =
2√
5
1√
5
.
Av2 =
2 3
0 2
− 2√
5
1√
5
=
− 1√
5
2√
5
= 1
− 1√
5
2√
5
= σ2u2
Thus, the left singular vector associated with the second singular value σ2 = 1 is u2 =
− 1√
5
2√
5
.
Hence, the singular value decomposition of A is:
A =
2√
5
− 1√
5
1√
5
2√
5
4 0
0 1
1√
5
− 2√
5
2√
5
1√
5
T
=
2√
5
− 1√
5
1√
5
2√
5
4 0
0 1
1√
5
2√
5
− 2√
5
1√
5
This SVD of A is equivalent to σ1u1vT
1 +σ2u2vT
2 = 4
2√
5
1√
5
1√
5
2√
5
+
− 1√
5
2√
5
− 2√
5
1√
5
Since we are looking for a rank-1 approximation of A, a rank 2 matrix, we will only take the ﬁrst
component of the SVD of A as the rank-1 approximation A1:
Ak = A1 = σ1u1vT
1 = 4
2√
5
1√
5
1√
5
2√
5
= 4
2/5 4/5
1/5 2/5
=
8/5 16/5
4/5 8/5

The error of this approximation, A − A1 2, is equal to 1, the excluded singular value from the
approximation of A. No rank-1 approximation of A can get closer than this; the best one can do
is to have an error of 1.
2.4 Problem 2.4
Now that we have a firmer grasp on singular value decomposition of matrices and computing
optimal rank-k approximations of these matrices, we will apply our knowledge to an important use
of matrices: image approximation. Images can take up a large amount of data, and it is possible
to have a nearly identical image that takes up much less data. If one uses a matrix to represent
an image, one can compute the SVD of that matrix and make smaller-rank approximations, which
will have less complexity and therefore use less data. We want the optimal rank-k matrices that
have a small error, however. For example, consider the Lena image. The matrix that holds the
data for this image is a rank-508 matrix. One can build a Matlab code (see appendix C) to find
the value of k for which the optimal rank-k approximation of the Lena matrix has a relative error
( A−Ak
A
) less than a specified value. First, we will compute the SVD of the Lena matrix (A) using
Matlab and plot the normalized singular values.
From this plot, we can see that the first hundred or so singular values are within 2 orders of
magnitude of each other; similarly, almost every singular value is within 4 orders of magnitude
from the first singular value. With this in mind we can construct approximation for the optimal
rank-k matrix approximation for which the relative error is less than 10−1
, 10−2
, and 5 × 10−3
.
Using the Matlab code for different values of k, however, and computing the error, gives a more

accurate result. Displaying the Lena image, matrix rank, and relative error for the three error
cases yields:
Original image, Rank k = 508
Rank k = 53, Rel. Error = .0100
Hence, it is possible to have an approximation of the original image that is nearly flawless, but
it has a much lower rank; hence, it has a lower complexity (takes up less space).
2.5 Problem 2.5
We will now utilize our knowledge of how to apply SVDs to images to work with a large group of
images. We will be working with faces from the Yale database. Our goal is to make a compilation
matrix that has the information for every face in the database. With this data, we will be able to
reconstruct and recognize similar images (other faces). Since SVD reveals optimal information for
approximation, it is the main tool in this case as well.
We will be utilizing a Matlab code (see appendix D) for the remainder of this discussion.
Given a file that contains all the faces in the database, we will create a single database matrix
for this file in the following way:

1. Vectorize every matrix in the database.
2. Find the average face.
(a) Add all the matrices in the database and divide by the total number of matrices.
(b) Vectorize the resultant ”average” matrix.
3. Subtract the average face from every (vectorized) image.
4. Concatenate each subtracted quantity next to each other in a single matrix.
5. Scale the resultant matrix.
After this process is done, we will be left with a single matrix which is our new database. We can
then plot the average face:
Now we will examine the SVD of the database matrix. Let the database matrix, D, be equal
to USVT
. The leading vectors in the U matrix, the left singular vectors, contain the dominant
information in the database. The leading four left singular vectors plot the following images, or
”eigenfaces:”

These eigenfaces have the most dominant information for the faces in the database; indeed,
they resemble faces, and they each have facial features that one would see on many faces in the
database. These facial features ”build” the faces in the database, in way.
So now we have database of faces that we can access, and we have the relevant (dominant)
information used to construct these images. Using this information, then,we can take advantage of
these matrices to reconstruct other matrices (images) that are not in our database. Applying these
methods yields the following results for reconstructing images not in our database. The original
images are on the left; the reconstructed images are on the right.
While the reconstructions are not perfect representations of the original images, they are fair
representations of the images. The images were constructed from the database of images, so they
are not meant to be free from error. However, the resemblance is apparent.

3 PART III: Model Reduction of Large-Scale ODEs
We will now revisit approximating ordinary differential equations using the Trapezoidal method.
However, instead of the simple case we investigated in Part I (y = Ay), we will be examining the
full equation x = Ax + bu(t), where the sizes of A and b are much greater than before. Using
implicit solvers for these functions, then, have the potential to be very costly, so we want to reduce
the models if at all possible, while keeping our reduced models still accurate. Thus, we want a
reduced dynamical system of the form xr = Arxr + bru(t), where the system has been reduced
from order n to order r.
We will use the Trapezoidal Method to obtain X = [x0, x1, . . . , xN−1] ∈ Rn×N
. Using this
matrix, X, we will compute the SVD: X = USVT
, and create our reduced model using Ur, the
fist r columns of U:
xr = Arxr + bru(t) where Ar = UT
r AUr and br = UT
r b
Our solution, however, xr, has length r, where our original solution has length n. To rectify this,
we make our approximation to the original solution as follows: x(t) ≈ ˆx(t) = Urxr(t)
3.1 Problem 3.1
We will now apply our methodology for solving large-scale ODEs (called POD – Proper Orthogonal
Decomposition) to a specific case in which the order is n = 1412, i.e., A ∈ R1412×1412
, b ∈ R1412
.
Using POD, we will approximate the behavior of ISS 12A module using the A and b quantities
describing its differential equations. We will choose a constant input (u(t) = 1) and the initial
state will be zero (x(0) = 0), and we will use the Trapezoidal Method to approximate the function
on the interval t ∈ [0, 50] with a time step h = 0.1.
In order to begin, however, we must decide how accurate we want our approximation to be, and,
ultimately, what r-value we choose. To do this, we will again look at the relative error we examined
in Problem 2.5 using our information about matrix norms from Problem 2.1: X−Xk
X
= σk+1
σ1
. Thus,
if we want to generate an approximation Xr to X, it will have a relative error of σr+1
σ1
. We will
choose three values of relative error which we want to replicate: 10−1
, 10−2
, and 5 × 10−3
. The
r-values for these respective errors are r = 12, r = 36, and r = 43.
To approximate these functions, we will use a Matlab code (see appendix E) that uses the
Trapezoidal code as well (appendix B). Graphing these approximations against the original function
yields the following:

r = 12
r = 36
r = 43
So, as we expected, the more leading columns in U that we keep (the higher r is), the more accu-
rate our approximation to the original function. Indeed, when r = 43, the approximation is nearly

indistinguishable from the original function. Thus, this reduced model is a good approximation to
the original.
Now, we built our reduced model with the input u(t) = 1. Let us use this same model, keeping
r at 43, to approximate the way the system behaves when u(t) = sin (4t):
Indeed, even though we are approximating the system for a diﬀerent input than the input for
which we constructed the reduced model, the approximation is still incredibly accurate. This is
the power of POD.
3.2 Problem 3.2
We will now apply the same process in POD to a slightly diﬀerent system: a complex mass-spring
damper system which is of order n = 20000, i.e., A ∈ R20000×20000
, b ∈ R20000
. For this system,
we will choose u(t) = sin (5t), h = 0.01, and we will select an r such that the relative error is less
than 5 × 10−3
. Plotting the second component of the time-evolution for the original function in
Matlab (see appendix F) and the approximation yields:

Again, this result is incredibly accurate, and our reduced system models the full system very
well. As in the previous problem, we will use the reduced model built from the input u(t) = sin (5t)
to approximate the system for a diﬀerent input – in this case, u(t) = square(10t).
Once more the approximation and the original function are indistinguishable, and the POD
approximation provides an accurate representation of the behavior of the system.

Appendices
Appendix A
What follows is the Matlab code for approximating the scalar function using Euler’s Method.
TermProjectEuler1.m :
1 f = @(t,y) [−8003 1999; 23988 −6004]*[y(1);y(2)]; % y' = Ay.
2
3 h = 1.3571*10ˆ−4;
4 t0 = 0;
5 tf = .02;
6 y0 = [1;4];
7 [t,y] = euler(t0,tf,h,f,y0); % This uses a separate file; see below.
8 subplot(2,1,1)
9 plot(t,y(1,:));
10 grid on
11 subplot(2,1,2)
12 plot(t,y(2,:));
13 grid on
euler.m :
1 function [t,y] = euler(t0,tf,h,f,y0,k)
2 %input t0 initial time
3 %input tf final time
4 %input h size of time steps
5 %input f function so that y'=f(t,y)
6 %input y0 starting value for y
7 %input k if given, the y(k,:) to plot against t.
8 % if not given, nothing is plotted.
9
10 % Created by Christian Zinck
11
12 t = t0:h:tf;
13 y = zeros(length(y0),length(t));
14 y(:,1) = y0;
15
16 for i = 1:length(t)−1
17 y(:,i+1) = y(:,i) + h*f(t(i),y(:,i));
18 end
19
20
21 if nargin > 5
22 plot(t,y(k,:));
23 end
24 end

Appendix B
What follows is the Matlab code for approximating functions using the Trapezoidal Method. For
the scalar case, u=0 and b=0.
Trapezoidal.m :
1 function [t,y] = Trapezoidal(t0,tf,h,A,x,b,k)
2 %This function computes the Trapezoidal Method approximation
3 %for a function of the form y' = Ay + bu(t).
4
5 if nargin == 5 %For the case y' = Ay
6 b = 0;
7 u = @(t) 0;
8 end
9 m = isa(k,'function handle');
10 if m == 0 %Checks to see if k is a function.
11 u = @(t) k;
12 elseif m == 1
13 u = @(t) k(t);
14 end
15 [m n] = size(x);
16 if m == 1 && n == 1 && x == 0
17 y0 = zeros(length(A),1);
18 elseif m = 1 && n == 1
19 y0 = x;
20 elseif m==1 && n = 1
21 y0 = transpose(x);
22 else
23 error('Initial vector value needed.')
24 end
25
26 M = speye(length(A)) − h/2*(A);
27 [L U] = lu(M);
28 t = t0:h:tf;
29 y = zeros(length(y0),length(t));
30 y(:,1) = y0;
31
32 for i = 1:length(t)−1
33 z(:,i) = y(:,i) + h/2.*(A*y(:,i)+b.*u(t(i))+b.*u(t(i+1)));
34 v(:,i+1) = Lz(:,i);
35 y(:,i+1) = Uv(:,i+1);
36 end
37 if length(y0) ≤ 5
38 for i = 1:length(y0)
39 subplot(length(y0),1,i)
40 plot(t,y(i,:))
41 grid on
42 end
43 end
44 end

Appendix C
What follows is the Matlab code for computing a rank-k approximation of the Lena Matrix and
plotting the resulting image.
LenaSVD.m :
1 load 'LenaImage'
2 [U,S,V] = svd(A,0);
3 n = 409;
4 k = rank(A) − n;
5 Ak = zeros(size(A));
6 for i=1:k
7 Ak = Ak + S(i,i)*U(:,i)*transpose(V(:,i));
8 end
9 error = norm(A − Ak,2) / norm(A,2);
10 k
11 error
12 imshow(Ak)
13 image(Ak)
14 axis off

Appendix D
What follows is the Matlab code for creating a database of faces and reconstructing faces
YaleFace.m :
1 load 'yale faces'
2 N = length(Y);
3 Mave = zeros(size(Y{1})); %Makes a blank matrix that is the size of an ...
individual matrix in yale faces.mat
4 [n m] = size(Y{1});
5
6 for i=1:N
7 Mave = Mave + Y{i}; %Adds together all the matrices in yale faces.mat
8 end
9
10 Mave = (Mave)/N; %Average matrix in yale faces.mat
11 mave = reshape(Mave,n*m,1); %Vectorizes the average matrix
12
13 for i = 1:N
14 D(:,i) = reshape(Y{i},n*m,1) − mave; %Subtracting each vectorized matrix ...
in yale faces.mat from the vector of the average matrix
15 end
16
17 D = D/sqrt(N); %"Database" of Yale faces.
18 [U S V] = svd(D,0);
19 %for i = 1:4
20 %subplot(2,2,i)
21 %imagesc(reshape(U(:,i),n,m));colormap(gray) %This plots the eigenfaces
22 %end
23 load 'images to reconstruct'
24 F{1} = F1; F{2} = F2; F{3} = F3; %Store the three images in a cell block so ...
they can be accessed by index.
25
26 for i=1:length(F)
27 [m(i) n(i)] = size(F{i}); %Acquire the dimensions of the matrices to be ...
28 reconstructed
29
30 f{i} = reshape(F{i},m(i)*n(i),1) − mave; %Vectorize the matrices and ...
subtract the ...
31 average face.
32
33 p{i} = U*U'*f{i}+mave; %Construct the best approximation using our database.
34
35 P{i} = reshape(p{i},m(i),n(i)); %Unvectorize the vectors back into matrices.
36
37 subplot(3,2,2*i−1) %These lines plot the original images on the left
38 imagesc(F{i});colormap(gray) %and the database approximations on the right.
39 subplot(3,2,2*i)
40 imagesc(P{i});colormap(gray)
41 end

Appendix E
What follows is the Matlab code for applying POD to a system of rank n = 1412
POD 1.m :
1 load 'iss12a'
2 t0 = 0;
3 tf = 50;
4 h = 0.1;
5 k = @(t) 1;
6 [t,X] = Trapezoidal(t0,tf,h,A,0,b,k);
7 [U S V] = svd(X,0);
8
9 i = 1;
10 error = 5*10ˆ−3;
11 while S(i,i)/S(1,1) > error %This loop determines the error when u(t) = 1.
12 r=i; %If a different r−value is desired (say,
13 i = i+1; %r = 43 when u(t) = sin(4*t)), it must be
14 end %manually entered, and these lines must be deleted.
15
16 Ur = U(:,1:r);
17 Ar = Ur'*A*Ur;
18 br = Ur'*b;
19
20 [t,Xhat] = Trapezoidal(t0,tf,h,Ar,0,br,k);
21 Xhat = Ur*Xhat;
22
23 plot(t,X(1,:),'b')
24 hold on
25 plot(t,Xhat(1,:),'r')
26 grid on

Appendix F
What follows is the Matlab code for applying POD to a system of rank n = 20000
POD 2.m :
1 load 'msd20000'
2 t0 = 0;
3 tf = 5;
4 h = 0.01;
5 x0 = 0;
6 u1 = @(t) sin(5*t);
7 [t,X] = Trapezoidal(t0,tf,h,A,x0,b,u1);
8 [U S V] = svd(X,0);
9
10 i = 1;
11 error = 5*10ˆ−3;
12 while S(i,i)/S(1,1) > error
13 r=i;
14 i = i+1;
15 end
16
17 Ur = U(:,1:r);
18 Ar = Ur'*A*Ur;
19 br = Ur'*b;
20
21 u2 = @(t) square(10*t);
22
23 [t,X1] = Trapezoidal(t0,tf,h,A,x0,b,u2);
24 [t,X2] = Trapezoidal(t0,tf,h,Ar,0,br,u2);
25 Xhat = Ur*X2;
26
27 figure
28 plot(t,X1(2,:),'b')
29 hold on
30 plot(t,Xhat(2,:),'r')
31 grid on

Math Model Reduction Using SVD

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (18)

Similaire à Math Model Reduction Using SVD

Similaire à Math Model Reduction Using SVD (20)

Dernier

Dernier (20)

Math Model Reduction Using SVD