Lecture 7: Quadratic optimization

Transcript

1 Lecture 7: Quadratic optimization 1. Positive definite och semidefinite matrices 2. LDL T factorization 3. Quadratic optimization without constraints 4. Quadratic optimization with constraints 5. Least-squares problems Föreläsning 7 1 Quadratic optimization - Ulf Jönsson & Per Enqvist

2 Positive definite and positive semi-definite matrices Definition 1 (Positive semidefinite). A symmetric matrix H = H T R n n is called positive semidefinite if x T Hx 0 for all x R n Definition 2 (Positivt definit). A symmetric matrix H = H T R n n is called positive definite if x T Hx > 0 for all x R n {0} Föreläsning 7 2 Quadratic optimization - Ulf Jönsson & Per Enqvist

3 How do you determine if a symmetric matrix H = H T R n n is positive (semi-) definite: LDL T factorization (suitable for calculations by hand). Use the spectral factorization of the matrix H = QΛQ T, where Q T Q = I and Λ = diag(λ 1,...,λ n ) where λ k are the eigenvalues of the matrix. From this it follows that H is positive semidefinite if and only if λ k 0, k = 1,...,n H is positive definite if and only if λ k > 0, k = 1,...,n Föreläsning 7 3 Quadratic optimization - Ulf Jönsson & Per Enqvist

4 LDL T faktorization Theorem 1. A symmetric matrix H = H T R n n is positive semidefinite if and only if there is a factorization H = LDL T where d l d L = l 31 l D = 0 0 d l n1 l n2 l n d n and d k 0, k = 1,...,n. H is positive definite if and only if d k > 0, k = 1,...,n, in the LDL T factorization. Föreläsning 7 4 Quadratic optimization - Ulf Jönsson & Per Enqvist

5 The factorization LDL T is determined by one of the following methods Elementary row and column operations, Completion of squares, Föreläsning 7 5 Quadratic optimization - Ulf Jönsson & Per Enqvist

6 Quadratic optimization without constraints minimize 1 2 xt Hx + c T x + c 0 where H = H T R n n, c R n, c 0 R. Common in many applications, e.g. linear regression (model fitting) minimization of physically derived objective functions, e.g., minimization of energy, variance, e.t.c. Quadratic approximation of nonlinear optimization problems Föreläsning 7 6 Quadratic optimization - Ulf Jönsson & Per Enqvist

7 Unbounded problems Let f(x) = 1 2 xt Hx + c T x + c 0. Assume that H is not positive semidefinite. Then there is a d R n such that d T Hd < 0. It follows that f(td) = 1 2 t2 d T Hd + tc T d + c 0 when t Conclusion: A necessary condition for f(x) to have a minimum is that H is positive semidefinite. Föreläsning 7 7 Quadratic optimization - Ulf Jönsson & Per Enqvist

8 Assume that H is postive semidefinite. Then we can use the factorization H = LDL T which gives minimize 1 2 xt Hx + c T x + c 0 = minimize 1 2 xt LDL T x + c T (L T ) 1 L T x + c 0 = minimize = c 0 + n k=1 1 2 d ky 2 k + c k y k + c 0 n minimize 1 2 d kyk 2 + c k y k k=1 where we have introduced the new coordinates y = L T x and c = L 1 c. The optimization problem separates to n independent scalar quadratic optimization problems, which are easy to solve. Föreläsning 7 8 Quadratic optimization - Ulf Jönsson & Per Enqvist

9 Scalar Quadratic optimization without constraints The scalar quadratic optimization problem has a finite solution, if and only if, h = 0 and c = 0, or minimize x R 1 2 hx2 + cx + c 0 h > 0. In the first case there is no unique optimum. In the second case 1 2 hx2 + cx + c 0 = 1 2 h(x + c/h)2 + c 0 c 2 /(2h), and the optimum is achieved for x = c/h. Föreläsning 7 9 Quadratic optimization - Ulf Jönsson & Per Enqvist

10 Solving the minimization problems separately shows that [ ] T ŷ = ŷ 1... ŷ n is a minimizer if (i) d k 0, k = 1,...,n (ii) d k ŷ k = c k (i) H is positive semidefinite (ii) DL Tˆx = L 1 c Constraint (ii) can be replaced with LDL Tˆx = c i.e., Hˆx = c. Furthermore, f(x) = 1 2 xt Hx + c T x + c 0 is unbounded (from below) if (i) (ii) some d k < 0, or one of the equations d k ŷ k = c k do not have a solution i.e., d k = 0 and c k 0 (i) H is not positive semi-definite (ii) the equation Hˆx = c do not have a solution Föreläsning 7 10 Quadratic optimization - Ulf Jönsson & Per Enqvist

11 The above arguments can be formalized as the following theorem Theorem 2. Let f(x) = 1 2 xt Hx + c T x + c 0. Then ˆx is a minimizer if (i) H is positive semidefinite (ii) Hˆx = c If H is positive definite then ˆx = H 1 c. If H is not positive semidefinite or Hx = c do not have a solution, then f unlimited from below.. Comment 1. The condition that Hx = c has a solution is equivalent to c R(H). Therefore, f is a minimizer if and only if H is positive semidefinite and c R(H). Föreläsning 7 11 Quadratic optimization - Ulf Jönsson & Per Enqvist

12 Quadratic optimization with linear constraints 1 minimize 2 xt Hx + c T x + c 0 s.t. Ax = b (1) where H = H T R n n, A R m n, c R n, and b R m, and c 0 R. We assume that b R(A) and n > m, i.e., N(A) {0}. Assume that N(A) = span{z ] 1,...,z k } and define the nullspace matrix Z = [z 1... z k. If A x = b it holds that an arbitrary solution to the linear constraint has the form x = x + Zv, for some v R k. This follows since A( x + Zv) = A x + AZv = b + 0 = b Föreläsning 7 12 Quadratic optimization - Ulf Jönsson & Per Enqvist

13 With x = x + Zv inserted in f(x) = 1 2 xt Hx + c T x + c 0 we get f(x) = 1 2 vt Z T HZv + (Z T (H x + c)) T v + f( x). The minimization problem (1) is thus equivalent with minimize 1 2 vt Z T HZv + (Z T (H x + c)) T v + f( x). We can now apply Theorem 2 to obtain the following result: Theorem 3. ˆx is an optimal solution to (1) if (i) Z T HZ is positive semidefinite. (ii) ˆx = x + Zˆv where Z T HZˆv = Z T (H x + c) Föreläsning 7 13 Quadratic optimization - Ulf Jönsson & Per Enqvist

14 Comment 2. The second condition in the theorem is equivalent with the existence of a û R m such that H AT A 0 ˆx = û c b Proof: The second constraint in the theorem can be written Z T (H( x + Zˆv) + c) = 0 Since N(A) = R(Z) this is equivalent with H( x } + {{ Zˆv } ) + c N(A) = R(A T ) ˆx Hˆx + c = A T û, for some û R m Föreläsning 7 14 Quadratic optimization - Ulf Jönsson & Per Enqvist

15 Example: Analysis of a resistance circuit I u 1 x 12 x 13 v 13 r 13 u 3 x 34 x 35 v 35 v 12 r 12 r 23 r 34 r 35 v 34 u 5 x 23 v 23 r 45 r 24 u 2 x 24 v 24 u 4 x 45 v 45 Let a current I go through the circuit from node 1 to node 5. What is the solution to the circuit equation? Föreläsning 7 15 Quadratic optimization - Ulf Jönsson & Per Enqvist

16 The current x ij goes from node i to node j if x ij > 0. Kirchoff s currentlaw gives This can be written Ax = b where x = x 12 + x 13 = I x 12 x 23 x 24 = 0 x 13 + x 23 x 34 x 35 = 0 x 24 + x 34 x 45 = 0 x 35 + x 45 = I [ x 12 x 13 x 23 x 24 x 34 x 35 x 45 ] T, I A = , b = I Föreläsning 7 16 Quadratic optimization - Ulf Jönsson & Per Enqvist

17 The Node potentials are denoted u j, j = 1,...,5 The voltage drop over the resistor r ij is denoted v ij and is given by the equations v ij = u i u j v ij = r ij x ij (Ohms law) This can also be written v = A T u v = Dx where [ ] T ] T u = u 1 u 2 u 3 u 4 u 5 v = [v 12 v 13 v 23 v 24 v 34 v 35 v 45 D = diag(r 12,r 13,r 23,r 24,r 34,r 35,r 45 ) Föreläsning 7 17 Quadratic optimization - Ulf Jönsson & Per Enqvist

18 The circuit equations: Ax = b v = A T u v = Dx D AT A 0 x = 0 b Since D is positive definite (r ij > 0) this is, see comment 2, equivalent to the optimality conditions for the following optimization problem 1 minimize 2 xt Dx s.t. Ax = b Föreläsning 7 18 Quadratic optimization - Ulf Jönsson & Per Enqvist

19 Comment In the lecture on network optimization we saw that the matrix N(A) 0 (then the matrix was called Ã) and that the nullspace corresponded to cycles in the circuit. The laws of electricity (Ohms law + Kirchoffs law) determines the current that minimizes the loss of power. This follows since the objective function can be written 1 2 xt Dx = 1 2 r ij x 2 ij ij Föreläsning 7 19 Quadratic optimization - Ulf Jönsson & Per Enqvist

20 Comment 3. It is common to connect one of the nodes to earth. Let us, e.g., connect node 5 to earth, then u 5 = 0. I u 1 x 12 x 13 v 13 r 13 u 3 x 34 x 35 v 35 v 12 r 12 x 23 r 23 v 23 r 34 r 35 v 34 r 45 u 5 = 0 u 2 x 24 r 24 v 24 u 4 x 45 v 45 Föreläsning 7 20 Quadratic optimization - Ulf Jönsson & Per Enqvist

21 With the potential in node 5 fixed at u 5 = 0 the last column in the voltage equation v = A T u is eliminated. The Voltage equation and the current balance can be replaced with v = Ā T ū and Āx = b where I u u 2 Ā =, b =, ū = u The advantage with this is that the rows of Ā are linearly independent. This facilitates the solution of the optimization problem. u 4 Föreläsning 7 21 Quadratic optimization - Ulf Jönsson & Per Enqvist

22 Least-Squares problems Application: Linear regression (model fitting) e(t) s(t) y(t) System Problem: Fit a linear regressionsmodel to measured data. n Regressionmodel : y(t) = α j ψ j (t) ψ k (t), k = 1,...,n are the regressors (known functions) α k = k = 1,...,n are the model parameters (to be determined) e(t) measurement noise (not known) j=1 s(t) observations. Föreläsning 7 22 Quadratic optimization - Ulf Jönsson & Per Enqvist

23 Idéa for fitting the model: Minimize the quadratic sum of the prediction errors minimize 1 m n ( α j ψ j (t i ) s(t i )) 2 2 =minimize 1 2 i=1 j=1 m (ψ(t i ) T x s(t i )) 2 i=1 =minimize 1 2 (Ax b)t (Ax b) (2) ψ(t) = ψ 1 (t)., x = ψ n (t) α 1. α n, A = ψ(t 1 ) T., b = ψ(t n ) T s(t 1 ). s(t n ) Föreläsning 7 23 Quadratic optimization - Ulf Jönsson & Per Enqvist

24 The solution of the Least-Squares problem (LSQ) The Least-Squares problem (2) can equivalently be written minimize 1 2 xt Hx + c T x + c 0 where H = A T A, c = A T b, c 0 = 1 2 bt b. We note that H = A T A is positive semidefinite since x T A T Ax = Ax 2 0. c R(H) since c = A T b R(A T ) = R(A T A) = R(H). The conditions in Theorem 2 and Comment 1 are satisfied and it follows that the LSQ-estimate is given by Hˆx = c A T Aˆx = A T b (The Normal equation) Föreläsning 7 24 Quadratic optimization - Ulf Jönsson & Per Enqvist

25 Linear algebra interpretation N(A) N(A T ) R n A R m x n x b Aˆx ˆx = A T u R(A T ) Aˆx R(A) A T (b Aˆx) = 0 b Aˆx N(A T ) The Normal equationen can be written This orthogonality property has a geometric interpretation which is illustrated in the figure above. The Fundamental Theorem of Linear algebra explains the LSQ solution geometrically. Föreläsning 7 25 Quadratic optimization - Ulf Jönsson & Per Enqvist

26 Uniqueness of the LSQ solution Case 1 If the columns in A are linearly independent,i.e., N(A) = {0}, it holds that H = A T A is positive definite and hence invertible. The Normal equations then has the unique solution ˆx = (A T A) 1 Ab Case 2 If A has linearly dependent columns, i.e., N(A) {0} then it is natural to chose the solution of smallest length. From the figure on the previous slide it is clear that we should let ˆx = A T û, where AA T û = A x, and where x is an arbitrary solution to the LSQ problem. Föreläsning 7 26 Quadratic optimization - Ulf Jönsson & Per Enqvist

27 Our choice of ˆx in case 2 can equivalently be interpreted as the solution to the quadratic optimization problem 1 minimize 2 xt x s.t. Ax = A x According to Comment 2 the solution to this optimization problem is given by the solution to I AT ˆx = A 0 û i.e. ˆx = A T û, where AA T û = A x 0 A x Föreläsning 7 27 Quadratic optimization - Ulf Jönsson & Per Enqvist