21. Underdetermined Systems

— CLEVE B. MOLER, The World's Simplest Impossible Problem (1990)

This problem arises in important algorithms used in mathematical programming … In these cases, B is usually very large and sparse and, because of storage difficulties, it is often uneconomical to store and access Q₁ … Sometimes it has been thought that [the seminormal equations method] could be disastrously worse than [the Q method] … It is the purpose of this note to show that such algorithms are numerically quite satisfactory.

— C. C. PAIGE, An Error Analysis of a Method for Solving Matrix Equations (1973)

Having considered well-determined and over determined linear systems, we now turn to the remaining class of linear systems: those that are underdetermined.

Consider the underdetermined system Ax = b, where A ∈ ℝ^{m × n} with m ≤ n. The system can be analysed using a QR factorizationwhere Q ∈ ℝ^{n × n} is orthogonal and R ∈ ℝ^{m × m} is upper triangular. (We could, alternatively, use an LQ factorization of A, but we will keep to the standard notation.) We havewhereIf A has full rank then y₁ = R^−Tb is uniquely determined and all solutions of Ax = b are given byThe unique solution x_LS that minimizes ‖x‖₂ is obtained by setting y₂ = 0. We havewhere A⁺ = A^T(AA^T)⁻¹ is the pseudo-inverse of A. Hence x_LS can be characterized as x_ls = A^Ty, where y solves the normal equations AA^Ty = b.

Equation (21.3) defines one way to compute x_LS. We will refer to this method as the “Q method”. When A is large and sparse it is desirable to avoid storing and accessing Q, which can be expensive. An alternative method with this property uses the QR factorization (21.1) but computes x_LS as x_LS = A^Ty, where(cf. (21.4)). These latter equations are called the seminormal equations (SNE). As the “semi” denotes, however, this method does not explicitly form AA^T, which would be undesirable from the standpoint of numerical stability. Note that equations (21.5) are different from the equations R^TRx = A^Tb for an overdetermined least squares (LS) problem, where A = Q[R^T 0]^T ∈ ℝ^{m × n} with m ≥ n, which are also called seminormal equations (see §20.6).