15. Condition Number Estimation

Most of LAPACK's condition numbers and error bounds are based on estimated condition numbers … The price one pays for using an estimated rather than an exact condition number is occasional (but very rare) underestimates of the true error; years of experience attest to the reliability of our estimators, although examples where they badly underestimate the error can be constructed.

— E. ANDERSON et al., LAPACK Users' Guide (1999)

The importance of the counter-examples is that they make clear that any effort toward proving that the algorithms always produce useful estimations is fruitless. It may be possible to prove that the algorithms produce useful estimations in certain situations, however, and this should be pursued. An effort simply to construct more complex algorithms is dangerous.

— A. K. CLINE and R. K. REW, A Set of Counter-Examples to Three Condition Number Estimators (1983)

Singularity is almost invariably a clue.

— SIR ARTHUR CONAN DOYLE, The Boscombe Valley Mystery (1892)

Condition number estimation is the process of computing an inexpensive estimate of a condition number, where “inexpensive” usually means that the cost is an order of magnitude less than would be required to compute the condition number exactly. Condition estimates are required when evaluating forward error bounds for computed solutions to many types of linear algebra problems. An estimate of the condition number that is correct to within a factor 10 is usually acceptable, because it is the magnitude of an error bound that is of interest, not its precise value. In this chapter we are concerned with condition numbers involving A⁻¹, where A is a given nonsingular n × n matrix; these arise in linear systems Ax = b but also more generally in, for example, least squares problems and eigenvalue problems.

An important question for any condition estimator is whether there exists a “counterexample” : a parametrized matrix for which the quotient “estimate divided by true condition number” can be made arbitrarily small (or large, depending on whether the estimate is a lower bound or an upper bound) by varying a parameter. Intuitively, it might be expected that any condition estimator has counterexamples or, equivalently, that estimating a condition number to within a constant factor independent of the matrix A must cost as much, asymptotically, as computing A⁻¹. This property was conjectured by Demmel [314, 1992] and strong evidence for it is provided by Demmel, Diament, and Malajovich [321, 2001]. In the latter paper it is shown that the cost of computing an estimate of ‖A⁻¹ ‖ of guaranteed quality is at least the cost of testing whether the product of two n × n matrices is zero, and performing this test is conjectured to cost as much as actually computing the product [185, 1997, Problem 16.3].

In the perturbation theory of Chapter 7 for linear equations we obtained perturbation bounds that involve the condition numbersand their variants. To compute these condition numbers exactly we need to compute A⁻¹, which requires O(n³) operations, even if we are given a factorization of A. Is it possible to produce reliable estimates of both condition numbers in O(n²) operations? The answer is yes, but to see why we first need to rewrite the expression for cond_{E, f}(A, x). Consider the general expression ‖ |Ais usually true that ‖ | $\hat{L}$ |‾‖Û| ‖_∞⁻¹|d‖_∞, where d is a given nonnegative vector (thus, d = f + E|x| for cond_{E, f}(A, x)); note that the practical error bound (7.31) is of this form. Writing D = diag(d) and e = [1,1, …, 1]^T, we have