Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions

Abstract

Low-rank matrix approximations, such as the truncated singular value decomposition and the rank-revealing QR decomposition, play a central role in data analysis and scientific computing. This work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low-rank matrix approximation. These techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets. This paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions. These methods use random sampling to identify a subspace that captures most of the action of a matrix. The input matrix is then compressed—either explicitly or implicitly—to this subspace, and the reduced matrix is manipulated deterministically to obtain the desired low-rank factorization. In many cases, this approach beats its classical competitors in terms of accuracy, robustness, and/or speed. These claims are supported by extensive numerical experiments and a detailed error analysis. The specific benefits of randomized techniques depend on the computational environment. Consider the model problem of finding the k dominant components of the singular value decomposition of an $m \times n$ matrix. (i) For a dense input matrix, randomized algorithms require $\bigO(mn \log(k))$ floating-point operations (flops) in contrast to $ \bigO(mnk)$ for classical algorithms. (ii) For a sparse input matrix, the flop count matches classical Krylov subspace methods, but the randomized approach is more robust and can easily be reorganized to exploit multiprocessor architectures. (iii) For a matrix that is too large to fit in fast memory, the randomized techniques require only a constant number of passes over the data, as opposed to $\bigO(k)$ passes for classical algorithms. In fact, it is sometimes possible to perform matrix approximation with a single pass over the data.

MSC codes

  1. 65F30
  2. 68W20
  3. 60B20

Keywords

  1. dimension reduction
  2. eigenvalue decomposition
  3. interpolative decomposition
  4. Johnson–Lindenstrauss lemma
  5. matrix approximation
  6. parallel algorithm
  7. pass-efficient algorithm
  8. principal component analysis
  9. randomized algorithm
  10. random matrix
  11. rank-revealing QR factorization
  12. singular value decomposition
  13. streaming algorithm

Get full access to this article

View all available purchase options and get full access to this article.

References

1.
D. Achlioptas, Database-friendly random projections: Johnson–Lindenstrauss with binary coins, J. Comput. System Sci., 66 (2003), pp. 671–687.
2.
D. Achlioptas and F. McSherry, Fast computation of low-rank matrix approximations, J. Assoc. Comput. Mach., 54 (2007), article 9.
3.
N. Ailon and B. Chazelle, Approximate nearest neighbors and the fast Johnson–Lindenstrauss transform, in Proceedings of the 38th Annual ACM Symposium on Theory of Computing (STOC '06), 2006, pp. 557–563.
4.
N. Ailon and E. Liberty, Fast dimension reduction using Rademacher series on dual BCH codes, in Proceedings of the 40th Annual ACM Symposium on Theory of Computing (STOC '08), 2008, pp. 1–9.
5.
N. Alon, P. Gibbons, Y. Matias, and M. Szegedy, Tracking join and self-join sizes in limited storage, in Proceedings of the 18th ACM Symposium on Principles of Database Systems (PODS), 1999, pp. 10–20.
6.
N. Alon, Y. Matias, and M. Szegedy, The space complexity of approximating frequency moments, in Proceedings of the 28th Annual ACM Symposium on Theory of Computing (STOC '96), 1996, pp. 20–29.
7.
S. Arora, E. Hazan, and S. Kale, A fast random sampling algorithm for sparsifying matrices, in Approximation, Randomization and Combinatorial Optimization: Algorithms and Techniques, Springer, Berlin, 2006, pp. 272–279.
8.
A. R. Barron, Universal approximation bounds for superpositions of a sigmoidal function, IEEE Trans. Inform. Theory, 39 (1993), pp. 930–945.
9.
I. Beichl, The Metropolis algorithm, Comput. Sci. Eng., 2 (2000), pp. 65–69.
10.
M.-A. Bellabas and P. J. Wolfe, On sparse representations of linear operators and the approximation of matrix products, in Proceedings of the 42nd Annual Conference on Information Sciences and Systems (CISS), 2008, pp. 258–263.
11.
R. Bhatia, Matrix Analysis, Grad. Texts in Math. 169, Springer, Berlin, 1997.
12.
Å. Björck, Numerics of Gram–Schmidt orthogonalization, Linear Algebra Appl., 197–198 (1994), pp. 297–316.
13.
Å. Björck, Numerical Methods for Least Squares Problems, SIAM, Philadelphia, PA, 1996.
14.
V. Bogdanov, Gaussian Measures, AMS, Providence, RI, 1998.
15.
J. Bourgain, On Lipschitz embedding of finite metric spaces in Hilbert space, Israel J. Math., 52 (1985), pp. 46–52.
16.
C. Boutsidis and P. Drineas, Random projections for nonnegative least squares, Linear Algebra Appl., 431 (2009), pp. 760–771.
17.
C. Boutsidis, M. W. Mahoney, and P. Drineas, An improved approximation algorithm for the column subset selection problem, in Proceedings of the 20th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 2009, pp. 968–977.
18.
C. Boutsidis, M. W. Mahoney, and P. Drineas, Unsupervised feature selection for principal components analysis, in Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2008, pp. 61–69.
19.
E. Candès and J. K. Romberg, Sparsity and incoherence in compressive sampling, Inverse Problems, 23 (2007), pp. 969–985.
20.
E. Candès, J. K. Romberg, and T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete Fourier information, IEEE Trans. Inform. Theory, 52 (2006), pp. 489–509.
21.
E. J. Candès, Compressive sampling, in Proceedings of the 2006 International Congress of Mathematicians, Madrid, 2006.
22.
E. J. Candès and B. Recht, Exact matrix completion via convex optimization, Found. Comput. Math., 9 (2009), pp. 717–772.
23.
E. J. Candès and T. Tao, The power of convex relaxation: Near-optimal matrix completion, IEEE Trans. Inform. Theory, 56 (2010), pp. 2053–2080.
24.
B. Carl, Inequalities of Bernstein–Jackson-type and the degree of compactness in Banach spaces, Ann. Inst. Fourier (Grenoble), 35 (1985), pp. 79–118.
25.
Z. Chen and J. J. Dongarra, Condition numbers of Gaussian random matrices, SIAM J. Matrix Anal. Appl., 27 (2005), pp. 603–620.
26.
H. Cheng, Z. Gimbutas, P. G. Martinsson, and V. Rokhlin, On the compression of low rank matrices, SIAM J. Sci. Comput., 26 (2005), pp. 1389–1404.
27.
A. Çivril and M. Magdon-Ismail, On selecting a maximum volume sub-matrix of a matrix and related problems, Theoret. Comput. Sci., 410 (2009), pp. 4801–4811.
28.
K. L. Clarkson, Subgradient and sampling algorithms for $\ell_1$ regression, in Proceedings of the 16th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 2005, pp. 257–266.
29.
K. L. Clarkson and D. P. Woodruff, Numerical linear algebra in the streaming model, in Proceedings of the 41st Annual ACM Symposium on Theory of Computing (STOC '09), 2009, pp. 205–214.
30.
R. R. Coifman, S. Lafon, A. B. Lee, M. Maggioni, B. Nadler, F. Warner, and S. W. Zucker, Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps, Proc. Natl. Acad. Sci. USA, 102 (2005), pp. 7426–7431.
31.
A. Dasgupta, P. Drineas, B. Harb, R. Kumar, and M. W. Mahoney, Sampling algorithms and coresets for $\ell_p$ regression, SIAM J. Comput., 38 (2009), pp. 2060–2078.
32.
S. Dasgupta and A. Gupta, An Elementary Proof of the Johnson–Lindenstrauss Lemma, Tech. Report 99-006, University of California at Berkeley, 1999.
33.
A. d'Aspremont, Subsampling Algorithms for Semidefinite Programming, preprint, 2009; available online from http://arxiv.org/abs/0803.1990.
34.
K. R. Davidson and S. J. Szarek, Local operator theory, random matrices, and Banach spaces, in Handbook of Banach Space Geometry, W. B. Johnson and J. Lindenstrauss, eds., Elsevier, 2002, pp. 317–366.
35.
J. Demmel, I. Dumitriu, and O. Holtz, Fast linear algebra is stable, Numer. Math., 108 (2007), pp. 59–91.
36.
A. Deshpande and L. Rademacher, Efficient Volume Sampling for Row/Column Subset Selection, preprint, 2010; available online from http://arxiv.org/abs/1004.4057.
37.
A. Deshpande, L. Rademacher, S. Vempala, and G. Wang, Matrix approximation and projective clustering via volume sampling, in Proceedings of the 17th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 2006, pp. 1117–1126.
38.
A. Deshpande and S. Vempala, Adaptive sampling and fast low-rank matrix approximation, in Approximation, Randomization and Combinatorial Optimization, J. Díaz, K. Jansen, J. D. P. Rolim, and U. Zwick, eds., Lecture Notes in Comput. Sci. 4110, Springer, Berlin, 2006, pp. 292–303.
39.
J. D. Dixon, Estimating extremal eigenvalues and condition numbers of matrices, SIAM J. Numer. Anal., 20 (1983), pp. 812–814.
40.
J. Dongarra and F. Sullivan, The top $10$ algorithms, Comput. Sci. Eng., 2 (1) (2000), pp. 22–23.
41.
D. L. Donoho, Compressed sensing, IEEE Trans. Inform. Theory, 52 (2006), pp. 1289–1306.
42.
D. L. Donoho, M. Vetterli, R. A. DeVore, and I. Daubechies, Data compression and harmonic analysis, IEEE Trans. Inform. Theory, 44 (1998), pp. 2433–2452.
43.
P. Drineas, A. Frieza, R. Kannan, S. Vempala, and V. Vinay, Clustering of large graphs via the singular value decomposition, Machine Learning, 56 (2004), pp. 9–33.
44.
P. Drineas, A. Frieze, R. Kannan, S. Vempala, and V. Vinay, Clustering in large graphs and matrices, in Proceedings of the 10th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 1999, pp. 291–299.
45.
P. Drineas, R. Kannan, and M. W. Mahoney, Fast Monte Carlo algorithms for matrices. I. Approximating matrix multiplication, SIAM J. Comput., 36 (2006), pp. 132–157.
46.
P. Drineas, R. Kannan, and M. W. Mahoney, Fast Monte Carlo algorithms for matrices. II. Computing a low-rank approximation to a matrix, SIAM J. Comput., 36 (2006), pp. 158–183.
47.
P. Drineas, R. Kannan, and M. W. Mahoney, Fast Monte Carlo algorithms for matrices. III. Computing a compressed approximate matrix decomposition, SIAM J. Comput., 36 (2006), pp. 184–206.
48.
P. Drineas and M. W. Mahoney, On the Nyström method for approximating a Gram matrix for improved kernel-based learning, J. Mach. Learn. Res., 6 (2005), pp. 2153–2175.
49.
P. Drineas and M. W. Mahoney, A randomized algorithm for a tensor-based generalization of the singular value decomposition, Linear Algebra Appl., 420 (2007), pp. 553–571.
50.
P. Drineas, M. W. Mahoney, and S. Muthukrishnan, Subspace sampling and relative-error matrix approximation: Column-based methods, in Approximation, Randomization and Combinatorial Optimization, J. Díaz, K. Jansen, J. D. P. Rolim, and U. Zwick, eds., Lecture Notes in Comput. Sci. 4110, Springer, Berlin, 2006, pp. 321–326.
51.
P. Drineas, M. W. Mahoney, and S. Muthukrishnan, Relative-error CUR matrix decompositions, SIAM J. Matrix Anal. Appl., 30 (2008), pp. 844–881.
52.
P. Drineas, M. W. Mahoney, S. Muthukrishnan, and T. Sarlós, Faster least squares approximation, Numer. Math., 117 (2011), pp 219–249.
53.
A. Dvoretzky, Some results on convex bodies and Banach spaces, in Proceedings of the International Symposium on Linear Spaces, Jerusalem Academic Press, Jerusalem, Pergamon, Oxford, 1961, pp. 123–160.
54.
C. Eckart and G. Young, The approximation of one matrix by another of lower rank, Psychometrika, 1 (1936), pp. 211–218.
55.
A. Edelman, Eigenvalues and Condition Numbers of Random Matrices, Ph.D. thesis, Massachusetts Institute of Technology, Boston, MA, 1989.
56.
B. Engquist and O. Runborg, Wavelet-based numerical homogenization with applications, in Multiscale and Multiresolution Methods: Theory and Applications, T. J. Barth, T. Chan, and R. Haimes, eds., Lect. Notes Comput. Sci. Eng. 20, Springer, Berlin, 2001, pp. 97–148.
57.
A. Frieze, R. Kannan, and S. Vempala, Fast Monte Carlo algorithms for finding low-rank approximations, in Proceedings of the 39th Annual IEEE Symposium on Foundations of Computer Science (FOCS), 1998, pp. 370–378.
58.
A. Frieze, R. Kannan, and S. Vempala, Fast Monte Carlo algorithms for finding low-rank approximations, J. Assoc. Comput. Mach., 51 (2004), pp. 1025–1041.
59.
A. Y. Garnaev and E. D. Gluskin, The widths of a Euclidean ball, Dokl. Akad. Nauk. SSSR, 277 (1984), pp. 1048–1052 (in Russian).
60.
A. Gittens and J. A. Tropp, Error Bounds for Random Matrix Approximation Schemes, preprint, 2009; available online from http://arxiv.org/abs/0911.4108.
61.
G. H. Golub and C. F. Van Loan, Matrix Computations, 3rd ed., Johns Hopkins University Press, Baltimore, MD, 1996.
62.
Y. Gordon, Some inequalities for Gaussian processes and applications, Israel J. Math., 50 (1985), pp. 265–289.
63.
Y. Gordon, Gaussian processes and almost spherical sections of convex bodies, Ann. Probab., 16 (1988), pp. 180–188.
64.
S. A. Goreinov, E. E. Tyrtyshnikov, and N. L. Zamarashkin, Theory of pseudo-skeleton matrix approximations, Linear Algebra Appl., 261 (1997), pp. 1–21.
65.
L. Grasedyck and W. Hackbusch, Construction and arithmetics of $\mathcal{H}$-matrices, Computing, 70 (2003), pp. 295–334.
66.
L. Greengard and V. Rokhlin, A new version of the fast multipole method for the Laplace equation in three dimensions, Acta Numer., 17 (1997), pp. 229–269.
67.
M. Gu, Personal communication, 2007.
68.
M. Gu and S. C. Eisenstat, Efficient algorithms for computing a strong rank-revealing QR factorization, SIAM J. Sci. Comput., 17 (1996), pp. 848–869.
69.
N. Halko, P.-G. Martinsson, Y. Shkolnisky, and M. Tygert, An Algorithm for the Principal Component Analysis of Large Data Sets, preprint, 2010; available online from http://arxiv.org/abs/1007.5510.
70.
S. Har-Peled, Matrix Approximation in Linear Time, manuscript, 2006; available online from http://valis.cs.uiuc.edu/~sariel/research/papers/05/lrank/.
71.
T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed., Springer, Berlin, 2008.
72.
R. A. Horn and C. R. Johnson, Matrix Analysis, Cambridge University Press, Cambridge, UK, 1985.
73.
P. Indyk and R. Motwani, Approximate nearest neighbors: Toward removing the curse of dimensionality, in Proceedings of the 30th Annual ACM Symposium on Theory of Computing (STOC '98), 1998, pp. 604–613.
74.
W. B. Johnson and J. Lindenstrauss, Extensions of Lipschitz mappings into a Hilbert space, Contemp. Math., 26 (1984), pp. 189–206.
75.
P. W. Jones, A. Osipov, and V. Rokhlin, A Randomized Approximate Nearest Neighbors Algorithm, Tech. Report YALEU/DCS/RR-1434, Yale University, New Haven, CT, 2010.
76.
D. R. Karger, Random sampling in cut, flow, and network design problems, Math. Oper. Res., 24 (1999), pp. 383–413.
77.
D. R. Karger, Minimum cuts in near-linear time, J. ACM, 47 (2000), pp. 46–76.
78.
B. S. Kašin, On the widths of certain finite-dimensional sets and classes of smooth functions, Izv. Akad. Nauk. SSSR Ser. Mat., 41 (1977), pp. 334–351, 478 (in Russian).
79.
J. Kleinberg, Two algorithms for nearest neighbor search in high dimensions, in Proceedings of the 29th ACM Symposium on Theory of Computing (STOC '97), 1997, pp. 599–608.
80.
J. Kuczyński and H. Woźniakowski, Estimating the largest eigenvalue by the power and Lanczos algorithms with a random start, SIAM J. Matrix Anal. Appl., 13 (1992), pp. 1094–1122.
81.
E. Kushilevitz, R. Ostrovski, and Y. Rabani, Efficient search for approximate nearest neighbor in high dimensional spaces, SIAM J. Comput., 30 (2000), pp. 457–474.
82.
D. Le and D. S. Parker, Using randomization to make recursive matrix algorithms practical, J. Funct. Programming, 9 (1999), pp. 605–624.
83.
M. Ledoux, The Concentration of Measure Phenomenon, Math. Surveys Monogr. 89, AMS, Providence, RI, 2001.
84.
M. Ledoux and M. Talagrand, Probability in Banach Spaces: Isoperimetry and Processes, Springer, Berlin, 1991.
85.
W. S. Lee, P. L. Bartlett, and R. C. Williamson, Efficient agnostic learning of neural networks with bounded fan-in, IEEE Trans. Inform. Theory, 42 (1996), pp. 2118–2132.
86.
Z. Leyk and H. Woźniakowski, Estimating the largest eigenvector by Lanczos and polynomial algorithms with a random start, Numer. Linear Algebra Appl., 5 (1998), pp. 147–164.
87.
E. Liberty, Accelerated Dense Random Projections, Ph.D. thesis, Yale University, New Haven, CT, 2009.
88.
E. Liberty, N. Ailon, and A. Singer, Dense fast random projections and lean Walsh transforms, in Approximation, Randomization and Combinatorial Optimization, A. Goel, K. Jansen, J. Rolim, and R. Rubinfeld, eds., Lecture Notes in Comput. Sci. 5171, Springer, Berlin, 2008, pp. 512–522.
89.
E. Liberty, F. F. Woolfe, P.-G. Martinsson, V. Rokhlin, and M. Tygert, Randomized algorithms for the low-rank approximation of matrices, Proc. Natl. Acad. Sci. USA, 104 (2007), pp. 20167–20172.
90.
M. W. Mahoney and P. Drineas, CUR matrix decompositions for improved data analysis, Proc. Natl. Acad. Sci. USA, 106 (2009), pp. 697–702.
91.
P.-G. Martinsson, V. Rokhlin, Y. Shkolnisky, and M. Tygert, ID: A Software Package for Low-Rank Approximation of Matrices via Interpolative Decompositions, Ver. 0.2, http://cims.nyu.edu/~tygert/software.html, 2008.
92.
P.-G. Martinsson, V. Rokhlin, and M. Tygert, A randomized algorithm for the decomposition of matrices, Appl. Comput. Harmon. Anal., 30 (2011), pp. 47–68.
93.
P.-G. Martinsson, A. Szlam, and M. Tygert, Normalized Power Iterations for the Computation of SVD, manuscript, 2010.
94.
J. Matoušek, Lectures on Discrete Geometry, Springer, Berlin, 2002.
95.
F. McSherry, Spectral Methods in Data Analysis, Ph.D. thesis, University of Washington, Seattle, WA, 2004.
96.
N. Metropolis and S. Ulam, The Monte Carlo method, J. Amer. Statist. Assoc., 44 (1949), pp. 335–341.
97.
V. D. Milman, A new proof of A. Dvoretsky's theorem on cross-sections of convex bodies, Funkcional. Anal. i Priložen., 5 (1971), pp. 28–37.
98.
L. Mirsky, Symmetric gauge functions and unitarily invariant norms, Quart. J. Math. Oxford Ser. (2), 11 (1960), pp. 50–59.
99.
R. Motwani and P. Raghavan, Randomized Algorithms, Cambridge University Press, Cambridge, UK, 1995.
100.
R. J. Muirhead, Aspects of Multivariate Statistical Theory, Wiley, New York, 1982.
101.
S. Muthukrishnan, Data Streams: Algorithms and Applications, now Publishers, Boston, MA, 2005.
102.
D. Needell, Randomized Kaczmarz solver for noisy linear systems, BIT, 50 (2010), pp. 395–403.
103.
N. H. Nguyen, T. T. Do, and T. D. Tran, A fast and efficient algorithm for low-rank approximation of a matrix, in Proceedings of the 41st Annual ACM Symposium on Theory of Computing (STOC '09), 2009, pp. 215–224.
104.
C.-T. Pan, On the existence and computation of rank-revealing LU factorizations, Linear Algebra Appl., 316 (2000), pp. 199–222.
105.
C. H. Papadimitriou, P. Raghavan, H. Tamaki, and S. Vempala, Latent semantic indexing: A probabilistic analysis, in Proceedings of the 17th ACM Symposium on Principles of Database Systems (PODS), 1998, pp. 159–168.
106.
C. H. Papadimitriou, P. Raghavan, H. Tamaki, and S. Vempala, Latent semantic indexing: A probabilistic analysis, J. Comput. System Sci., 61 (2000), pp. 217–235.
107.
D. S. Parker and B. Pierce, The Randomizing FFT: An Alternative to Pivoting in Gaussian Elimination, Tech. Report CSD 950037, University of California at Los Angeles, 1995.
108.
P. J. Phillips, H. Moon, S. Rizvi, and P. Rauss, The FERET evaluation methodology for face recognition algorithms, IEEE Trans. Pattern Anal. Mach. Intelligence, 22 (2000), pp. 1090–1104.
109.
P. J. Phillips, H. Wechsler, J. Huang, and P. Rauss, The FERET database and evaluation procedure for face recognition algorithms, Image Vision Comput., 16 (1998), pp. 295–306.
110.
W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes: The Art of Scientific Computing, 3rd ed., Cambridge University Press, Cambridge, UK, 2007.
111.
A. Rahimi and B. Recht, Random features for large-scale kernel machines, in Proceedings of the 21st Annual Conference on Advances in Neural Information Processing Systems (NIPS), 2007.
112.
B. Recht, M. Fazel, and P. A. Parillo, Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization, SIAM Rev., 52 (2010), pp. 471–501.
113.
V. Rokhlin, A. Szlam, and M. Tygert, A randomized algorithm for principal component analysis, SIAM J. Matrix Anal. Appl., 31 (2009), pp. 1100–1124.
114.
V. Rokhlin and M. Tygert, A fast randomized algorithm for overdetermined linear least-squares regression, Proc. Natl. Acad. Sci. USA, 105 (2008), pp. 13212–13217.
115.
S. Roweis, EM algorithms for PCA and SPCA, in Proceedings of the 10th Annual Conference on Advances in Neural Information Processing Systems (NIPS), MIT Press, Cambridge, MA, 1997, pp. 626–632.
116.
M. Rudelson, Random vectors in the isotropic position, J. Funct. Anal., 164 (1999), pp. 60–72.
117.
M. Rudelson and R. Vershynin, Sampling from large matrices: An approach through geometric functional analysis, J. Assoc. Comput. Mach., 54 (2007), article 21.
118.
A. F. Ruston, Auerbach's theorem, Math. Proc. Cambridge Philos. Soc., 56 (1964), pp. 476–480.
119.
T. Sarlós, Improved approximation algorithms for large matrices via random projections, in Proceedings of the 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS), 2006, pp. 143–152.
120.
S. Shalev-Shwartz and N. Srebro, Low $\ell_1$-norm and guarantees on sparsifiability, in ICML/COLT/UAI Sparse Optimization and Variable Selection Workshop, 2008.
121.
X. Shen and F. G. Meyer, Low-dimensional embedding of fMRI datasets, Neuroimage, 41 (2008), pp. 886–902.
122.
N. D. Shyamalkumar and K. Varadarajan, Efficient subspace approximation algorithms, in Proceedings of the 18th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 2007, pp. 532–540.
123.
L. Sirovich and M. Kirby, Low-dimensional procedure for the characterization of human faces., J. Optical Soc. Amer. A, 4 (1987), pp. 519–524.
124.
D. Spielman and N. Srivastasa, Graph sparsification by effective resistances, in Proceedings of the 40th Annual ACM Symposium on Theory of Computing (STOC '08), 2008, pp. 563–568.
125.
G. Stewart, Accelerating the orthogonal iteration for the eigenvectors of a Hermitian matrix, Numer. Math., 13 (1969), pp. 362–376.
126.
G. W. Stewart, On the perturbation of pseudo-inverses, projections and linear least squares problems, SIAM Rev., 19 (1977), pp. 634–662.
127.
G. W. Stewart, Four algorithms for the efficient computation of truncated pivoted QR approximations to a sparse matrix, Numer. Math., 83 (1999), pp. 313–323.
128.
G. W. Stewart, The decompositional approach to matrix computation, Comput. Sci. Eng., 2 (1) (2000), pp. 50–59.
129.
T. Strohmer and R. Vershynin, A randomized Kaczmarz algorithm with exponential convergence, J. Fourier Anal. Appl., 15 (2009), pp. 262–278.
130.
J. Sun, Y. Xie, H. Zhang, and C. Faloutsos, Less is more: Compact matrix decomposition for large sparse graphs, Stat. Anal. Data Min., 1 (2008), pp. 6–22.
131.
S. J. Szarek, Spaces with large distance from $\ell^n_\infty$ and random matrices, Amer. J. Math., 112 (1990), pp. 899–942.
132.
A. Szlam, M. Maggioni, and R. R. Coifman, Regularization on graphs with function-adapted diffusion processes, J. Mach. Learn. Res., 9 (2008), pp. 1711–1739.
133.
L. N. Trefethen and D. Bau III, Numerical Linear Algebra, SIAM, Philadelphia, PA, 1997.
134.
J. A. Tropp, On the conditioning of random subdictionaries, Appl. Comput. Harmon. Anal., 25 (2008), pp. 1–24.
135.
J. A. Tropp, Improved analysis of the subsampled randomized Hadamard transform, Adv. Adaptive Data Anal., 3 (2011), to appear.
136.
J. von Neumann and H. H. Goldstine, Numerical inverting of matrices of high order, Bull. Amer. Math. Soc., 53 (1947), pp. 1021–1099.
137.
J. von Neumann and H. H. Goldstine, Numerical inverting of matrices of high order. II, Proc. Amer. Math. Soc., 2 (1952), pp. 188–202.
138.
F. Woolfe, E. Liberty, V. Rokhlin, and M. Tygert, A fast randomized algorithm for the approximation of matrices, Appl. Comput. Harmon. Anal., 25 (2008), pp. 335–366.

Information & Authors

Information

Published In

cover image SIAM Review
SIAM Review
Pages: 217 - 288
ISSN (online): 1095-7200

History

Submitted: 21 September 2009
Accepted: 2 December 2010
Published online: 5 May 2011

MSC codes

  1. 65F30
  2. 68W20
  3. 60B20

Keywords

  1. dimension reduction
  2. eigenvalue decomposition
  3. interpolative decomposition
  4. Johnson–Lindenstrauss lemma
  5. matrix approximation
  6. parallel algorithm
  7. pass-efficient algorithm
  8. principal component analysis
  9. randomized algorithm
  10. random matrix
  11. rank-revealing QR factorization
  12. singular value decomposition
  13. streaming algorithm

Authors

Affiliations

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited By