Parallel Algorithms for Sparse Linear Systems
Abstract
This paper surveys recent progress in the development of parallel algorithms for solving sparse linear systems on computer architectures having multiple processors. Attention is focused on direct methods for solving sparse symmetric positive definite systems, specifically by Cholesky factorization. Recent progress on parallel algorithms is surveyed for all phases of the solution process, including ordering, symbolic factorization, numeric factorization, and triangular solution.
MSC codes
MSC codes
Get full access to this article
View all available purchase options and get full access to this article.
References
1.
Gita Alaghband, Parallel pivoting combined with parallel reduction and fill-in control, Parallel Comput., 11 (1989), 201–221
2.
G. Alaghband, H. Jordan, Multiprocessor sparse L/U decomposition with controlled fill-in, Tech. Report, 85-48, ICASE, NASA Langley Research Center, Hampton, VA, 1985
3.
P. Amestoy, I. Duff, Vectorization of a multiprocessor multifrontal code, Internat. J. Supercomp. Appl., 3 (1989), 41–59
4.
E. Anderson, Y. Saad, Solving sparse triangular linear systems on parallel computers, Internat. J. High Speed Comput., 1 (1989), 73–95
5.
C. Ashcraft, A vector implementation of the multifrontal method for large sparse, symmetric positive definite linear systems, Tech. Report, ETA-TR-51, Engineering Technology Applications Division, Boeing Computer Services, Seattle, WA, 1987
6.
C. Ashcraft, 1990, Personal communication
7.
C. Ashcraft, Ph.D. Thesis, The aggregate model for the factorization of symmetric positive definite matrices, Dept. of Computer Science, Yale University, New Haven, CT, 1990
8.
C. Ashcraft, S. Eisenstat, J. Liu, B. Peyton, A. Sherman, A compute-ahead implementation of the fan-in sparse distributed factorization scheme, Tech. Report, ORNL/TM-11496, Oak Ridge National Laboratory, Oak Ridge, TN, 1990
9.
C. Ashcraft, S. Eisenstat, J. Liu, A. Sherman, A comparison of three column-based distributed sparse factorization schemes, Tech. Report, YALEU/DCS/RR-810, Dept. of Computer Science,Yale University, New Haven, CT, 1990
10.
C. Ashcraft, S. Eisenstat, J. W. H. Liu, A fan-in algorithm for distributed sparse numerical factorization, SIAM J. Sci. Statist. Comput., 11 (1990), 593–599
11.
C. Ashcraft, R. Grimes, J. Lewis, B. Peyton, H. Simon, Progress in sparse matrix methods for large linear systems on vector supercomputers, Internat. J. Supercomp. Appl., 1 (1987), 10–30
12.
R. Benner, G. Montry, G. Weigand, Concurrent multifrontal methods: shared memory, cache, and frontwidth issues, Internat. J. Supercomp. Appl., 1 (1987), 26–44
13.
Piotr Berman, Georg Schnitger, On the performance of the minimum degree ordering for Gaussian elimination, SIAM J. Matrix Anal. Appl., 11 (1990), 83–88
14.
J. Browne, J. Dongarra, A. Karp, K. Kennedy, D. Kuck, Gordon Bell prize, IEEE Software, 6 (1988), 78–85, (May 1989)
15.
I. Cavers, Masters Thesis, Tiebreaking the minimum degree algorithm for ordering sparse symmetric positive definite matrices, Master's thesis, Dept. of Computer Science, University of British Columbia, Vancouver, B.C., 1987
16.
Eleanor Chu, Alan George, Sparse orthogonal decomposition on a hypercube multiprocessor, SIAM J. Matrix Anal. Appl., 11 (1990), 453–465
17.
E. Chu, A. George, J. W.-H. Liu, E. G.-Y. Ng, User's guide for SPARSPAK-A: Waterloo sparse linear equations package, Tech. Report, CS-84-36, University of Waterloo, Waterloo, Ontario, 1984
18.
J. Conroy, Parallel direct solution of sparse linear systems of equations, Tech. Report, TR 1714, Dept. of Computer Science, University of Maryland, College Park, MD, 1986
19.
A. Dave, I. Duff, Sparse matrix calculations on the CRAY- 2, Parallel Comput., 5 (1987), 55–64
20.
T. Davis, P. Yew, A nondeterministic parallel algorithm for general unsymmetric sparse $LU$ factorization, SIAM J. Matrix Anal. Appl., 11 (1990), 383–402
21.
J. Dongarra, F. Gustavson, A. Karp, Implementing linear algebra algorithms for dense matrices on a vector pipeline machine, SIAM Rev., 26 (1984), 91–112
22.
I. Duff, Parallel implementation of multifrontal schemes, Parallel Comput., 3 (1986), 193–204
23.
I. Duff, Multiprocessing a sparse matrix code on the Alliant FX/8, J. Comput. Appl. Math., 27 (1989), 229–239
24.
I. Duff, A. Erisman, J. Reid, On George's nested dissection method, SIAM J. Numer. Anal., 13 (1976), 686–695
25.
I. Duff, A. Erisman, J. K. Reid, Direct Methods for Sparse Matrices, Oxford University Press, Oxford, U.K., 1987
26.
I. Duff, N. Gould, M. Lescrenier, J. K. Reid, M. Cox, S. Hammarling, The multifrontal method in a parallel environmentReliable numerical computation, Oxford Sci. Publ., Oxford Univ. Press, New York, 1990, 93–111
27.
I. Duff, J. Reid, MA27 - a set of Fortran subroutines for solving sparse symmetric sets of linear equations, Tech. Report, AERE R 10533, Harwell, 1982
28.
I. Duff, J. Reid, The multifrontal solution of indefinite sparse symmetric linear equations, ACM Trans. Math. Software, 9 (1983), 302–325
29.
S. Eisenstat, M. Gursky, M. Schultz, A. H. Sherman, The Yale sparse matrix package I. the symmetric codes, Internat. J. Numer. Methods Engrg., 18 (1982), 1145–1151
30.
S. Eisenstat, M. Heath, C. Henkel, C. Romine, Modified cyclic algorithms for solving triangular systems on distributed-memory multiprocessors, SIAM J. Sci. Statist. Comput., 9 (1988), 589–600
31.
C. Fiduccia, R. Mattheyses, A linear-time heuristic for improving network partitions, Proceedings of the 19th Design Automation Conference, 1982, 175–181
32.
Miroslav Fiedler, Algebraic connectivity of graphs, Czechoslovak Math. J., 23(98) (1973), 298–305
33.
Miroslav Fiedler, A property of eigenvectors of nonnegative symmetric matrices and its application to graph theory, Czechoslovak Math. J., 25(100) (1975), 619–633
34.
K. Gallivan, R. Plemmons, A. Sameh, Parallel algorithms for dense linear algebra computations, SIAM Rev., 32 (1990), 54–135
35.
F. Gao, B. Parlett, Communication cost of sparse Cholesky factorization on a hypercube, Tech. Report, PAM-436, Center for Pure and Applied Mathematics, University of California, Berkeley, CA, 1988
36.
G. Geist, M. T. Heath, Solving finite element problems with parallel multifrontal schemesHypercube Multiprocessors 1987, Society for Industrial and Applied Mathematics, Philadelphia, PA, 1987, 656–661
37.
G. Geist, M. Heath, Parallel Cholesky factorization on a hypercube multiprocessor, Tech. Report, ORNL-6211, Oak Ridge National Laboratory, Oak Ridge, TN, 1985
38.
G. Geist, E. G.-Y. Ng, Task scheduling for parallel sparse Cholesky factorization, Internat. J. Parallel Programming, 18 (1989), 291–314
39.
Alan George, Nested dissection of a regular finite element mesh, SIAM J. Numer. Anal., 10 (1973), 345–363
40.
A. George, M. Heath, J. W.-H. Liu, Parallel Cholesky factorization on a sharedmemory multiprocessor, Linear Algebra Appl., 77 (1986), 165–187
41.
A. George, M. Heath, J. W.-H. Liu, E. G.-Y. Ng, Solution of sparse positive definite systems on a shared-memory multiprocessor, Internat. J. Parallel Programming, 15 (1986), 309–325
42.
A. George, M. Heath, J. W.-H. Liu, E. G.-Y. Ng, Symbolic Cholesky factorization on a local-memory multiprocessor, Parallel Comput., 5 (1987), 85–95
43.
A. George, M. Heath, J. W.-H. Liu, E. G.-Y. Ng, Sparse Cholesky factorization on a local-memory multiprocessor, SIAM J. Sci. Statist. Comput., 9 (1988), 327–340
44.
A. George, M. Heath, J. W.-H. Liu, E. G.-Y. Ng, Solution of sparse positive definite systems on a hypercube, J. Comput. Appl. Math., 27 (1989), 129–156
45.
A. George, J. W.-H. Liu, An automatic nested dissection algorithm for irregular finite element problems, SIAM J. Numer. Anal., 15 (1978), 1053–1069
46.
A. George, J. W.-H. Liu, An optimal algorithm for symbolic factorization of symmetric matrices, SIAM J. Comput., 9 (1980), 583–593
47.
A. George, J. W.-H. Liu, Computer solution of large sparse positive definite systems, Prentice-Hall Inc., Englewood Cliffs, N.J., 1981xii+324
48.
A. George, J. W.-H. Liu, The evolution of the minimum degree ordering algorithm, SIAM Rev., 31 (1989), 1–19
49.
A. George, J. W.-H. Liu, E. G.-Y. Ng, Communication results for parallel sparse Cholesky factorization on a hypercube, Parallel Comput., 10 (1989), 287–298
50.
A. George, D. McIntyre, On the application of the minimum degree algorithm to finite element systems, SIAM J. Numer. Anal., 15 (1978), 90–112
51.
A. George, E. G.-Y. Ng, Parallel sparse Gaussian elimination with partial pivoting, Ann. Oper. Res., 22 (1990), 219–240
52.
J. Gilbert, An efficient parallel sparse partial pivoting algorithm, Tech. Report, CMI No. 88/45052-1, Centre for Computer Science, Dept. of Science and Technology, Chr. Michelsen Institute, Bergen, Norway, 1988
53.
J. Gilbert, H. Hafsteinsson, Parallel symbolic factorization of sparse linear systems, Parallel Comput., 14 (1990), 151–162
54.
J. Gilbert, R. Schreiber, Highly parallel sparse Cholesky factorization, SIAM J. Sci. Statist. Comput., 13 (1992), 1151–1172
55.
J. Gilbert, E. Zmijewski, A parallel graph partitioning algorithm for a message-passing multiprocessor, Internat. J. Parallel Programming, 16 (1987), 427–449
56.
A. Greenbaum, Solving sparse triangular linear systems using fortran with extensions on the NYU Ultracomputer prototype, Tech. Report, 99, NYU Ultracomputer Note, New York University, New York, 1986, April
57.
M. Heath, Visual animation of parallel algorithms for matrix computations, Proc. Fifth Distributed Memory Computing Conf., Charleston, SC, 1990
58.
M. Heath, C. Romine, Parallel solution of triangular systems on distributed-memory multiprocessors, SIAM J. Sci. Statist. Comput., 9 (1988), 558–588
59.
M. Heath, D. Sorensen, A pipelined Givens method for computing the $QR$ factorization of a sparse matrix, Linear Algebra Appl., 77 (1986), 189–203
60.
A. Hoffman, M. Martin, D. Rose, Complexity bounds for regular finite difference and finite element grids, SIAM J. Numer. Anal., 10 (1973), 364–369
61.
B. Irons, A frontal solution program for finite element analysis, Internat. J. Numer. Methods Engrg., 2 (1970), 5–32
62.
J. Jess, H. Kees, A data structure for parallel $L/U$ decomposition, IEEE Trans. Comput., 31 (1982), 231–239
63.
B. Kernighan, S. Lin, An efficient heuristic procedure for partitioning graphs, Bell Systems Tech. J., 49 (1970), 291–307
64.
C. Lawson, R. Hanson, D. Kincaid, F. Krogh, Basic linear algebra subprograms for Fortran usage, ACM Trans. Math. Software, 5 (1979), 308–371
65.
C. Leiserson, J. Lewis, G. Rodrigue, Orderings for parallel sparse symmetric factorizationParallel Processing for Scientific Computing, Society for Industrial and Applied Mathematics, Philadelphia, PA, 1989, 27–32
66.
M. Leuze, Independent set orderings for parallel matrix factorization by Gaussian elimination, Parallel Comput., 10 (1989), 177–191
67.
J. Lewis, B. Peyton, A. Pothen, A fast algorithm for reordering sparse matrices for parallel factorization, SIAM J. Sci. Statist. Comput., 10 (1989), 1146–1173
68.
R. Lipton, D. Rose, R. Tarjan, Generalized nested dissection, SIAM J. Numer. Anal., 16 (1979), 346–358
69.
R. Lipton, R. Tarjan, A separator theorem for planar graphs, SIAM J. Appl. Math., 36 (1979), 177–189
70.
J. W.-H. Liu, Modification of the minimum-degree algorithm by multiple elimination, ACM Trans. Math. Software, 11 (1985), 141–153
71.
J. W.-H. Liu, A compact row storage scheme for Cholesky factors using elimination trees, ACM Trans. Math. Software, 12 (1986), 127–148
72.
J. W.-H. Liu, Computational models and task scheduling for parallel sparse Cholesky factorization, Parallel Comput., 3 (1986), 327–342
73.
J. W.-H. Liu, Equivalent sparse matrix reordering by elimination tree rotations, SIAM J. Sci. Statist. Comput., 9 (1988), 424–444
74.
J. W.-H. Liu, A graph partitioning algorithm by node separators, ACM Trans. Math. Software, 15 (1989), 198–219
75.
J. W.-H. Liu, The minimum degree ordering with constraints, SIAM J. Sci. Statist. Comput., 10 (1989), 1136–1145
76.
J. W.-H. Liu, The multifrontal method and paging in sparse Cholesky factorization, ACM Trans. Math. Software, 15 (1989), 310–325
77.
J. W.-H. Liu, Reordering sparse matrices for parallel elimination, Parallel Comput., 11 (1989), 73–91
78.
J. W.-H. Liu, The multifrontal method for sparse matrix solution: theory and practice, Tech. Report, CS-90-04, Dept. of Computer Science, York University, North York, Ontario, 1990
79.
J. W.-H. Liu, The role of elimination trees in sparse factorization, SIAM J. Matrix Anal. Appl., 11 (1990), 134–172
80.
Joseph W. H. Liu, Andranik Mirzaian, A linear reordering algorithm for parallel pivoting of chordal graphs, SIAM J. Discrete Math., 2 (1989), 100–107
81.
J. W.-H. Liu, E. G.-Y. Ng, A supernodal symbolic Cholesky factorization on a local-memory multiprocessor, 1990, in preparation
82.
R. Lucas, Ph.D. Thesis, Solving planar systems of equations on distributed-memory multiprocessors, Dept. of Electrical Engineering, Stanford University, Stanford, CA, 1987
83.
R. Lucas, 1990, Personal communication
84.
R. Lucas, W. Blank, J. Tieman, A parallel solution method for large sparse systems of equations, IEEE Trans. Computer Aided Design, CAD-6 (1987), 981–991
85.
Rami Melhem, Parallel solution of linear systems with striped sparse matrices, Parallel Comput., 6 (1988), 165–184
86.
V. Naik, M. Patrick, Data traffic reduction schemes for sparse Cholesky factorization, Tech. Report, ICASE Report no. 88-14, ICASE, NASA Langley Research Center, Hampton, VA, 1988
87.
V. Naik, M. Patrick, Data traffic reduction schemes for Cholesky factorization on asynchronous multiprocessor systems, Tech. Report ICASE Report, 89-40, ICASE, NASA Langley Research Center, Hampton, VA, 1989
88.
E. Ng, B. Peyton, A supernodal Cholesky factorization algorithm for shared-memory multiprocessors, SIAM J. Sci. Comput., 14 (1993), 761–769
89.
J. Ortega, Introduction to parallel and vector solution of linear systems, Frontiers of Computer Science, Plenum Press, New York, 1989xii+305
90.
J. Ortega, R. Voigt, C. Romine, A bibliography on parallel and vector numerical algorithms, Tech. Report, ORNL/TM-10998, Oak Ridge National Laboratory, Oak Ridge, TN, 1989
91.
S. Parter, The use of linear graphs in Gauss elimination, SIAM Rev., 3 (1961), 119–130
92.
F. Peters, Parallel pivoting algorithms for sparse symmetric matrices, Parallel Comput., 1 (1984), 99–110
93.
A. Pothen, The complexity of optimal elimination trees, Tech. Report, Dept. of Computer Science, The Pennsylvania State University, University Park, PA, 1988
94.
A. Pothen, C.-J. Fan, Computing the block triangular form of a sparse matrix, ACM Trans. Math. Software, 16 (1990), 303–324
95.
A. Pothen, H. Simon, K. Liou, Partitioning sparse matrices with eigenvectors of graphs, SIAM J. Matrix Anal. Appl., 11 (1990), 430–452
96.
P. Raghavan, A. Pothen, Parallel orthogonal factorization, SIAM Symposium on Sparse Matrices, Gleneden Beach, OR, Society for Industrial and Applied Mathematics, Philadelphia, PA, 1989
97.
D. Rose, Triangulated graphs and the elimination process, J. Math. Anal. Appl., 32 (1970), 597–609
98.
D. Rose, R. C. Read, A graph-theoretic study of the numerical solution of sparse positive definite systems of linear equationsGraph theory and computing, Academic Press, New York, 1972, 183–217
99.
D. Rose, R. Tarjan, G. Lueker, Algorithmic aspects of vertex elimination on graphs, SIAM J. Comput., 5 (1976), 266–283
100.
E. Rothberg, A. Gupta, Fast sparse matrix factorization on modern workstations, Tech. Report, STAN-CS-89-1286, Stanford University, Stanford, CA, 1989
101.
P. Sadayappan, V. Visvanathan, Distributed sparse factorization of circuit matrices via recursive E-tree partitioning, SIAM Symposium on Sparse Matrices, Gleneden Beach, OR, Society for Industrial and Applied Mathematics, Philadelphia, PA, 1989
102.
J. Saltz, Aggregation methods for solving sparse triangular systems on multiprocessors, SIAM J. Sci. Statist. Comput., 11 (1990), 123–144
103.
Robert Schreiber, A new implementation of sparse Gaussian elimination, ACM Trans. Math. Software, 8 (1982), 256–276
104.
A. Sherman, Ph.D. Thesis, On the efficient solution of sparse systems of linear and nonlinear equations, Yale University, New Haven, CT, 1975
105.
P. Worley, R. Schreiber, A. Wouk, Nested dissection on a mesh-connected processor arrayNew computing environments: parallel, vector and systolic (Stanford, Calif., 1984), SIAM, Philadelphia, PA, 1986, 8–38
106.
C. Yang, P. Vu, A vector/parallel implementation of the multifrontal method for sparse symmetric definite linear systems on the Cray Y-MP, Tech. Report, CRAY Research, 1990
107.
Mihalis Yannakakis, Computing the minimum fill-in is NP-complete, SIAM J. Algebraic Discrete Methods, 2 (1981), 77–79
108.
E. Zmijewski, Ph.D. Thesis, Sparse Cholesky Factorization on a Multiprocessor, Dept. of Computer Science, Cornell University, Ithaca, NY, 1987, August
109.
E. Zmijewski, Limiting communication in parallel sparse Cholesky factorization, Tech. Report, TRCS89-18, Dept. of Computer Science, University of California, Santa Barbara, CA, 1989
110.
E. Zmijewski, J. Gilbert, A parallel algorithm for sparse symbolic Cholesky factorization on a multiprocessor, Parallel Comput., 7 (1988), 199–210
Information & Authors
Information
Published In
Copyright
Copyright © 1991 Society for Industrial and Applied Mathematics.
History
Submitted: 12 September 1990
Accepted: 10 October 1990
Published online: 18 July 2006
MSC codes
MSC codes
Authors
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.