Parallel Scalability of Three-Level FROSch Preconditioners to 220000 Cores using the Theta Supercomputer
Abstract
The parallel performance of the three-level fast and robust overlapping Schwarz (FROSch) preconditioners is investigated for linear elasticity. The FROSch framework is part of the Trilinos software library and contains a parallel implementation of different preconditioners with energy minimizing coarse spaces of generalized Dryja--Smith--Widlund type. The three-level extension is constructed by a recursive application of the FROSch preconditioner to the coarse problem. In this paper, the additional steps in the implementation in order to apply the FROSch preconditioner recursively are described in detail. Furthermore, it is shown that no explicit geometric information is needed in the recursive application of the preconditioner. In particular, the rigid body modes, including the rotations, can be interpolated on the coarse level without additional geometric information. Parallel results for a three-dimensional linear elasticity problem obtained on the Theta supercomputer (Argonne Leadership Computing Facility, Argonne, IL) using up to 220 000 cores are discussed and compared to results obtained on the SuperMUC-NG supercomputer (Leibniz Supercomputing Centre, Garching, Germany). Notably, it can be observed that a hierarchical communication operation in FROSch related to the coarse operator starts to dominate the computing time on Theta, which has a dragonfly interconnect, for 100 000 message passing interface (MPI) ranks or more. The same operation, however, scales well and stays within the order of a second in all experiments performed on SuperMUC-NG, which uses a fat tree network. Using hybrid MPI/OpenMP parallelization, the onset of the MPI communication problem on Theta can be delayed. Further analysis of the performance of FROSch on large supercomputers with dragonfly interconnects will be necessary.
Keywords
MSC codes
Get full access to this article
View all available purchase options and get full access to this article.
References
1.
Reference Documentation for deal.ii Version 9.3.0 - The Step-8 Tutorial Program, 2021, https://www.dealii.org/current/doxygen/deal.II/step_8.html.
2.
Trilinos Public Git Repository, 2021, https://github.com/trilinos/trilinos.
3.
D. Arndt, W. Bangerth, D. Davydov, T. Heister, L. Heltai, M. Kronbichler, M. Maier, J.-P. Pelteret, B. Turcksin, and D. Wells, The deal.II finite element library: Design, features, and insights, Comput. Math. Appl., 81 (2021), pp. 407--422, https://doi.org/10.1016/j.camwa.2020.02.022.
4.
S. Badia, A. F. Martín, and J. Principe, Multilevel balancing domain decomposition at extreme scales, SIAM J. Sci. Comput., 38 (2016), pp. C22--C52, https://doi.org/10.1137/15M1013511.
5.
A. H. Baker, A. Klawonn, T. Kolev, M. Lanser, O. Rheinbach, and U. M. Yang, Scalability of classical algebraic multigrid for elasticity to half a million parallel tasks, in Software for Exascale Computing - SPPEXA 2013-2015, H.-J. Bungartz, P. Neumann, and W. E. Nagel, eds., Springer, Cham, 2016, pp. 113--140, https://doi.org/10.1007/978-3-319-40528-5_6.
6.
A. H. Baker, T. V. Kolev, and U. M. Yang, Improving algebraic multigrid interpolation operators for linear elasticity problems, Numer. Linear Algebra Appl., 17 (2010), pp. 495--517, https://doi.org/https://doi.org/10.1002/nla.688.
7.
M. Bollhöfer, O. Schenk, R. Janalik, S. Hamm, and K. Gullapalli, State-of-the-art sparse direct solvers, in Parallel Algorithms in Computational Science and Engineering, A. Grama and A. H. Sameh, eds., Springer, Cham, 2020, pp. 3--33, https://doi.org/10.1007/978-3-030-43736-7_1.
8.
C. Burstedde, L. C. Wilcox, and O. Ghattas, p4est: Scalable algorithms for parallel adaptive mesh refinement on forests of octrees, SIAM J. Sci. Comput., 33 (2011), pp. 1103--1133, https://doi.org/10.1137/100791634.
9.
H. Carter Edwards, C. R. Trott, and D. Sunderland, Kokkos: Enabling manycore performance portability through polymorphic memory access patterns, J. Parallel Distrib. Comput., 74 (2014), pp. 3202--3216, https://doi.org/10.1016/j.jpdc.2014.07.003.
10.
S. Chunduri, K. Harms, S. Parker, V. Morozov, S. Oshin, N. Cherukuri, and K. Kumaran, Run-to-run variability on Xeon Phi based Cray xc systems, in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '17, Association for Computing Machinery, New York, NY, 2017, https://doi.org/10.1145/3126908.3126926.
11.
C. R. Dohrmann, A. Klawonn, and O. B. Widlund, Domain decomposition for less regular subdomains: Overlapping Schwarz in two dimensions, SIAM J. Numer. Anal., 46 (2008), pp. 2153--2168.
12.
C. R. Dohrmann, A. Klawonn, and O. B. Widlund, A family of energy minimizing coarse spaces for overlapping Schwarz preconditioners, in Domain Decomposition Methods in Science and Engineering XVII, Lecture Notes in Comput. Sci. Eng. 60, Springer, Berlin, 2008, pp. 247--254.
13.
C. R. Dohrmann and O. B. Widlund, On the design of small coarse spaces for domain decomposition algorithms, SIAM J. Sci. Comput., 39 (2017), pp. A1466--A1488, https://doi.org/10.1137/17M1114272.
14.
P. Fuentes, E. Vallejo, C. Camarero, R. Beivide, and M. Valero, Network unfairness in dragonfly topologies, J. Supercomput., 72 (2016), pp. 4468--4496, https://doi.org/10.1007/s11227-016-1758-z.
15.
GWT-TUD GmbH, Vampir 9 User Manual, Dresden, Germany, 2019, https://vampir.eu.
16.
W. Hackbusch, Multigrid Methods and Applications, Springer Ser. Comput. Math. 4, Springer-Verlag, Berlin, 1985, https://doi.org/10.1007/978-3-662-02427-0.
17.
A. Heinlein, C. Hochmuth, and A. Klawonn, Fully Algebraic Two-Level Overlapping Schwarz Preconditioners for Elasticity Problems, Tech. Rep., Universität zu Köln, 2019, https://kups.ub.uni-koeln.de/10441/.
18.
A. Heinlein, A. Klawonn, S. Rajamanickam, and O. Rheinbach, FROSch: A fast and robust overlapping Schwarz domain decomposition preconditioner based on Xpetra in Trilinos, in Domain Decomposition Methods in Science and Engineering XXV, R. Haynes, S. MacLachlan, X.-C. Cai, L. Halpern, H. H. Kim, A. Klawonn, and O. Widlund, eds., Springer, Cham, 2020, pp. 176--184, https://doi.org/10.1007/978-3-030-56750-7_19.
19.
A. Heinlein, A. Klawonn, and O. Rheinbach, A parallel implementation of a two-level overlapping Schwarz method with energy-minimizing coarse space based on Trilinos, SIAM J. Sci. Comput., 38 (2016), pp. C713--C747, https://doi.org/10.1137/16M1062843.
20.
A. Heinlein, A. Klawonn, O. Rheinbach, and F. Röver, A three-level extension of the GDSW overlapping Schwarz preconditioner in two dimensions, in Advanced Finite Element Methods with Applications: Selected Papers from the 30th Chemnitz Finite Element Symposium 2017, Springer, Cham, 2019, pp. 187--204, https://doi.org/10.1007/978-3-030-14244-5_10.
21.
A. Heinlein, A. Klawonn, O. Rheinbach, and F. Röver, A three-level extension of the GDSW overlapping Schwarz preconditioner in three dimensions, in Domain Decomposition Methods in Science and Engineering XXV, Springer, Cham, 2020, pp. 185--192, https://doi.org/10.1007/978-3-030-56750-7_20.
22.
A. Heinlein, A. Klawonn, O. Rheinbach, and F. Röver, A Three-Level Extension for Fast and Robust Overlapping Schwarz (FROSch) Preconditioners with Reduced Dimensional Coarse Space, Tech. report, TU Bergakademie Freiberg, February 2021, https://tu-freiberg.de/fakult1/forschung/preprints, submitted to the 26th International Conference on Domain Decomposition Methods.
23.
A. Heinlein, A. Klawonn, O. Rheinbach, and O. B. Widlund, Improving the parallel performance of overlapping Schwarz methods by using a smaller energy minimizing coarse space, in Domain Decomposition Methods in Science and Engineering XXIV, Springer, Cham, 2018, pp. 383--392.
24.
A. Heinlein, M. Perego, and S. Rajamanickam, Frosch Preconditioners for Land Ice Simulations of Greenland and Antarctica, Technical report, Universität zu Köln, January 2021, https://kups.ub.uni-koeln.de/30668/.
25.
A. Heinlein, O. Rheinbach, and F. Röver, Choosing the subregions in three-level FROSch preconditioners, in Proceedings of the 14th WCCM-ECCOMAS Congress, 2020, https://doi.org/10.23967/wccm-eccomas.2020.084.
26.
P. Jolivet, F. Hecht, F. Nataf, and C. Prud'homme, Scalable domain decomposition preconditioners for heterogeneous elliptic problems, in Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC '13, New York, NY, 2013, ACM, pp. 80:1--80:11, https://doi.org/10.1145/2503210.2503212.
27.
B. Kiefer, O. Rheinbach, S. Roth, and F. Röver, Variational methods and parallel solvers in chemo-mechanics, PAMM. Proc. Appl. Math. Mech., 20 (2021), e202000272, https://doi.org/10.1002/pamm.202000272.
28.
A. Klawonn and O. Rheinbach, Inexact FETI-DP methods, Internat. J. Numer. Methods Engrg., 69 (2007), pp. 284--307, https://doi.org/10.1002/nme.1758.
29.
F. Kong and X.-C. Cai, A highly scalable multilevel Schwarz method with boundary geometry preserving coarse spaces for $3$D elasticity problems on domains with complex geometry, SIAM J. Sci. Comput., 38 (2016), pp. C73--C95, https://doi.org/10.1137/15M1010567.
30.
J. Mandel, B. Sousedík, and C. R. Dohrmann, Multispace and multilevel BDDC, Computing, 83 (2008), pp. 55--85.
31.
L. F. Pavarino and S. Scacchi, Parallel multilevel Schwarz and block preconditioners for the bidomain parabolic-parabolic and parabolic-elliptic formulations, SIAM J. Sci. Comput., 33 (2011), pp. 1897--1919, https://doi.org/10.1137/100808721.
32.
S. Scacchi, A hybrid multilevel Schwarz method for the bidomain model, Comput. Methods Appl. Mech. Engrg., 197 (2008), pp. 4051--4061, https://doi.org/10.1016/j.cma.2008.04.008.
33.
J. Sístek, B. Sousedík, J. Mandel, and P. Burda, Parallel implementation of multilevel BDDC, in Proceedings of the 9th European Conference on Numerical Mathematics and Advanced Applications, Leicester, UK, 2011.
34.
A. Toselli and O. Widlund, Domain Decomposition Methods---Algorithms and Theory, Springer Ser. Comput. Math. 34, Springer-Verlag, Berlin, 2005.
35.
X. Tu, Three-level BDDC in two dimensions, Internat. J. Numer. Methods Engrg., 69 (2007), pp. 33--59, https://doi.org/10.1002/nme.1753.
36.
Xpetra Project Team, The Xpetra Project Website, https://trilinos.github.io/xpetra.html.
37.
Zoltan2 Project Team, The Zoltan2 Project Website, https://trilinos.github.io/zoltan2.html.
Information & Authors
Information
Published In
SIAM Journal on Scientific Computing
Pages: S173 - S198
DOI: 10.1137/21M1431205
ISSN (online): 1095-7197
Copyright
© 2022, Society for Industrial and Applied Mathematics.
History
Submitted: 7 July 2021
Accepted: 22 April 2022
Published online: 23 August 2022
Keywords
MSC codes
Authors
Funding Information
Gauss Centre
Deutsche Forschungsgemeinschaft https://doi.org/10.13039/501100001659 : 2256, 441509557
Deutsche Forschungsgemeinschaft https://doi.org/10.13039/501100001659 : 397252409
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited By
- A computational framework for pharmaco‐mechanical interactions in arterial walls using parallel monolithic domain decomposition methodsGAMM-Mitteilungen, Vol. 47, No. 1 | 8 January 2024
- A Multilevel Extension of the GDSW Overlapping Schwarz Preconditioner in Two DimensionsComputational Methods in Applied Mathematics, Vol. 23, No. 4 | 12 July 2023
- Monolithic parallel overlapping Schwarz methods in fully-coupled nonlinear chemo-mechanics problemsComputational Mechanics, Vol. 71, No. 4 | 31 January 2023
View Options
- Access via your Institution
- Questions about how to access this content? Contact SIAM at [email protected].