An Optimized, Parallel Computation of the Ghost Layer for Adaptive Hybrid Forest Meshes
Abstract
We discuss parallel algorithms to compute the ghost layer in computational, distributed memory, recursively adapted meshes. Its creation is a fundamental, necessary task in executing most parallel, element-based computer simulations. Common methods differ in that the ghost layer may either be inherently part of the mesh data structure that is maintained and modified, or kept separate and constructed/deleted as needed. In this work, we present a design following the latter approach, which we chose for its modularity of algorithms and data structures. We target arbitrary adaptive, nonconforming forest-of-trees meshes of mixed element shapes, such as cubes, prisms, and tetrahedra, and restrict ourselves to ghost elements across mesh faces. Our algorithm has low code complexity and redundancy since we reduce it to generic codimension-1 subalgorithms that can be flexibly combined. We recover older algorithms for cubic elements as special cases and optimize further using recursive, amortized tree searches and traversals.
Keywords
MSC codes
Get full access to this article
View all available purchase options and get full access to this article.
Supplementary Material
PLEASE NOTE: These supplementary files have not been peer-reviewed.
Index of Supplementary Materials
Title of paper: An Optimized, Parallel Computation of the Ghost Layer for Adaptive Hybrid Forest Meshes
Authors: Johannes Holke, David Knapp, Carsten Burstedde
File: Ghost_For_Hybrid_AMR_supp.pdf
Type: PDF
Contents: This supplementary material contains details for the implementation of the low-level algorithms for the face neighbor computation across tree boundaries as well as the children-at-face computation for the identification of owner processes.
References
1.
W. Bangerth, C. Burstedde, T. Heister, and M. Kronbichler, Algorithms and data structures for massively parallel generic adaptive finite element codes, ACM Trans. Math. Software, 38 (2011), pp. 1--28, https://doi.org/10.1145/2049673.2049678.
2.
W. Bangerth, R. Hartmann, and G. Kanschat, deal.II---a general-purpose object-oriented finite element library, ACM Trans. Math. Software, 33 (2007), 24, https://doi.org/10.1145/1268776.1268779.
3.
J. Bey, Der BPX-Vorkonditionierer in drei Dimensionen: Gitterverfeinerung, Parallelisierung und Simulation, preprint, Universität Heidelberg, Heidelberg, Germany, 1992.
4.
C. Burstedde, Parallel tree algorithms for AMR and non-standard data access, ACM Trans. Math. Software, 46 (2020), pp. 1--31, https://doi.org/10.1145/3401990.
5.
C. Burstedde and J. Holke, A tetrahedral space-filling curve for nonconforming adaptive meshes, SIAM J. Sci. Comput., 38 (2016), pp. C471--C503, https://doi.org/10.1137/15M1040049.
6.
C. Burstedde and J. Holke, Coarse mesh partitioning for tree-based AMR, SIAM J. Sci. Comput., 39 (2017), pp. C364--C392, https://doi.org/10.1137/16M1103518.
7.
C. Burstedde, L. C. Wilcox, and O. Ghattas, p4est: Scalable algorithms for parallel adaptive mesh refinement on forests of octrees, SIAM J. Sci. Comput., 33 (2011), pp. 1103--1133, https://doi.org/10.1137/100791634.
8.
C. Chevalier and F. Pellegrini, PT-Scotch: A tool for efficient parallel graph ordering, Parallel Comput., 34 (2008), pp. 318--331, https://doi.org/10.1016/j.parco.2007.12.001.
9.
T. Coupez, L. Silva, and H. Digonnet, A massively parallel multigrid solver using PETSc for unstructured meshes on a Tier-0 supercomputer, presentation at the PETSc User Meeting, 2016, https://www.mcs.anl.gov/petsc/meetings/2016/slides/digonnet.pdf.
10.
A. Dedner, R. Klöfkorn, and M. Nolte, The DUNE-ALUGrid Module, preprint, https://arxiv.org/abs/1407.6954, 2014.
11.
K. Devine, E. Boman, R. Heaphy, B. Hendrickson, and C. Vaughan, Zoltan data management services for parallel dynamic applications, Comput. Sci. Eng., 4 (2002), pp. 90--97.
12.
R. Egan, A. Guittet, F. Temprano-Coleto, T. Isaac, F. J. Peaudecerf, J. R. Landel, P. Luzzatto-Fegiz, C. Burstedde, and F. Gibou, Direct numerical simulation of incompressible flows on parallel octree grids, J. Comput. Phys., 428 (2021), 110084, https://doi.org/10.1016/j.jcp.2020.110084.
13.
J. Holke, Scalable Algorithms for Parallel Tree-Based Adaptive Mesh Refinement with General Element Types, Ph.D thesis, Rheinische Friedrich-Wilhelms-Universität Bonn, Bonn, Germany, 2018, http://d-nb.info/1173789790/34.
14.
J. Holke and C. Burstedde, t8code: Parallel AMR on Hybrid Non-conforming Meshes, http://github.com/holke/t8code, 2015 (last accessed October 18, 2019).
15.
T. Isaac, C. Burstedde, L. C. Wilcox, and O. Ghattas, Recursive algorithms for distributed forests of octrees, SIAM J. Sci. Comput., 37 (2015), pp. C497--C531, https://doi.org/10.1137/140970963.
16.
Jülich Supercomputing Centre, JUQUEEN: IBM Blue Gene/Q supercomputer system at the Jülich Supercomputing Centre, J. Large-Scale Research Facilities, 1 (2015), A1, https://doi.org/10.17815/jlsrf-1-18.
17.
Jülich Supercomputing Centre, JUWELS: Modular tier-0/1 supercomputer at the Jülich Supercomputing Centre, J. Large-Scale Research Facilities, 5 (2019), A135, https://doi.org/10.17815/jlsrf-5-171.
18.
G. Karypis and V. Kumar, A parallel algorithm for multilevel graph partitioning and sparse matrix ordering, J. Parallel Distrib. Comput., 48 (1998), pp. 71--95.
19.
D. Knapp, Adaptive Verfeinerung von Prismen, Bachelor's thesis, Rheinische Friedrich-Wilhelms-Universität Bonn, Bonn, Germany, 2017.
20.
D. Knapp, A Space-Filling Curve for Pyramidal Adaptive Mesh Refinement, Master's thesis, Rheinische Friedrich-Wilhelms-Universität Bonn, Bonn, Germany, 2020.
21.
O. S. Lawlor, S. Chakravorty, T. L. Wilmarth, N. Choudhury, I. Dooley, G. Zheng, and L. V. Kalé, ParFUM: A parallel framework for unstructured meshes for scalable dynamic physics applications, Eng. Comput., 22 (2006), pp. 215--235, https://doi.org/10.1007/s00366-006-0039-5.
22.
Q. Liu, W. Zhao, J. Cheng, Z. Mo, A. Zhang, and J. Liu, A programming framework for large scale numerical simulations on unstructured mesh, in Proceedings of the 2016 IEEE 2nd International Conference on Big Data Security on Cloud (BigDataSecurity), IEEE International Conference on High Performance and Smart Computing (HPSC), and IEEE International Conference on Intelligent Data and Security (IDS), 2016, pp. 310--315, https://doi.org/10.1109/BigDataSecurity-HPSC-IDS.2016.12.
23.
G. M. Morton, A Computer Oriented Geodetic Data Base; and a New Technique in File Sequencing, Tech. Report, IBM Ltd., Armonk, NY, 1966.
24.
G. Peano, Sur une courbe, qui remplit toute une aire plane, Math. Ann., 36 (1890), pp. 157--160.
25.
M. Rasquin, C. Smith, K. Chitale, E. S. Seol, B. A. Matthews, J. L. Martin, O. Sahni, R. M. Loy, M. S. Shephard, and K. E. Jansen, Scalable implicit flow solver for realistic wing simulations with flow control, Comput. Sci. Eng., 16 (2014), pp. 13--21, https://doi.org/10.1109/MCSE.2014.75.
26.
C. Richardson and G. Wells, High Performance Multi-physics Simulations with FEniCS/DOLFIN, Tech. Report eCSE03-10, CSE programme of the ARCHER UK National Supercomputing Service, 2016, https://doi.org/10.6084/M9.FIGSHARE.3406582.
27.
I. Sbalzarini, J. Walther, M. Bergdorf, S. Hieber, E. Kotsalis, and P. Koumoutsakos, PPM---a highly efficient parallel particle-mesh library for the simulation of continuum systems, J. Comput. Phys., 215 (2006), pp. 566--588, https://doi.org/10.1016/j.jcp.2005.11.017.
28.
F. Schornbaum and U. Rüde, Massively parallel algorithms for the lattice Boltzmann method on nonuniform grids, SIAM J. Sci. Comput., 38 (2016), pp. C96--C126, https://doi.org/10.1137/15M1035240.
29.
J. R. Stewart and H. C. Edwards, A framework approach for developing parallel adaptive multiphysics applications, Finite Elem. Anal. Des., 40 (2004), pp. 1599--1617, https://doi.org/10.1016/j.finel.2003.10.006.
30.
H. Sundar, R. S. Sampath, and G. Biros, Bottom-up construction and 2:1 balance refinement of linear octrees in parallel, SIAM J. Sci. Comput., 30 (2008), pp. 2675--2708, https://doi.org/10.1137/070681727.
31.
H. Sundar, R. S. Sampath, S. S. Adavani, C. Davatzikos, and G. Biros, Low-constant parallel algorithms for finite element simulations using linear octrees, in Proceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage, and Analysis (SC'07), 2007.
32.
T. J. Tautges, J. A. Kraftcheck, N. Bertram, V. Sachdeva, and J. Magerlein, Mesh interface resolution and ghost exchange in a parallel mesh representation, in Proceedings of the IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum, 2012, pp. 1670--1679.
33.
T. J. Tautges, R. Meyers, K. Merkley, C. Stimpson, and C. Ernst, MOAB: A Mesh-Oriented Database, Report SAND2004-1592, Sandia National Laboratories, Albuquerque, NM, 2004.
34.
J. Teunissen and R. Keppens, A geometric multigrid library for quadtree/octree AMR grids coupled to MPI-AMRVAC, Comput. Phys. Commun., 245 (2019), 106866, https://doi.org/10.1016/j.cpc.2019.106866.
35.
T. Tu, D. R. O'Hallaron, and O. Ghattas, Scalable parallel octree meshing for tera-scale applications, in Proceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage, and Analysis (SC'05), 2005, pp. 4--4, https://doi.org/10.1109/SC.2005.61.
36.
T. Weinzierl, The Peano software-parallel, automaton-based, dynamically adaptive grid traversals, ACM Trans. Math. Software, 45 (2019), 14, https://doi.org/10.1145/3319797.
37.
T. Weinzierl and M. Mehl, Peano---a traversal and storage scheme for octree-like adaptive Cartesian multiscale grids, SIAM J. Sci. Comput., 33 (2011), pp. 2732--2760, https://doi.org/10.1137/100799071.
Information & Authors
Information
Published In
SIAM Journal on Scientific Computing
Pages: C359 - C385
DOI: 10.1137/20M1383033
ISSN (online): 1095-7197
Copyright
© 2021, Society for Industrial and Applied Mathematics.
History
Submitted: 1 December 2020
Accepted: 5 August 2021
Published online: 16 November 2021
Keywords
MSC codes
Authors
Funding Information
Gauss Centre for Supercomputing : hbn26
Funding Information
Bonn International Graduate School for Mathematics
Funding Information
Deutsche Forschungsgemeinschaft https://doi.org/10.13039/501100001659 : 390685813
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited By
There are no citations for this item