We discuss parallel algorithms to compute the ghost layer in computational, distributed memory, recursively adapted meshes. Its creation is a fundamental, necessary task in executing most parallel, element-based computer simulations. Common methods differ in that the ghost layer may either be inherently part of the mesh data structure that is maintained and modified, or kept separate and constructed/deleted as needed. In this work, we present a design following the latter approach, which we chose for its modularity of algorithms and data structures. We target arbitrary adaptive, nonconforming forest-of-trees meshes of mixed element shapes, such as cubes, prisms, and tetrahedra, and restrict ourselves to ghost elements across mesh faces. Our algorithm has low code complexity and redundancy since we reduce it to generic codimension-1 subalgorithms that can be flexibly combined. We recover older algorithms for cubic elements as special cases and optimize further using recursive, amortized tree searches and traversals.


  1. adaptive mesh refinement
  2. parallel algorithms
  3. ghost layer
  4. forest of trees

MSC codes

  1. 65M50
  2. 68W10
  3. 65Y05
  4. 65D18

Get full access to this article

View all available purchase options and get full access to this article.

Supplementary Material

PLEASE NOTE: These supplementary files have not been peer-reviewed.

Index of Supplementary Materials

Title of paper: An Optimized, Parallel Computation of the Ghost Layer for Adaptive Hybrid Forest Meshes

Authors: Johannes Holke, David Knapp, Carsten Burstedde

File: Ghost_For_Hybrid_AMR_supp.pdf

Type: PDF

Contents: This supplementary material contains details for the implementation of the low-level algorithms for the face neighbor computation across tree boundaries as well as the children-at-face computation for the identification of owner processes.


W. Bangerth, C. Burstedde, T. Heister, and M. Kronbichler, Algorithms and data structures for massively parallel generic adaptive finite element codes, ACM Trans. Math. Software, 38 (2011), pp. 1--28, https://doi.org/10.1145/2049673.2049678.
W. Bangerth, R. Hartmann, and G. Kanschat, deal.II---a general-purpose object-oriented finite element library, ACM Trans. Math. Software, 33 (2007), 24, https://doi.org/10.1145/1268776.1268779.
J. Bey, Der BPX-Vorkonditionierer in drei Dimensionen: Gitterverfeinerung, Parallelisierung und Simulation, preprint, Universität Heidelberg, Heidelberg, Germany, 1992.
C. Burstedde, Parallel tree algorithms for AMR and non-standard data access, ACM Trans. Math. Software, 46 (2020), pp. 1--31, https://doi.org/10.1145/3401990.
C. Burstedde and J. Holke, A tetrahedral space-filling curve for nonconforming adaptive meshes, SIAM J. Sci. Comput., 38 (2016), pp. C471--C503, https://doi.org/10.1137/15M1040049.
C. Burstedde and J. Holke, Coarse mesh partitioning for tree-based AMR, SIAM J. Sci. Comput., 39 (2017), pp. C364--C392, https://doi.org/10.1137/16M1103518.
C. Burstedde, L. C. Wilcox, and O. Ghattas, p4est: Scalable algorithms for parallel adaptive mesh refinement on forests of octrees, SIAM J. Sci. Comput., 33 (2011), pp. 1103--1133, https://doi.org/10.1137/100791634.
C. Chevalier and F. Pellegrini, PT-Scotch: A tool for efficient parallel graph ordering, Parallel Comput., 34 (2008), pp. 318--331, https://doi.org/10.1016/j.parco.2007.12.001.
T. Coupez, L. Silva, and H. Digonnet, A massively parallel multigrid solver using PETSc for unstructured meshes on a Tier-0 supercomputer, presentation at the PETSc User Meeting, 2016, https://www.mcs.anl.gov/petsc/meetings/2016/slides/digonnet.pdf.
A. Dedner, R. Klöfkorn, and M. Nolte, The DUNE-ALUGrid Module, preprint, https://arxiv.org/abs/1407.6954, 2014.
K. Devine, E. Boman, R. Heaphy, B. Hendrickson, and C. Vaughan, Zoltan data management services for parallel dynamic applications, Comput. Sci. Eng., 4 (2002), pp. 90--97.
R. Egan, A. Guittet, F. Temprano-Coleto, T. Isaac, F. J. Peaudecerf, J. R. Landel, P. Luzzatto-Fegiz, C. Burstedde, and F. Gibou, Direct numerical simulation of incompressible flows on parallel octree grids, J. Comput. Phys., 428 (2021), 110084, https://doi.org/10.1016/j.jcp.2020.110084.
J. Holke, Scalable Algorithms for Parallel Tree-Based Adaptive Mesh Refinement with General Element Types, Ph.D thesis, Rheinische Friedrich-Wilhelms-Universität Bonn, Bonn, Germany, 2018, http://d-nb.info/1173789790/34.
J. Holke and C. Burstedde, t8code: Parallel AMR on Hybrid Non-conforming Meshes, http://github.com/holke/t8code, 2015 (last accessed October 18, 2019).
T. Isaac, C. Burstedde, L. C. Wilcox, and O. Ghattas, Recursive algorithms for distributed forests of octrees, SIAM J. Sci. Comput., 37 (2015), pp. C497--C531, https://doi.org/10.1137/140970963.
Jülich Supercomputing Centre, JUQUEEN: IBM Blue Gene/Q supercomputer system at the Jülich Supercomputing Centre, J. Large-Scale Research Facilities, 1 (2015), A1, https://doi.org/10.17815/jlsrf-1-18.
Jülich Supercomputing Centre, JUWELS: Modular tier-0/1 supercomputer at the Jülich Supercomputing Centre, J. Large-Scale Research Facilities, 5 (2019), A135, https://doi.org/10.17815/jlsrf-5-171.
G. Karypis and V. Kumar, A parallel algorithm for multilevel graph partitioning and sparse matrix ordering, J. Parallel Distrib. Comput., 48 (1998), pp. 71--95.
D. Knapp, Adaptive Verfeinerung von Prismen, Bachelor's thesis, Rheinische Friedrich-Wilhelms-Universität Bonn, Bonn, Germany, 2017.
D. Knapp, A Space-Filling Curve for Pyramidal Adaptive Mesh Refinement, Master's thesis, Rheinische Friedrich-Wilhelms-Universität Bonn, Bonn, Germany, 2020.
O. S. Lawlor, S. Chakravorty, T. L. Wilmarth, N. Choudhury, I. Dooley, G. Zheng, and L. V. Kalé, ParFUM: A parallel framework for unstructured meshes for scalable dynamic physics applications, Eng. Comput., 22 (2006), pp. 215--235, https://doi.org/10.1007/s00366-006-0039-5.
Q. Liu, W. Zhao, J. Cheng, Z. Mo, A. Zhang, and J. Liu, A programming framework for large scale numerical simulations on unstructured mesh, in Proceedings of the 2016 IEEE 2nd International Conference on Big Data Security on Cloud (BigDataSecurity), IEEE International Conference on High Performance and Smart Computing (HPSC), and IEEE International Conference on Intelligent Data and Security (IDS), 2016, pp. 310--315, https://doi.org/10.1109/BigDataSecurity-HPSC-IDS.2016.12.
G. M. Morton, A Computer Oriented Geodetic Data Base; and a New Technique in File Sequencing, Tech. Report, IBM Ltd., Armonk, NY, 1966.
G. Peano, Sur une courbe, qui remplit toute une aire plane, Math. Ann., 36 (1890), pp. 157--160.
M. Rasquin, C. Smith, K. Chitale, E. S. Seol, B. A. Matthews, J. L. Martin, O. Sahni, R. M. Loy, M. S. Shephard, and K. E. Jansen, Scalable implicit flow solver for realistic wing simulations with flow control, Comput. Sci. Eng., 16 (2014), pp. 13--21, https://doi.org/10.1109/MCSE.2014.75.
C. Richardson and G. Wells, High Performance Multi-physics Simulations with FEniCS/DOLFIN, Tech. Report eCSE03-10, CSE programme of the ARCHER UK National Supercomputing Service, 2016, https://doi.org/10.6084/M9.FIGSHARE.3406582.
I. Sbalzarini, J. Walther, M. Bergdorf, S. Hieber, E. Kotsalis, and P. Koumoutsakos, PPM---a highly efficient parallel particle-mesh library for the simulation of continuum systems, J. Comput. Phys., 215 (2006), pp. 566--588, https://doi.org/10.1016/j.jcp.2005.11.017.
F. Schornbaum and U. Rüde, Massively parallel algorithms for the lattice Boltzmann method on nonuniform grids, SIAM J. Sci. Comput., 38 (2016), pp. C96--C126, https://doi.org/10.1137/15M1035240.
J. R. Stewart and H. C. Edwards, A framework approach for developing parallel adaptive multiphysics applications, Finite Elem. Anal. Des., 40 (2004), pp. 1599--1617, https://doi.org/10.1016/j.finel.2003.10.006.
H. Sundar, R. S. Sampath, and G. Biros, Bottom-up construction and 2:1 balance refinement of linear octrees in parallel, SIAM J. Sci. Comput., 30 (2008), pp. 2675--2708, https://doi.org/10.1137/070681727.
H. Sundar, R. S. Sampath, S. S. Adavani, C. Davatzikos, and G. Biros, Low-constant parallel algorithms for finite element simulations using linear octrees, in Proceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage, and Analysis (SC'07), 2007.
T. J. Tautges, J. A. Kraftcheck, N. Bertram, V. Sachdeva, and J. Magerlein, Mesh interface resolution and ghost exchange in a parallel mesh representation, in Proceedings of the IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum, 2012, pp. 1670--1679.
T. J. Tautges, R. Meyers, K. Merkley, C. Stimpson, and C. Ernst, MOAB: A Mesh-Oriented Database, Report SAND2004-1592, Sandia National Laboratories, Albuquerque, NM, 2004.
J. Teunissen and R. Keppens, A geometric multigrid library for quadtree/octree AMR grids coupled to MPI-AMRVAC, Comput. Phys. Commun., 245 (2019), 106866, https://doi.org/10.1016/j.cpc.2019.106866.
T. Tu, D. R. O'Hallaron, and O. Ghattas, Scalable parallel octree meshing for tera-scale applications, in Proceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage, and Analysis (SC'05), 2005, pp. 4--4, https://doi.org/10.1109/SC.2005.61.
T. Weinzierl, The Peano software-parallel, automaton-based, dynamically adaptive grid traversals, ACM Trans. Math. Software, 45 (2019), 14, https://doi.org/10.1145/3319797.
T. Weinzierl and M. Mehl, Peano---a traversal and storage scheme for octree-like adaptive Cartesian multiscale grids, SIAM J. Sci. Comput., 33 (2011), pp. 2732--2760, https://doi.org/10.1137/100799071.

Information & Authors


Published In

cover image SIAM Journal on Scientific Computing
SIAM Journal on Scientific Computing
Pages: C359 - C385
ISSN (online): 1095-7197


Submitted: 1 December 2020
Accepted: 5 August 2021
Published online: 16 November 2021


  1. adaptive mesh refinement
  2. parallel algorithms
  3. ghost layer
  4. forest of trees

MSC codes

  1. 65M50
  2. 68W10
  3. 65Y05
  4. 65D18



Funding Information

Gauss Centre for Supercomputing : hbn26

Funding Information

Bonn International Graduate School for Mathematics

Funding Information

Deutsche Forschungsgemeinschaft https://doi.org/10.13039/501100001659 : 390685813

Metrics & Citations



If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited By

There are no citations for this item







Copy the content Link

Share with email

Email a colleague

Share on social media

The SIAM Publications Library now uses SIAM Single Sign-On for individuals. If you do not have existing SIAM credentials, create your SIAM account https://my.siam.org.