Multiple meshes and MPI in FDS: how to speed up a run

fdsrun editorial team · 6 November 2026 · 6 min read

Why one big mesh is slow, how to split a model into meshes, balance MPI processes and where extra cores stop paying off.

An FDS model on a single mesh runs on one processor core, even if the computer has sixteen. To use more cores you have to split the domain into several meshes (&MESH) and run FDS with MPI, so that each process computes its own part of the model. A good split cuts run time several or a dozen times; a bad one achieves almost nothing.

Rule 1: equal cells per process

MPI processes compute in parallel, but in every time step they exchange data at mesh interfaces and wait for each other. If one process has twice as many cells as the others, the rest sit idle half the time. The NIST documentation is explicit: for MPI to work well, each process needs a comparable number of cells — and even then two processes will not finish in half the time, because of communication and extra boundary cells (FDS User Guide, sections 6.3.3 and 3.3.1).

A simple test: divide the total cell count by the number of processes and compare it with the largest mesh. If the largest mesh is much bigger than the average, it sets the pace.

Rule 2: several small meshes can share a process

The MPI_PROCESS parameter on &MESH assigns several meshes to one process. This makes sense when meshes differ in size: small meshes are grouped so that every process gets a similar total. Neighbouring meshes on the same process exchange data directly, without MPI communication. Process numbers must be consecutive and ascending, starting from 0.

fdsrun does this automatically: before the run it assigns meshes to processes so that the busiest process has as few cells as possible, and shows how the forecast time changes with the number of processes.

Rule 3: mesh interfaces

• At an interface, cells should have the same cross-section or an integer ratio (for example 1:2).
• Watch VELOCITY_ERROR at interfaces in CHID.out. If FDS keeps hitting MAX_PRESSURE_ITERATIONS (default 10), increase it or change the pressure solver settings (User Guide, sections 6.3.4 and 21.1).
• In practice it is best not to run a mesh boundary through the fire plume itself — gradients are steepest there.

How many processes make sense

Each additional process gains less: communication and boundary cells grow. In our calibration runs on a single cloud server (FDS 6.10.1) the speed-up stopped growing at around 16 processes, and servers with dedicated cores were not faster than shared ones at several times the price. For typical building models, 8–16 well-balanced processes usually give the best time-to-cost ratio.

According to NIST, OpenMP (threads within one process) speeds a calculation up by at most about a factor of two; it is usually better to divide the domain into more meshes and run one thread per process.

Example

Checklist before the run

• meshes have similar cell counts (or are grouped with MPI_PROCESS);
• interfaces have matching or integer cell ratios, with no overlapping meshes;
• the number of processes does not exceed the number of meshes;
• DT_RESTART is set so a long run can be resumed.

The free tool below checks all of these points — together with the forecast time for the best number of processes.

Sources: FDS User Guide (NIST SP 1019), chapter 3 and sections 6.3.3, 6.3.4, 21.1; fdsrun measurements. FDS and Smokeview are software of the National Institute of Standards and Technology; NIST is not affiliated with fdsrun.