HPC administration has become continuously more difficult. In addition to the classic compute, interconnects, storage, and software balance, you now have the addition of artificial intelligence (AI)-specific hardware and all the joys of the accompanying complex software stack. When everything is combined, one of the genuinely gnarly issues is the software stack: the layered, interdependent web of compilers, MPI libraries, math libraries, runtime environments, container images, and application codes on which users depend every day.

Software stack fragmentation – or the state in which different users, research groups, or workloads require incompatible combinations of these layers – is one of the most pervasive and underappreciated sources of lost productivity on HPC systems. Jobs fail mysteriously at runtime. Users spend hours debugging linking errors. Admins field support tickets that turn out to be version conflicts. All of these problems are headaches caused by software stack fragmentation. The situation was not great before the advent of AI, but it is getting even worse, not better, as new workloads with their own dependency trees land on clusters.

Research into HPC user support bears out this surmise. A 2020 study by DeLucia and Moore analyzed the ticketing system of the Los Alamos National Laboratory HPC Consult Team and found that developing better tools to categorize and track software environment issues was among the most pressing needs for support teams [1]. An earlier study of San Diego Supercomputer Center (SDSC) help tickets from 2004 to 2006 identified software build and environment problems as a dominant category of user-reported issues [2].

AI workloads have only compounded this problem. Although a definitive ranked list of ticket types across all HPC sites, particularly those with combined HPC and AI workloads, remains unpublished, software stack issues are consistently among the most discussed pain points in practitioner literature and community forums. In this article, I dig into why the problem exists, what it looks like in practice, and, perhaps most importantly, some concrete strategies and tools you can use to bring it under control.

Why HPC Software Stacks Are So Fragile

An HPC software stack is not a single thing. It is a vertical dependency chain in which each layer constrains the layers above and below it:

• Hardware firmware and drivers (BIOS, network interface card (NIC) firmware, GPU (device) firmware)
• Operating system and kernel, including drivers
• Compiler toolchains (e.g., GCC, Intel oneAPI, LLVM, NVIDIA HPC SDK (NVHPC))
• MPI implementations (e.g., OpenMPI, MPICH, Intel MPI, Cray MPICH)
• Math and communication libraries (e.g., basic linear algebra subprograms (BLAS), numerical linear algebra (LAPACK), fast Fourier transforms (FFTW), NVIDIA collective communications library (NCCL), unified communication X (UCX))
• Runtime environments (e.g., CUDA, ROCm, oneAPI)
• Application frameworks (e.g., Python, R, Julia, containers)
• End-user scientific applications (e.g., Vienna ab initio simulation package (VASP), molecular dynamics simulation suite (GROMACS), computational fluid dynamics software (OpenFOAM), PyTorch deep learning library)

It is important to recognize that interfaces between these layers are not always stable. An MPI binary compiled against OpenMPI 4.1 might fail silently or catastrophically when run against OpenMPI 5.0 at runtime. A Python package built against CUDA 11.8 will not load on a node with CUDA 12.x if application binary interface (ABI) compatibility breaks, particularly when other Python modules are involved. A BLAS call dispatched through a vendor library could produce incorrect results, not errors, if the underlying CPU feature flags mismatch.

Unlike web applications, HPC codes cannot rely on containerizing everything away, despite the use of Docker and Apptainer (formerly Singularity) containers. Many workloads require direct, low-latency access to hardware-specific interfaces, particularly high-speed interconnects like InfiniBand or Slingshot, which must be provided by host-side libraries that match the hardware driver. Therefore, even with containers, the host network stack bleeds into the container.

What Fragmentation Looks Like in Practice

The following three examples show failure patterns that illustrate how stack fragmentation comes about in day-to-day HPC administration. These are not ranked by frequency but represent problems that are common in practice.

1. Runtime Library Mismatches

A user compiles their code in an interactive session against OpenMPI 4.1.5. The resource manager (scheduler) assigns it to nodes that load a default MPI module different from the one used during compilation. The job launches, MPI_Init completes, and the job hangs or crashes during collective operations with an opaque error:

--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of them are due to configuration settings
and some are due to hardware failures.
--------------------------------------------------------------------------

The root cause for the error is that the runtime linker found a different libmpi.so than the one the binary used when linking. The error message gives no indication that this is the case.

2. Python and Conda Environments

Python's packaging ecosystem was not really designed for HPC. Users create conda environments or pip-installed stacks that pull in conflicting dependencies that sometimes go ignored or unnoticed or sometimes aren’t even presented to the user. A common failure pattern is:

$ python train.py
ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30'
not found (required by /home/user/.conda/envs/ml/lib/python3.10/
site-packages/torch/lib/libtorch_cpu.so)

The user's conda-installed PyTorch was built against a newer libstdc++ than the system provides. Their environment is self-consistent in isolation but fails on the cluster's system libraries.

3. Container and Host Stack Mismatch

Containers are often proposed as the solution to software stack fragmentation. They help but they do not eliminate the fragmentation problem for HPC workloads. If the CUDA version that is baked into a container image is newer than the host driver supports, the container will fail to initialize the GPU, resulting in messages like:

CUDA error: no kernel image is available for execution on the device
# or
Failed to initialize NVML: Driver/library version mismatch

Similarly, MPI-enabled containers for multinode jobs typically bind-mount the host MPI implementation into the container at runtime with the use of tools like Apptainer (Singularity; now continued as Apptainer under the Linux Foundation). The container’s internal MPI must be ABI-compatible with the host MPI for this to work – a constraint that is easy to miss when pulling a pre-built image from a public registry.

Suggested Strategies

Software stack fragmentation has no magic solution, but some strategies can help reduce it. Four of these solutions are discussed here.

Strategy 1: Adopt a Hierarchical Module System

The first line of defense is a well-structured environment module system. I have written about these before, but now they are even more important. One of the classic module tools is Lmod [3], which introduces hierarchical module organization. In my opinion, this feature is critical for managing compiler and MPI-dependent software stacks and works just as well for AI stacks. For more background on Lmod's hierarchical approach, see my overview [4].

To use Lmod, you define a hierarchy so that loading a compiler toolchain then exposes only the MPI implementations compiled with that toolchain; then, when an MPI toolchain is loaded, only the libraries or applications compiled by the combination of those compiler and MPI toolchains are exposed, which prevents users from inadvertently mixing incompatible layers, as shown here:

# Lmod hierarchy: compiler -> MPI -> applications
$ module load gcc/12.3.0
$ module load openmpi/4.1.5     # only shows OpenMPI built with GCC 12
$ module load fftw/3.3.10       # only shows FFTW built with GCC 12 + OpenMPI 4.1.5
 
# Switching compilers automatically swaps dependent modules
$ module swap gcc/12.3.0 intel/2023.2
# Lmod automatically unloads openmpi/4.1.5 and fftw/3.3.10,
# and reloads compatible versions if available

Lmod also provides the spider command, which lets you search across the entire module tree for available software; it emits clear warnings when a module swap would invalidate loaded dependencies.

Strategy 2: Build with Spack

Maintaining a consistent, reproducible software stack across a cluster, especially across multiple architecture targets, is where Spack shines [5]. The Spack package manager [6] is designed specifically for HPC and is capable of building the same package in dozens of configurations and tracking the full dependency graph, such as:

# Install a specific software combination
$ spack install openmpi@4.1.5 %gcc@12.3.0 +ucx fabrics=ucx
 
# Install an application against a specific toolchain
$ spack install gromacs@2023.3 %gcc@12.3.0 ^openmpi@4.1.5 +cuda cuda_arch=80
 
# Generate Lmod module files from the Spack build
$ spack module lmod refresh --delete-tree -y
 
# View the full dependency tree
$ spack graph gromacs@2023.3 %gcc@12.3.0 ^openmpi@4.1.5

Note that Spack can generate the needed Lmod files.

Spack also supports environments and lock files, which allow you to reproduce an exact software stack down to every transitive dependency across multiple clusters or after a system upgrade:

# Create and activate a Spack environment
$ spack env create production-2024
$ spack env activate production-2024
$ spack add gromacs@2023.3 %gcc@12.3.0 ^openmpi@4.1.5
$ spack install
 
# Export the full lock file for reproducibility
$ spack env depfile > spack.lock
 
# Recreate identical environment on another system
$ spack env create reproduced-env spack.lock
$ spack install

For sites supporting GPU workloads, Spack has first-class CUDA and ROCm variant support, allowing you to build GPU-enabled software stacks with explicit architecture targeting.

Strategy 3: Containers with MPI Awareness

For workloads that genuinely benefit from software portability, particularly AI/machine learning (ML) workloads with complex Python dependency trees, Apptainer (Singularity) containers [7] [8] (or Docker for non-MPI cases) remain valuable. The key is making the container MPI- and GPU-aware correctly.

MPI Bind-Mount Pattern (Apptainer)

The standard approach for multinode MPI in containers is the “host MPI” model: The container ships a compatible internal MPI, and the job launcher bind-mounts the host MPI at runtime:

# Container must include MPI matching the host major.minor version
# (e.g., in your Dockerfile or definition file):
# RUN apt-get install -y libopenmpi-dev openmpi-bin
 
# Launch multi-node MPI job via Apptainer
$ mpirun -n 64 --hostfile nodes.txt apptainer exec
  --bind /opt/openmpi:/opt/openmpi myapp.sif /app/myapp

The --bind flag overlays the host MPI into the container's filesystem namespace, ensuring the runtime linker finds the host-side libmpi.so while the container provides everything else.

GPU Containers

Whereas the MPI bind-mount pattern addresses the interconnect, GPU-enabled containers face an analogous version-matching problem with the NVIDIA driver. The host driver determines the maximum CUDA toolkit version that any container on that node can use; a container built against a newer CUDA release than the host driver supports will fail at runtime rather than at build time, which makes it a frustrating surprise for users. The commands below show how to check the host driver version and confirm GPU access from inside a container before submitting a full job:

# Check host driver version
$ nvidia-smi --query-gpu=driver_version --format=csv,noheader
535.129.03
 
# The container CUDA version must be <= what the driver supports
# Driver 535.x supports up to CUDA 12.2
# Confirm compatibility before pulling an image:
$ apptainer exec --nv cuda:12.2.0-runtime-ubuntu22.04 python 
  -c "import torch; print(torch.cuda.is_available())"

The --nv flag in Apptainer (equivalent to --gpus all in Docker with the nvidia-container-toolkit package) injects the host driver libraries into the container at runtime, bridging the container's CUDA toolkit with the host's physical GPU driver.

Strategy 4: Enforce Stack Consistency with CI

The most effective long-term approach is to treat your software stack the same way software engineers treat application code: with automated testing and continuous integration (CI). Every time a module or package is deployed or updated, a suite of smoke tests should run automatically.

ReFrame [9] is a regression testing framework designed specifically for HPC systems. It can run tests across multiple system partitions, programming environments, and parameter combinations simultaneously and is used at national computing centers, including the Swiss National Supercomputing Centre (CSCS), National Energy Research Scientific Computing Center (NERSC), and Oak Ridge National Laboratory (ORNL). For example:

# ReFrame regression test: verify GROMACS MPI launch
# File: tests/gromacs_mpi_test.py
 
import reframe as rfm
import reframe.utility.sanity as sn
 
@rfm.simple_test
class GromacsTest(rfm.RegressionTest):
    descr = 'GROMACS MPI correctness test'
    valid_systems = ['cluster:gpu', 'cluster:cpu']
    valid_prog_environs = ['gnu-mpi']
    modules = ['gcc/12.3.0', 'openmpi/4.1.5', 'gromacs/2023.3']
    executable = 'gmx_mpi mdrun'
     executable_opts = [
       '-s', 'benchMEM.tpr', '-nsteps', '1000', '-ntmpi', '4']
    num_tasks = 4
    num_tasks_per_node = 2
 
    @sanity_function
    def validate(self):
        return sn.assert_found('Finished mdrun', self.stdout)
# Run the test suite
$ reframe -c tests/ -r --performance-report

Although I haven’t run ReFrame myself, it is widely adopted at major centers and well worth evaluating.

A Practical Stack Management Checklist

No single tool solves software stack fragmentation. An effective approach combines several practices (Table 1).

A few additional practices are worth consideration to help attack the problem:

• Pin module defaults explicitly and document every change. Users should never encounter a silent change to which module load python resolves.
• Maintain a compatibility matrix for your site. Document which compiler, MPI, and CUDA/ROCm combinations are tested and supported. Unsupported combinations should either be blocked at the module level or be flagged clearly.
• Provide a stack audit tool. A simple script that inspects a user’s loaded modules and loaded shared libraries (with ldd on their binary) can bring most runtime mismatch issues to the surface before job submission.
• Version control your Spack configurations and module trees. Treat them as infrastructure code, with the same review and change-management discipline.
• Make regular backups of critical configuration files, including Spack configurations, module trees, ReFrame scripts, and results, because version control is not a substitute for backups. Develop a recovery process for the use of these backups and then test that process with a disaster scenario drill.

Conclusion

Software stack fragmentation is not a glamorous problem. It does not appear in benchmark papers or procurement requests for proposals (RFPs), but it is responsible for an enormous amount of lost researcher time, wasted compute allocations, and administrator support burden at HPC sites of every size.

The good news is that the tools to address fragmentation have matured significantly. Lmod, Spack, Apptainer, and ReFrame form a coherent toolkit that, deployed together with clear policies, can transform a chaotic, ticket-heavy environment into one in which users spend their time doing science instead of debugging linker errors.

The investment is real: Building a Spack-managed Lmod hierarchical stack from scratch takes time, and writing ReFrame tests for your key applications requires domain knowledge. However, the payoff in reduced support load, improved reproducibility, and user satisfaction is equally real and compounds over time as the stack grows.

Reducing the software stack fragmentation problem really requires developing and maintaining a disciplined approach. To develop good habits, start small by choosing your three most commonly installed software packages; build them with Spack, expose them through an Lmod hierarchy, and write one ReFrame smoke test for each. That foothold is enough to demonstrate the value and build momentum toward a fully managed stack.

References

[1] DeLucia, A., and E. Moore. Analyzing HPC support tickets: Experience and recommendations. arXiv:2010.04321 [cs.DC]; 2020;10p, https://arxiv.org/abs/2010.04321

[2] Wolter, N., M.O. McCracken, A. Snavely, L. Hochstein, T. Nakamura, and V. Basili. What’s working in HPC: Investigating HPC user behavior and productivity. CTWatch Quarterly. 2006;2(4A):9p, https://icl.utk.edu/ctwatch/quarterly/articles/2006/11/whats-working-in-hpc-investigating-hpc-user-behavior-and-productivity/

[3] Lmod: https://lmod.readthedocs.io/

[4] “New Release of Lmod Environment Modules System” by Jeff Layton, ADMIN HPC, July 2015, https://www.admin-magazine.com/HPC/Articles/Lmod-6.0-Exploring-the-Latest-Edition-of-the-Powerful-Environment-Module-System/

[5] Gamblin, T., M. LeGendre, M.R. Collette, G.L. Lee, A. Moody, B.R. de Supinski, and S. Futral. "The Spack package manager: Bringing order to HPC software chaos" In: Proceedings of SC15: International Conference for High Performance Computing, Networking, Storage and Analysis. (ACM, November 2015), article 40:12p, https://dl.acm.org/doi/10.1145/2807591.2807623.

[6] Spack: https://spack.io/

[7] Apptainer: https://apptainer.org/

[8] Kurtzer, G.M., B. Sochat, and M.W. Bauer. Singularity: Scientific containers for mobility of compute. PLOS ONE, 2017;12(5):e0177459, https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0177459

[9] ReFrame: https://reframe-hpc.readthedocs.io/