Containers are useful for all kinds of projects, from simple test scripts, to testing code, to testing HPC code. I often use both Docker and Apptainer in conjunction with HPC Container Maker (HPCCM), and I use containers for all of the usual reasons (e.g., isolation from the underlying operating system, bundling code and prerequisites, portability). I really like having a container with almost everything I need, so I can copy it to a different system and just run it without getting into the dependency loop.

Most of you know the pluses and minuses of both Docker and Apptainer, so I won’t go over that. Neither option fulfills all my needs, so I thought I would check out Podman. Testing it with a few simple containers worked well, including to run Docker and Apptainer containers. However, as I started trying more complicated containers, such as those in HPC, things didn’t work quite the way I wanted (i.e., they didn’t work). I tried tracking down the issues, but I eventually gave up.

That investigation was about 2 years ago, so I thought it was time to try Podman again, primarily because of the ability to create and run containers without being root or having elevated privileges. Before starting the journey, I looked around for newer articles about Podman, particularly about HPC. I got a hit very quickly on a tool that sounds perfect: podman-hpc .

Podman-HPC

The US National Energy Research Scientific Computing Center (NERSC), in particular it’s HPC capability, has some of the largest systems in the world; more importantly, it has one of the best user experiences of any large HPC system anywhere. The staff really focuses on supporting their users and providing top-notch tools, as well as making sure the systems are rock solid stable, including a very good storage system.

NERSC created podman-hpc , which is a wrapper script – an add-on, if you will – around the Pod Manager (Podman) container engine. The tool adds HPC configuration and infrastructure for Podman so you can build and run containers that use MPI and GPUs.

Installing podman-hpc

In the example I present here, I test podman-hpc  on an Ubuntu 26.04 laptop with an NVIDIA RTX GPU, although I won’t be testing the GPU in this article. Obviously, the first thing I need to do is install Podman itself and check the version:

$ sudo apt update
$ sudo install podman -y
 
$ podman --version
podman version 5.7.0

The next step is obviously to install podman-hpc . When I looked, it had no releases, so I just cloned the Git repository and installed from there:

$ git clone https://github.com/NERSC/podman-hpc
$ cd podman-hpc

At this point, I just used pip  to install. I already had the Anaconda Python distribution, but I wasn’t worried about conflicts or problems. Yes, I realize I could have created a Python environment and installed it there, and that is the best practice, but I used pip  for this article:

$ pip install .

The next step is critical: You need to run the built-in configuration to set up the necessary hooks for your particular environment:

$ python -m podman_hpc.configure_hooks

First Container

Once podman-hpc  is installed and appears to be running correctly, the next test is to have it build and run a container. During this process, notice that no elevated privileges are needed. Everything is done as a user.

To begin, build a simple container. The file named Containerfile  defines an image that uses Ubuntu 22.04 as a base and has an entry point that prints hello from inside a container  when the image is instantiated:

FROM docker.io/library/ubuntu:22.04
ENTRYPOINT echo "hello from inside a container!"

That’s it. Very simple.

The next step is to build the image with podman-hpc . I used the steps shown in Listing 1 and include the output from the command. To make sure the image was created, I listed all the available images (Listing 2). Yep. There it is under localhost  because that is where I’m storing the images for this example.

Listing 1: Building the Image

$ podman-hpc build -t hello-world:1.0 .
STEP 1/3: FROM docker.io/library/ubuntu:22.04
Trying to pull docker.io/library/ubuntu:22.04...
Getting image source signatures
Copying blob 98c4455a9898 done   |
Copying config e8df6c74f6 done   |
Writing manifest to image destination
STEP 2/3: ENV "PODMANHPC_MODULES_DIR"="/etc/podman_hpc/modules.d"
--> 4909584e8f55
STEP 3/3: ENTRYPOINT echo "hello from inside a container!"
COMMIT hello-world:1.0
--> fc5d86a0bd05
Successfully tagged localhost/hello-world:1.0
fc5d86a0bd05185367a5ecaff73615d4e856b33072f6c7fe8e32b34ca5f1c6d4

Listing 2: Listing the Images

$ podman-hpc images
REPOSITORY                TAG         IMAGE ID      CREATED         SIZE
localhost/hello-world     1.0         fc5d86a0bd05  47 seconds ago  80.6 MB
docker.io/library/ubuntu  22.04       e8df6c74f650  2 weeks ago     80.7 MB

The last step is to “run” the container:

$ podman-hpc run --rm hello-world:1.0
hello from inside a container!

That is success. You got the message you wanted when the container was instantiated. If you want a few more details, you can add the -it  option along with specifying the entry point (Listing 3).

Listing 3: Container Details

$ podman-hpc run --rm -it --entrypoint= hello-world:1.0 bash
root@28d383c3ca9f:/# cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"

First HPC Container (MPI)

The next logical step is to test the creation of an HPC image and instantiate it. In this case, I ran a simple MPI program that was in the container. It’s just a simple Hello World  from each process used in the MPI run. The code is amazingly simple (Listing 4). I first compiled the code and ran on the node I was using for testing without a container. It ran correctly.

Listing 4: Hello World

#include 
#include 
 
int main(int argc, char** argv) {
    MPI_Init(&argc, &argv);
    int world_size, world_rank;
    MPI_Comm_size(MPI_COMM_WORLD, &world_size);
    MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
 
    char processor_name[MPI_MAX_PROCESSOR_NAME];
    int name_len;
    MPI_Get_processor_name(processor_name, &name_len);
 
    printf("Hello World from rank %d out of %d processors on %s\n",
           world_rank, world_size, processor_name);
 
    MPI_Finalize();
    return 0;
}

The next step is to put it in a container and instantiate the container with mpirun . The Containerfile used is not much more complicated than the Hello World I used earlier (Listing 5).

Listing 5: Adding Code to a Container

FROM ubuntu:26.04
 
ENV DEBIAN_FRONTEND=noninteractive
 
# Install Ubuntu 26.04 build essentials and its native MPICH library
RUN apt-get update && \
    apt-get install -y build-essential gfortran mpich libmpich-dev && \
    rm -rf /var/lib/apt/lists/*
 
# Set working directory for your application
WORKDIR /app
 
# Copy the simple MPI test code (mpi_hello.c) into the container
COPY mpi_hello.c /app/mpi_hello.c
 
# Compile the binary using the native MPICH wrappers
RUN mpicc -o mpi_hello mpi_hello.c
 
# Default run target
CMD ["/app/mpi_hello"]

I used Ubuntu 26.4 this time (I used 22.4 last time), then installed the prerequisites, including the GNU compilers and MPICH. The Containerfile then builds the sample code inside the container. The first command it runs when instantiated is the application.

Building the image was straightforward, with the command looking like that used to build the Hello World image:

$ podman-hpc build -t mpi-test-image:latest .

I’m not including the output from the build because it is too long. Of course, I need to check that the build was successful by checking the image list (Listing 6). In this case, everything looks reasonable. The final step is to instantiate the container with the mpirun  command (Listing 7).

Listing 6: Checking the Image List

$ podman-hpc images
REPOSITORY                TAG      IMAGE ID      CREATED            SIZE
localhost/mpi-test-image  latest   3d87297f408c  7 seconds ago      821 MB
localhost/hello-world     1.0      fc5d86a0bd05  About an hour ago  80.6 MB
docker.io/library/ubuntu  26.04    6e92c4fd8dfd  2 weeks ago        112 MB
docker.io/library/ubuntu  22.04    e8df6c74f650  2 weeks ago        80.7 MB

Listing 7: Instantiating the Container

$ mpirun -n 4 podman-hpc run --net=host --pid=host --ipc=host mpi-test-image:latest /app/mpi_hello
Hello World from rank 0 out of 1 processors on laytonjb-Nitro-ANV15-52
Hello World from rank 0 out of 1 processors on laytonjb-Nitro-ANV15-52
Hello World from rank 0 out of 1 processors on laytonjb-Nitro-ANV15-52
Hello World from rank 0 out of 1 processors on laytonjb-Nitro-ANV15-52

I used a few options for the test that are used with containers. For example, net=host  bypasses Podman (Docker) network isolation, which means all containers share the same IP address (the host IP address).

The pid=host  option flag is used with containers to share the host’s process ID namespace with a container. In parallel environments, this option allows the MPICH process manager running on the host to monitor and manage the containerized MPI processes directly over native signals.

The ipc=host  option is passed to Podman so that the interprocess communication (IPC) namespace is shared with the container (bypassing the network stack), which I find makes things easier when first testing on a single physical node.

Summary

Containers are a great way to build and run HPC applications that provide isolation and operating system independence (to some degree). However, building and running containers isn’t the easiest way.

I tried to use Podman in the past because it allowed me to build and run the container without being root or without elevated privileges. I had some problems when trying to build images for HPC applications, but Podman-HPC solved all of those problems.

Podman-HPC has been around for a while, and I’m sorry I didn’t know about it when I did the original Podman article. Now, however, I know about it and all the kinds of problems it solves. Podman-HPC allows you to build images without root or its privileges and then run the containers.