The Message Passing Interface (MPI) is one of the frameworks used in high-performance computing (HPC) for a wide variety of parallel computing architectures. It can be used for running applications on multicore systems and on multicore/multinode systems, where you map a “process” to a processing unit, typically a CPU core. MPI has been around for many years and is used by a great number of applications to scale, in terms of performance.

Classically, the way to run an MPI application is something like:

$ mpirun -np 16 --hostfile  ./ []

Note that this command does not really correspond to a specific MPI implementation; rather, it is meant to illustrate the basics of most MPI implementations.

The mpirun command launches and coordinates the application (<application>), the -np option is the number of processes to be used, and --hostfile names a simple text file (<list_of_hosts>)that has a list of hostnames (servers) to be used in running the application.

With the development of containers and their use in HPC and artificial intelligence (AI), the challenge becomes how to run MPI applications that are in the container. In this article, I show an approach to accomplishing this goal.

Test Software

I'm sure a number of people have come up with several ways to run MPI applications in containers. The methods presented in this article are the best ways I’ve found.

Although these tests are not performance tests that require detailed specifications about the hardware and software, I offer a brief description. The system ran Linux Mint 19.3 and used the latest PGI Community Edition compilers, version 19.10. Open MPI 3.1.3, which came prebuilt with the PGI compilers, was used in the tests.

The Docker-CE (Community Edition) v19.03.8 build afacb8b7f0, Ubuntu version (on which Linux Mint is based), was downloaded. For Singularity, the latest version as of the writing of this article, 3.5.3, was built and installed. It uses Go 1.13, which was installed from binaries on the golang website.

The simple MPI used is the classic 2D Poisson equation solver, poisson_mpi, which I procured from an online set of Fortran 90 examples.

HPCCM

HPC Container Maker (HPCCM) was used to create the specification files for creating the container images. It was installed in the Anaconda distribution (Python plus R programming language), which included conda (a package and environment manager) version 4.8.2 and Python 3.7. HPCCM version 20.2.0 was installed (the latest as of the writing of this article).

HPCCM was discussed in a previous HPC article. The tool, written in Python, is very easy to use and allows you to specify a basic description of how you want your container built. This HPCCM “recipe” can then create a Dockerfile for Docker or a singularity description file for Singularity, which you can modify. These files are then used to create the container images. For this example, the same HPCCM recipe is used for both Docker and Singularity.

Creating Container Specification Files

The first step in creating the container specification file is to create the HPCCM recipe, for which you can find the instructions in the HPCCM tutorial. The recipe file used in this article (Listing 1) also can be found in that tutorial.

Listing 1: poisson.py

import hpccm
 
Stage0 += baseimage(image='ubuntu:18.04')
Stage0 += pgi(eula=True, mpi=True)
 
Stage0 += copy(src='poisson_mpi.f90', dest='/var/tmp/poisson_mpi.f90')
Stage0 += shell(commands=['mpif90 /var/tmp/poisson_mpi.f90 -o /usr/local/bin/poisson_mpi'])

From the recipe, you can see that Ubuntu 18.04 is the base image and that the PGI compilers, version 19.10, are installed. Note that the PGI compilers require you to accept the end-user license agreement (EULA). (If you install the compilers on your home system, for example, you are presented the EULA to read and accept.) Also notice that I’m installing the prebuilt MPI libraries that come with the compiler.

The next line in the recipe copies the application source code into the image, and the last line is a shell directive to compile the code and put the binary in /usr/local/bin. The location of the binary was chosen arbitrarily.

Taking the poisson.py recipe in Listing 1, you can create a Singularity definition file or Dockerfile with just one command:

$ hpccm --recipe poisson.py --format singularity > Singularity.def
$ hpccm --recipe poisson.py --format docker > Dockerfile

After the specification files are created, be sure to examine them, even if you aren’t modifying anything. Alternatively, if you remove the redirection to an output file, HPCCM prints the specification file to the screen.

For Singularity, I made a small modification to the recipe file because I wanted to try pulling the PGI compilers from the local host rather than over the Internet (Listing 2). The resulting Singularity definition file is shown in Listing 3, and the Dockerfile is shown in Listing 4.

Listing 2: Modified poisson.py File

import hpccm
 
Stage0 += baseimage(image='ubuntu:18.04')
Stage0 += pgi(eula=True, tarball='/home/laytonjb/pgilinux-2019-1910-x86-64.tar.gz', mpi=True)
 
Stage0 += copy(src='poisson_mpi.f90', dest='/var/tmp/poisson_mpi.f90')
Stage0 += shell(commands=['mpif90 /var/tmp/poisson_mpi.f90 -o /usr/local/bin/poisson_mpi'])

Listing 3: Singularity Definition File

From: ubuntu:18.04
%post
    . /.singularity.d/env/10-docker*.sh
 
# PGI compiler version 19.10
%files
    /home/laytonjb/pgilinux-2019-1910-x86-64.tar /var/tmp/pgilinux-2019-1910-x86-64.tar
%post
    apt-get update -y
    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        g++ \
        gcc \
        libnuma1 \
        openssh-client \
        perl
    rm -rf /var/lib/apt/lists/*
%post
    cd /
    mkdir -p /var/tmp/pgi && tar -x -f /var/tmp/pgilinux-2019-1910-x86-64.tar -C /var/tmp/pgi
    cd /var/tmp/pgi && PGI_ACCEPT_EULA=accept PGI_INSTALL_DIR=/opt/pgi PGI_INSTALL_MPI=true PGI_INSTALL_NVIDIA
      =true PGI_MPI_GPU_SUPPORT=true PGI_SILENT=true ./install
    echo "variable LIBRARY_PATH is environment(LIBRARY_PATH);" >> /opt/pgi/linux86-64/19.10/bin/siterc
    echo "variable library_path is default(\$if(\$LIBRARY_PATH,\$foreach(ll,\$replace(\$LIBRARY_PATH,":",), -L \$ll)));" >> /opt/pgi/linux86-64/19.10/bin/siterc
    echo "append LDLIBARGS=\$library_path;" >> /opt/pgi/linux86-64/19.10/bin/siterc
    ln -sf /usr/lib/x86_64-linux-gnu/libnuma.so.1 /opt/pgi/linux86-64/19.10/lib/libnuma.so
    ln -sf /usr/lib/x86_64-linux-gnu/libnuma.so.1 /opt/pgi/linux86-64/19.10/lib/libnuma.so.1
    rm -rf /var/tmp/pgilinux-2019-1910-x86-64.tar /var/tmp/pgi
%environment
    export LD_LIBRARY_PATH=/opt/pgi/linux86-64/19.10/mpi/openmpi-3.1.3/lib:/opt/pgi/linux86-64/19.10/lib:$LD_LIBRARY_PATH
    export PATH=/opt/pgi/linux86-64/19.10/mpi/openmpi-3.1.3/bin:/opt/pgi/linux86-64/19.10/bin:$PATH
%post
    export LD_LIBRARY_PATH=/opt/pgi/linux86-64/19.10/mpi/openmpi-3.1.3/lib:/opt/pgi/linux86-64/19.10/lib:$LD_LIBRARY_PATH
    export PATH=/opt/pgi/linux86-64/19.10/mpi/openmpi-3.1.3/bin:/opt/pgi/linux86-64/19.10/bin:$PATH
 
%files
    poisson_mpi.f90 /var/tmp/poisson_mpi.f90
 
%post
    cd /
    mpif90 /var/tmp/poisson_mpi.f90 -o /usr/local/bin/poisson_mpi

Listing 4: Dockerfile

FROM ubuntu:18.04
 
# PGI compiler version 19.10
RUN apt-get update -y && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        g++ \
        gcc \
        libnuma1 \
        openssh-client \
        perl \
        wget && \
    rm -rf /var/lib/apt/lists/*
RUN mkdir -p /var/tmp && wget -q -nc --no-check-certificate -O /var/tmp/pgi-community-linux-x64-latest.tar.gz 
--referer https://www.pgroup.com/products/community.htm?utm_source=hpccm\&utm_medium=wgt\&utm_campaign=CE\&nvi
d=nv-int-14-39155 -P /var/tmp https://www.pgroup.com/support/downloader.php?file=pgi-community-linux-x64 && \
    mkdir -p /var/tmp/pgi && tar -x -f /var/tmp/pgi-community-linux-x64-latest.tar.gz -C /var/tmp/pgi -z && \
    cd /var/tmp/pgi && PGI_ACCEPT_EULA=accept PGI_INSTALL_DIR=/opt/pgi PGI_INSTALL_MPI=true PGI_INSTALL_NVIDIA
      =true PGI_MPI_GPU_SUPPORT=true PGI_SILENT=true ./install && \
    echo "variable LIBRARY_PATH is environment(LIBRARY_PATH);" >> /opt/pgi/linux86-64/19.10/bin/siterc && \
    echo "variable library_path is default(\$if(\$LIBRARY_PATH,\$foreach(ll,\$replace(\$LIBRARY_PATH,":",), -L\$ll)));" >> /opt/pgi/linux86-64/19.10/bin/siterc && \
    echo "append LDLIBARGS=\$library_path;" >> /opt/pgi/linux86-64/19.10/bin/siterc && \
    ln -sf /usr/lib/x86_64-linux-gnu/libnuma.so.1 /opt/pgi/linux86-64/19.10/lib/libnuma.so && \
    ln -sf /usr/lib/x86_64-linux-gnu/libnuma.so.1 /opt/pgi/linux86-64/19.10/lib/libnuma.so.1 && \
    rm -rf /var/tmp/pgi-community-linux-x64-latest.tar.gz /var/tmp/pgi
ENV LD_LIBRARY_PATH=/opt/pgi/linux86-64/19.10/mpi/openmpi-3.1.3/lib:/opt/pgi/linux86-64/19.10/lib:$LD_LIBRARY_PATH \
    PATH=/opt/pgi/linux86-64/19.10/mpi/openmpi-3.1.3/bin:/opt/pgi/linux86-64/19.10/bin:$PATH
 
COPY poisson_mpi.f90 /var/tmp/poisson_mpi.f90
 
RUN mpif90 /var/tmp/poisson_mpi.f90 -o /usr/local/bin/poisson_mpi

Building the Images

I leave the details on how to build images to you. The build command for the Singularity image is:

$ sudo singularity build poisson.sif Singularity.def

Note that sudo was used instead of fakeroot, which would have allowed the image to be built as a non-privileged user. At the time of writing, I did not build fakeroot capability into Singularity. I also did not use encryption, so the steps for running MPI applications in containers would be more clear.

You can easily check whether the build was successful by listing the files in the directory (Listing 5).

Listing 5: Checking the Singularity Build

$ ls -s
total 2444376
      4 buildit.docker        4 poisson-docker.py        4 runit.docker
      4 buildit.sing         20 poisson_mpi.f90          4 runit.sing
      4 Dockerfile            4 poisson_mpi.txt          4 Singularity.def
      4 hosts                 4 poisson.py              16 summary-notes.txt
      4 ompi.env        2444296 poisson.sif

The build command for the Docker image is:

$ sudo docker build -t poisson -f Dockerfile .

Notice that a version was not used with the image tag (the image is named poisson).To check whether the build was successful, display a list of images on the host system (Listing 6).

Listing 6: Checking the Docker Build

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
poisson             latest              0a7e2fad652e        2 hours ago         9.83GB
                                        49cbd14ae32f        3 hours ago         269MB
ubuntu              18.04               72300a873c2c        3 weeks ago         64.2MB
hello-world         latest              fce289e99eb9        14 months ago       1.84kB

Running the Containers

In this section, I show how to run MPI applications that are in containers for both Singularity and Docker.

Singularity

Rather than show the output from the MPI application for each command-line options, Listing 7 shows sample output from a Singularity container run that represents the output for all options.

Listing 7: Singularity Container Run

POISSON_MPI - Master process:
  FORTRAN90 version
  
  A program to solve the Poisson equation.
  The MPI message passing library is used.
  
  The number of interior X grid lines is        9
  The number of interior Y grid lines is        9
  
  The number of processes is   2
  
INIT_BAND - Master Cartesian process:
  The X grid spacing is   0.100000    
  The Y grid spacing is   0.100000    
  
INIT_BAND - Master Cartesian process:
  Max norm of boundary values =   0.841471    
  
POISSON_MPI - Master Cartesian process:
  Max norm of right hand side F at interior nodes =    1.17334
  
Step    ||U||         ||Unew||     ||Unew-U||
  
       1  0.403397      0.497847      0.144939
       2  0.570316      0.604994      0.631442E-01
       3  0.634317      0.651963      0.422323E-01
       4  0.667575      0.678126      0.308531E-01
       5  0.687708      0.694657      0.230252E-01
       6  0.701077      0.705958      0.185730E-01
       7  0.710522      0.714112      0.154310E-01
       8  0.717499      0.720232      0.131014E-01
       9  0.722829      0.724966      0.111636E-01
      10  0.727009      0.728716      0.955921E-02
      11  0.730356      0.731745      0.825889E-02
      12  0.733083      0.734229      0.726876E-02
      13  0.735336      0.736293      0.641230E-02
      14  0.737221      0.738029      0.574042E-02
      15  0.738814      0.739502      0.514485E-02
      16  0.740172      0.740762      0.461459E-02
      17  0.741339      0.741849      0.414240E-02
      18  0.742348      0.742791      0.372162E-02
      19  0.743225      0.743613      0.334626E-02
      20  0.743993      0.744333      0.301099E-02
      21  0.744667      0.744967      0.271109E-02
      22  0.745261      0.745526      0.244257E-02
      23  0.745787      0.746022      0.220180E-02
      24  0.746254      0.746463      0.198567E-02
      25  0.746669      0.746855      0.179151E-02
      26  0.747039      0.747205      0.161684E-02
      27  0.747369      0.747518      0.145969E-02
      28  0.747665      0.747799      0.131971E-02
      29  0.747930      0.748050      0.119370E-02
      30  0.748168      0.748276      0.107971E-02
      31  0.748382      0.748479      0.976622E-03
  
POISSON_MPI - Master process:
  The iteration has converged
  
POISSON_MPI:
  Normal end of execution.

You can execute an MPI application in Singularity containers in two ways, both of which execute the application, which is the ultimate goal. Note that in both cases, being a privileged user (e.g.,root) is not required.

The first way to run the MPI code in the Singularity container is:

$ singularity exec poisson.sif mpirun --mca mpi_cuda_support 0 -n 2 /usr/local/bin/poisson_mpi

In this case, any support for CUDA is turned off explicitly with the --mca mpi_cuda_support 0 option, which is also used in the next method.

The second way to execute MPI code in a Singularity container is:

$ mpirun -verbose --mca mpi_cuda_support 0 -np 2 singularity exec poisson.sif /usr/local/bin/poisson_mpi

Running the MPI code with mpirun, uses the Singularity container as an application, just like any other application.

If you examine the command a little closer, you will notice that mpirun is used “outside” the container and is run in the host operating system (OS). Make sure (1) the command in $PATH and (2) the MPI version on the host OS are compatible with the MPI library in the container. On examination of the command, you can see that the MPI initialization is done in the host OS, not in the container. The command line looks just like you run any other MPI application.

Overall, this approach could make the container more portable, because if the MPI versions are compatible, you don’t have to worry about which version of the MPI implementation is in the container, compared with on the host. It just all works, as long as the MPI versions are compatible.

Docker

Docker was not originally designed for HPC, but over time, it has adopted HPC and AI capabilities. The method for running MPI applications requires running as a privileged user (root):

$ sudo docker run --rm -it poisson mpirun --allow-run-as-root -n 2 /usr/local/bin/poisson_mpi

Note that although you can run an MPI application in a Docker container other ways, here, the container is run as a privileged user (root), because in my opinion, this is the simplest method I’ve found. You don’t treat the container as an application, but it does get the job of running the MPI application done.

Summary

When containers broke into the mainstream (I'm not counting chroot), HPC was ignored. However, some container ideals are very useful in the HPC environment, and one key HPC capability is running MPI-based applications.

In this article, I explained how to run MPI applications in Singularity and Docker containers. The best way to get started is to use HPCCM to write a simple recipe that can then be used to create the appropriate specification file for Singularity or Docker. Then, you can keep the HPCCM recipe as the “source” for the container spec files, making container life just a bit easier.

I also illustrated how to execute the containers so that the MPI application can run. Although the application was very simple, the examples did show how the process works and how straightforward and, dare I say, simple, the process can be.