Tips and Tricks for Containers

Inspecting a Docker Container

Containers are relatively new in computing, so when you pull or download container images, it might be a good idea to inspect them as best you can before using them to create a container. Moreover, inspecting a container to learn something from it that you can use in your own containers is a great way to move forward.

Docker has a couple of commands that can be useful in inspecting or learning about the container. The first straightforward command is docker inspect <image> (note that the output has been abbreviated):

$ docker inspect nvidia/cuda:10.1-base-ubuntu18.04
        "Id": "sha256:3b55548ae91f1928ae7315b9fe43b3ffa097a3da68f4be86d3481e857241acbb",
        "RepoTags": [
        "RepoDigests": [
        "Parent": "",
        "Comment": "",
        "Created": "2019-11-27T20:00:08.137590731Z",
        "Container": "f8cdd4d69d0b5123a712b66cd12a46799daff6e23896e73c6bfd247a981daa71",
        "ContainerConfig": {
            "Hostname": "f8cdd4d69d0b",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
        "RootFS": {
            "Type": "layers",
            "Layers": [
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"

A second option useful in inspecting a docker image is docker history:

$ docker history nvidia/cuda:10.1-base-ubuntu18.04
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
3b55548ae91f        5 months ago        /bin/sh -c #(nop)  ENV NVIDIA_REQUIRE_CUDA=c...   0B
           5 months ago        /bin/sh -c #(nop)  ENV NVIDIA_DRIVER_CAPABIL...   0B
           5 months ago        /bin/sh -c #(nop)  ENV NVIDIA_VISIBLE_DEVICE...   0B
           5 months ago        /bin/sh -c #(nop)  ENV LD_LIBRARY_PATH=/usr/...   0B
           5 months ago        /bin/sh -c #(nop)  ENV PATH=/usr/local/nvidi...   0B
           5 months ago        /bin/sh -c echo "/usr/local/nvidia/lib" >> /...   46B
           5 months ago        /bin/sh -c apt-get update && apt-get install...   25.1MB
           5 months ago        /bin/sh -c #(nop)  ENV CUDA_PKG_VERSION=10-1...   0B
           5 months ago        /bin/sh -c #(nop)  ENV CUDA_VERSION=10.1.243    0B
           5 months ago        /bin/sh -c apt-get update && apt-get install...   16.5MB
           5 months ago        /bin/sh -c #(nop)  LABEL maintainer=NVIDIA C...   0B
           6 months ago        /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B
           6 months ago        /bin/sh -c mkdir -p /run/systemd && echo 'do...   7B
           6 months ago        /bin/sh -c set -xe   && echo '#!/bin/sh' > /...   745B
           6 months ago        /bin/sh -c [ -z "$(apt-get indextargets)" ]     987kB
           6 months ago        /bin/sh -c #(nop) ADD file:a48a5dc1b9dbfc632...   63.2MB

By using the docker history command, you can almost reverse engineer a Dockerfile from an existing container:

$ docker history --format "{{.CreatedBy}}" --no-trunc nvidia/cuda:10.1-base-ubuntu18.04 | tac
/bin/sh -c #(nop) ADD file:a48a5dc1b9dbfc632f6cf86fe27b770b63f07a115c98c4465dc184e303a4efa1 in /
/bin/sh -c [ -z "$(apt-get indextargets)" ]
/bin/sh -c #(nop)  ENV CUDA_VERSION=10.1.243
/bin/sh -c #(nop)  ENV CUDA_PKG_VERSION=10-1=10.1.243-1
/bin/sh -c apt-get update && apt-get install -y --no-install-recommends         cuda-cudart-$CUDA_PKG_VERSION cuda-compat-10-1 && ln -s cuda-10.1 /usr/local/cuda &&     rm -rf /var/lib/apt/lists/*
/bin/sh -c echo "/usr/local/nvidia/lib" >> /etc/ &&     echo "/usr/local/nvidia/lib64" >> /etc/
/bin/sh -c #(nop)  ENV PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/bin/sh -c #(nop)  ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
/bin/sh -c #(nop)  ENV NVIDIA_VISIBLE_DEVICES=all
/bin/sh -c #(nop)  ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
/bin/sh -c #(nop)  ENV NVIDIA_REQUIRE_CUDA=cuda>=10.1 brand=tesla,driver>=384,driver<385 brand=tesla,driver>=396,driver<397 brand=tesla,driver>=410,driver<411

Inspecting a Singularity Container

As previously mentioned, you can inspect a Singularity image for the definition file. The same command with other options extracts more information. For example:

$ singularity inspect -l -r -d -e -t cuda_10_1-base-ubuntu18_04.simg
WARNING: No SIF metadata partition, searching in container...
bootstrap: docker
from: nvidia/cuda:10.1-base-ubuntu18.04
# prepare command line arguments for evaluation
for arg in "$@"; do
# ENTRYPOINT only - run entrypoint plus args
if [ -z "$OCI_CMD" ] && [ -n "$OCI_ENTRYPOINT" ]; then
    if [ $# -gt 0 ]; then
# CMD only - run CMD or override with args
if [ -n "$OCI_CMD" ] && [ -z "$OCI_ENTRYPOINT" ]; then
    if [ $# -gt 0 ]; then
# ENTRYPOINT and CMD - run ENTRYPOINT with CMD as default args
# override with user provided args
if [ $# -gt 0 ]; then
# Evaluate shell expressions first and set arguments accordingly,
# then execute final command as first container process
exec "$@"
# Custom environment shell code should follow Saturday_2_May_2020_10:52:41_EDT
org.label-schema.schema-version: 1.0
org.label-schema.usage.singularity.deffile.bootstrap: docker
org.label-schema.usage.singularity.deffile.from: nvidia/cuda:10.1-base-ubuntu18.04
org.label-schema.usage.singularity.version: 3.5.3

The various options applied to the original Singularity image (the one without Octave) are:

  • -d: show the image definition file
  • -e: show the environment settings for the image
  • -l: show the labels for the image
  • -t: show the test script for the image