A Container for HPC

Fusion

Singularity Is Not Just for HPC

Although the motivation for Singularity was based in the HPC world, it can be used for virtually any application in which mobility of compute, application appliances, or workflow distribution is important. Nothing limits the type of application that can be in a Singularity container. In fact, Singularity goes a step beyond other container systems and provides a very clever way of running virtually all applications.

Summary

Containers are all the rage today for a very good reason: Users and developers want to be able to develop on one distribution and execute on another. This capability divorces the developers from the production environment, allowing them to develop an application whenever they want (or need) and insert that application into the production environment.

Containers also allow applications and services to be isolated from one another on a system for a variety of reasons, including security and stability. I've heard of people running each of their applications in a separate container isolated from each other. They might run Firefox in its own container, gcc in another container, and even a code editor in another container. Granted this is not the most space-efficient way of running systems, but it does allow you to isolate applications from one another.

The most popular container system today is arguably Docker. People have experimented with it for HPC applications but have not found it to be the most appropriate container because – among other reasons – it's possible for users to gain root access. Solutions for getting around this usually involve running a cluster environment on an existing cluster.

Singularity is a new container paradigm that focuses specifically on one of the core missions of containers: mobility of compute. Using a SPEC file, very much like a SPEC file for RPMs, Singularity creates a container (.sapp file), examines the targeted binary for dependencies, and packages the application, the dependencies, and a couple of small tools into an executable package. These Singularity containers can interact with files local to the container in the directory where the container is executed.

Singularity containers can be created for any application and are not just limited to HPC applications. However, one of the absolute coolest things that Singularity can do is create simple containers for MPI applications. These have been very difficult for other container systems to handle, but Singularity treats the applications like an executable, so the mpirun command treats it like any other executable.

Singularity is still developing, but it can already be used to create and run containers for large MPI applications. I tested several existing applications, and they ran without a problem. I highly recommend giving Singularity a try, especially if you are running HPC applications.

Infos

  1. SR-IOV: https://en.wikipedia.org/wiki/X86_virtualization#PCI-SIG_Single_Root_I.2FO_Virtualization_.28SR-IOV.29
  2. Hardware-level virtualization: https://en.wikipedia.org/wiki/Hardware_virtualization
  3. Operating system virtualization: https://en.wikipedia.org/wiki/Operating-system-level_virtualization
  4. Docker: http://www.docker.com/
  5. Cgroups: https://en.wikipedia.org/wiki/Cgroups
  6. Namespaces: https://en.wikipedia.org/wiki/Cgroups#Namespace_isolation
  7. Union filesystem: http://stackoverflow.com/questions/32775594/why-does-docker-need-a-union-file-system
  8. AuFS: https://en.wikipedia.org/wiki/Aufs
  9. UberCloud: https://www.theubercloud.com/
  10. Shifter: https://www.nersc.gov/research-and-development/user-defined-images/
  11. NERSC: http://www.nersc.gov/
  12. CHOS: http://www.nersc.gov/users/computational-systems/pdsf/software-and-tools/chos/
  13. PDSF: http://www.nersc.gov/users/computational-systems/pdsf/about-pdsf/
  14. "Contain This, Unleashing Docker for HPC" by Doug Jacobsen and Shane Canon: https://www.nersc.gov/assets/Uploads/nersc-brownbag-docker-jacobsen-canon.pdf
  15. "Shifter Expands Container Capabilities for HPC" by Nicole Hemsoth: http://www.nextplatform.com/2015/09/23/shifter-expands-container-capabilities-for-hpc/
  16. Gregory M. Kurtzer on GitHub: http://gmkurtzer.github.io/
  17. "Interview with the Developer of Singularity" by Jeff Layton: http://www.admin-magazine.com/HPC/Articles/Interview-with-Gregory-Kurtzer-Developer-of-Singularity
  18. Singularity: http://singularity.lbl.gov/#home
  19. Singularity installation: http://singularity.lbl.gov/#install
  20. Linux Mint 17.2: https://www.linuxmint.com/

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy ADMIN Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

comments powered by Disqus
Subscribe to our ADMIN Newsletters
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs



Support Our Work

ADMIN content is made possible with support from readers like you. Please consider contributing when you've found an article to be beneficial.

Learn More”>
	</a>

<hr>		    
			</div>
		    		</div>

		<div class=