2023-04-02: The Singularity software was migrated to Apptainer in November 2021. We will suggest using the apptainer command, but the alias singularity will continue to work for the time being.
The Wynton HPC cluster supports Linux containers via the Apptainer software (formerly known as Singularity). A Linux container is an efficient and powerful virtualization method for running isolated Linux systems (“containers”) on any Linux system including CentOS used by our cluster.
Apptainer, and Linux containers in general, provides solutions to common problems and hurdles that HPC users often face, e.g.
Because you can create and customize your own containers, and because Apptainer also supports Docker containers, you have immediate access to a very large number of Apptainer and Docker containers available via repositories such as:
All tasks for using Linux containers, such as downloading, building, and running containers, is done via the apptainer client and supported on Wynton HPC. The most common command calls are:
apptainer run <image> - run predefined script within containerapptainer exec <image> - execute any command within containerapptainer shell <image> - run bash shell within containerapptainer build <url> - import an existing Apptainer or Docker container available onlineapptainer build --remote <image> <def file> - build a Apptainer from local specifications in definition fileFor full details, see apptainer --help, man apptainer, and the Apptainer website.
As an illustration on how to use Linux containers with Apptainer, we will use the Docker container rocker/r-base available on Docker Hub. This particular container provides the latest release of the R software in an Ubuntu OS environment. Containers available from Docker Hub, Biocontainers, and elsewhere, can be downloaded and used analogously.
To use this rocker/r-base container, we first pull it down to a Apptainer image file ~/lxc/rocker_r-base.sif as:
[alice@dev2 ~]$ mkdir lxc
[alice@dev2 ~]$ cd lxc/
[alice@dev2 lxc]$ apptainer build rocker_r-base.sif docker://rocker/r-base
INFO: Starting build...
Getting image source signatures
Copying blob sha256:7b303595d9b321a9020d0ddbf1dea4c83237e2367117606a8d5466c446714ba1
Copying blob sha256:83ca482c84ae83b4e39c6cad8519cde7b2cb5b8f1d181b5be11271d91c01b583
Copying blob sha256:6046dc19e2b25f9181d9e5e9347fff916012d426f1de82aaaba5f0d29b647aeb
Copying blob sha256:ed3a653d2e7952b3187ab9cfc7be8cde4651e994d1085a1102a621e952530d9e
Copying blob sha256:111255bd9d45300e28c8a7727ad43f2979d66d6ac94d472699fe40fb079945a8
Copying blob sha256:f9510faa489c34c75dd8d92128e890d135ddd7aeb64c293306563abee64eb53b
Copying config sha256:b23cdc375e4c59d8a9c321b7b308116e0e3c2329c45f9be18d35dc95398565dc
Writing manifest to image destination
Storing signatures
2022/03/29 18:43:54 info unpack layer: sha256:7b303595d9b321a9020d0ddbf1dea4c83237e2367117606a8d5466c446714ba1
2022/03/29 18:43:57 info unpack layer: sha256:83ca482c84ae83b4e39c6cad8519cde7b2cb5b8f1d181b5be11271d91c01b583
2022/03/29 18:43:57 info unpack layer: sha256:6046dc19e2b25f9181d9e5e9347fff916012d426f1de82aaaba5f0d29b647aeb
2022/03/29 18:43:58 info unpack layer: sha256:ed3a653d2e7952b3187ab9cfc7be8cde4651e994d1085a1102a621e952530d9e
2022/03/29 18:43:59 info unpack layer: sha256:111255bd9d45300e28c8a7727ad43f2979d66d6ac94d472699fe40fb079945a8
2022/03/29 18:43:59 info unpack layer: sha256:f9510faa489c34c75dd8d92128e890d135ddd7aeb64c293306563abee64eb53b
INFO: Creating SIF file...
INFO: Build complete: rocker_r-base.sif
[alice@dev2 lxc]$ ls -l rocker_r-base.sif
-rwxr-xr-x. 1 alice boblab 318054400 Mar 29 18:44 rocker_r-base.sif
The above may take a minute or two to complete.
After this, we can run R within this container using:
[alice@dev2 lxc]$ apptainer run rocker_r-base.sif
R version 4.1.3 (2022-03-10) -- "One Push-Up"
Copyright (C) 2022 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> sum(1:10)
[1] 55
> q()
Save workspace image? [y/n/c]: n
[alice@dev2 lxc]$
Exactly what is “run” is defined by the so called %runscript of the Apptainer container, or the CMD entry if imported from a Docker container. An alternative way to launch R within this container is by explicitly executing R, e.g.
[alice@dev2 lxc]$ apptainer exec rocker_r-base.sif R --quiet
> sum(1:10)
[1] 55
> q("no")
[alice@dev2 lxc]$
Note that, the Apptainer image is marked as an executable, which means you can run it as any other executable, e.g.
[alice@dev2 lxc]$ ./rocker_r-base.sif
R version 4.1.3 (2022-03-10) -- "One Push-Up"
Copyright (C) 2022 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> sum(1:10)
[1] 55
> q("no")
[alice@dev2 lxc]$
To launch a shell within this container, and to also convince yourselves that the container runs Ubuntu (and not CentOS as on the Wynton HPC host system), do:
[alice@dev2 lxc]$ apptainer shell rocker_r-base.sif
Apptainer> head -3 /etc/os-release
PRETTY_NAME="Debian GNU/Linux buster/sid"
NAME="Debian GNU/Linux"
ID=debian
Apptainer> Rscript --version
R scripting front-end version 4.1.3 (2022-03-10)
Copyright (C) 2022 The R Foundation for Statistical Computing
Apptainer> exit
[alice@dev2 lxc]$ head -3 /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
When running a container, only a few of the folders available “outside” are available “inside” the container. By default, you have access to the current working directory (= $PWD) and your home folder (= $HOME). In contrast, without further specifications, you will not have access to standard folders such as local /scratch and global /wynton/scratch. Similarly, lab folders such as /wynton/group/boblab are not available from inside the container.
[alice@dev2 lxc]$ apptainer shell rocker_r-base.sif
Apptainer> ls /scratch
ls: cannot access '/scratch': No such file or directory
Apptainer> ls /wynton/scratch
ls: cannot access '/wynton/scratch': No such file or directory
Apptainer> ls /wynton/group/boblab
ls: cannot access '/wynton/group/boblab': No such file or directory
Apptainer> echo $TMPDIR
/scratch/alice
Apptainer> ls "$TMPDIR"
ls: cannot access '/scratch/alice': No such file or directory
To make also these folders available within the container, we can use apptainer option --bind. In its simplest form, we can just list the folders we want to make available, e.g.
[alice@dev2 lxc]$ apptainer shell --bind /scratch,/wynton/scratch,/wynton/group/boblab rocker_r-base.sif
Apptainer> ls /scratch
alice
Apptainer> ls /wynton/scratch
alice
Apptainer> ls /wynton/group/boblab
data1 data2
See apptainer help instance start for more details and other ways to mount and rename folders within the container.
When it comes to the scheduler, there is nothing special about Apptainer per se - the Apptainer software can be used as any other software on the cluster. As a proof of concept, here is how to calculate the sum of one to ten using R within the above Linux container at the command line:
[alice@dev2 lxc]$ apptainer exec rocker_r-base.sif Rscript -e "sum(1:10)"
[1] 55
[alice@dev2 lxc]$
To run this as a batch job, we need to create a job script.
[alice@dev2 lxc]$ cat demo-apptainer.sh
#!/usr/bin/bash
#$ -N demo-apptainer
#$ -l mem_free=100M
#$ -l h_rt=00:05:00
#$ -j y
apptainer exec rocker_r-base.sif Rscript -e "sum(1:10)"
And now submit with qsub:
[alice@dev2 lxc]$ qsub demo-apptainer.sh
Your job 1657 ("hello_world") has been submitted
Check results:
[alice@dev2 lxc]$ cat demo-apptainer.o5987
[1] 55
Sometimes you need to build custom Linux container from a definition
file, e.g. a Dockerfile or a *.def file. In the past, when
Singularity was used, this required administrative (“sudo”)
privileges. However, with the introduction of Apptainer, any user can
now build container images also from scratch from one of the
Wynton HPC development nodes.
For example, consider the following isoseq3.def file, which builds
upon a Docker Miniconda3 image (available at
https://hub.docker.com/r/continuumio/miniconda3) and extends it by
installing isoseq3 from the Bioconda channel (available at
https://anaconda.org/bioconda/isoseq3):
Bootstrap: docker
From: continuumio/miniconda3
%post
/opt/conda/bin/conda config --add channels bioconda
/opt/conda/bin/conda install isoseq3
%runscript
isoseq3 "$@"
To build a container image from this definition file, use apptainer
build as in:
[alice@dev2 lxc]$ apptainer build isoseq3.sif isoseq3.def
INFO: Starting build...
Getting image source signatures
Copying blob 7b4354700ca4 done
Copying blob 3f4ca61aafcd done
Copying blob 69a5d9e1ecd6 done
Copying config 8c73281886 done
Writing manifest to image destination
Storing signatures
2023/01/24 13:50:37 info unpack layer: sha256:3f4ca61aafcd4fc07267a105067db35c0f0ac630e1970f3cd0c7bf552780e985
2023/01/24 13:50:39 info unpack layer: sha256:69a5d9e1ecd6566da53d0978004bdf37dddfaba1d8a6117966f397b41cbbc529
2023/01/24 13:50:42 info unpack layer: sha256:7b4354700ca480732ead22a553cc45916dc5466709ca64d964c4647b5b9343e9
INFO: Running post scriptlet
+ /opt/conda/bin/conda config --add channels bioconda
+ /opt/conda/bin/conda install isoseq3
Collecting package metadata (current_repodata.json): done
…
done
INFO: Adding runscript
INFO: Creating SIF file...
INFO: Build complete: isoseq3.sif
[alice@dev2 lxc]$
The results is a container image file named isoseq3.sif:
[alice@dev2]$ ls -l isoseq3.sif
-rwxr-xr-x. 1 alice boblab 164265984 Jan 24 13:51 isoseq3.sif
Because the definition file has an %runscript entry, we can call this image directly as-is, e.g.
[alice@dev2]$ ./isoseq3.sif --version
isoseq3 3.8.2 (commit v3.8.2)
Using:
pbbam : 2.2.0 (commit v2.2.0-1-g8c081f6)
pbcopper : 2.1.0 (commit v2.1.0)
pbmm2 : 1.9.0 (commit v1.9.0-2-gbec8e28)
minimap2 : 2.15
parasail : 2.1.3
boost : 1.77
htslib : 1.14
zlib : 1.2.11
[alice@dev2]$
Q. Why not Docker?
A. Docker is one of the most popular and well-known software solutions for using Linux Containers. However, contrary to Apptainer, it turns out that it is hard to get Docker to play well with multi-tenant HPC environments.
Q. What happened to Singularity?
A. The Apptainer software is a fork of the Singularity software from 2021. For backward compatibility, the singularity command is an alias to the apptainer command. We suggest that you update your script to use apptainer.
Q. What’s the filename extension *.sif?
A. First of all, the filename extension is optional, and some prefer to drop them, e.g. rocker_r-base instead of rocker_r-base.sif. SIF, which is short for the Singularity Container Image Format, is a file format that can hold a Linux container environments in a single file. You might also run into old Singularity images named *.img and *.simg, which are legacy file formats that Singularity used in the past, where *.img indicates a writable (ext3) images whereas *.simg indicates a read-only (squashfs) image.