Examples not working?

Work with Conda #

Conda is a package manager and an environment management system. It’s popular, because it simplifies installation of many scientific software tools. There are two main distributions of Conda:

  1. Anaconda - comes with more than 1,500 scientific packages (~3 GiB of disk space) [not preinstalled on Wynton]
  2. Miniconda - a small subset of the much larger Anaconda distribution (~0.5 GiB of disk space) [recommended; preinstalled on Wynton]

Both come with Python and conda commands. We recommend working with the smaller Miniconda distribution, especially since it is preinstalled on Wynton. Using Miniconda, you can install additional scientific packages as needed using the conda install ... command.

Note: The aim of this document is to give the essential best-practices for working with Conda on Wynton. It is not meant to be a complete introduction on how to work with Conda in general. A good complement to this document, is the official Conda documentation on Managing environments.

Loading Miniconda #

On Wynton HPC, up-to-date versions of the Miniconda distribution are available via the CBI software stack. There is no need for you to install this yourself. To load Miniconda v3 with Python 3.9, call:

[alice@dev1 ~]$ module load CBI miniconda3/23.5.2-0-py311

This gives access to:

[alice@dev1 ~]$ conda --version
conda 23.5.2
[alice@dev1 ~]$ python --version
Python 3.11.4

To see what software packages come with this Miniconda distribution, call:

[alice@dev1 ~]$ conda list
# packages in environment at /wynton/home/boblab/shared/software/CBI/miniconda3-23.5.2-0-py311:
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
boltons                   23.0.0          py311h06a4308_0  
brotlipy                  0.7.0           py311h5eee18b_1002  
bzip2                     1.0.8                h7b6447c_0  
c-ares                    1.19.0               h5eee18b_0  
ca-certificates           2023.05.30           h06a4308_0  
certifi                   2023.5.7        py311h06a4308_0  
cffi                      1.15.1          py311h5eee18b_3  
charset-normalizer        2.0.4              pyhd3eb1b0_0  
conda                     23.5.2          py311h06a4308_0  
conda-content-trust       0.1.3           py311h06a4308_0  
conda-libmamba-solver     23.5.0          py311h06a4308_0  
conda-package-handling    2.1.0           py311h06a4308_0  
conda-package-streaming   0.8.0           py311h06a4308_0  
cryptography              39.0.1          py311h9ce1e76_2  
fmt                       9.1.0                hdb19cb5_0  
icu                       58.2                 he6710b0_3  
idna                      3.4             py311h06a4308_0  
jsonpatch                 1.32               pyhd3eb1b0_0  
jsonpointer               2.1                pyhd3eb1b0_0  
krb5                      1.20.1               h143b758_1  
ld_impl_linux-64          2.38                 h1181459_1  
libarchive                3.6.2                h6ac8c49_2  
libcurl                   8.1.1                h251f7ec_1  
libedit                   3.1.20221030         h5eee18b_0  
libev                     4.33                 h7f8727e_1  
libffi                    3.4.4                h6a678d5_0  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libmamba                  1.4.1                h2dafd23_1  
libmambapy                1.4.1           py311h2dafd23_1  
libnghttp2                1.52.0               h2d74bed_1  
libsolv                   0.7.22               he621ea3_0  
libssh2                   1.10.0               hdbd6064_2  
libstdcxx-ng              11.2.0               h1234567_1  
libuuid                   1.41.5               h5eee18b_0  
libxml2                   2.10.3               hcbfbd50_0  
lz4-c                     1.9.4                h6a678d5_0  
ncurses                   6.4                  h6a678d5_0  
openssl                   3.0.9                h7f8727e_0  
packaging                 23.0            py311h06a4308_0  
pcre2                     10.37                he7ceb23_1  
pip                       23.1.2          py311h06a4308_0  
pluggy                    1.0.0           py311h06a4308_1  
pybind11-abi              4                    hd3eb1b0_1  
pycosat                   0.6.4           py311h5eee18b_0  
pycparser                 2.21               pyhd3eb1b0_0  
pyopenssl                 23.0.0          py311h06a4308_0  
pysocks                   1.7.1           py311h06a4308_0  
python                    3.11.4               h955ad1f_0  
readline                  8.2                  h5eee18b_0  
reproc                    14.2.4               h295c915_1  
reproc-cpp                14.2.4               h295c915_1  
requests                  2.29.0          py311h06a4308_0  
ruamel.yaml               0.17.21         py311h5eee18b_0  
setuptools                67.8.0          py311h06a4308_0  
six                       1.16.0             pyhd3eb1b0_1  
sqlite                    3.41.2               h5eee18b_0  
tk                        8.6.12               h1ccaba5_0  
toolz                     0.12.0          py311h06a4308_0  
tqdm                      4.65.0          py311h92b7b1e_0  
tzdata                    2023c                h04d1e81_0  
urllib3                   1.26.16         py311h06a4308_0  
wheel                     0.38.4          py311h06a4308_0  
xz                        5.4.2                h5eee18b_0  
yaml-cpp                  0.7.0                h295c915_1  
zlib                      1.2.13               h5eee18b_0  
zstandard                 0.19.0          py311h5eee18b_0  
zstd                      1.5.5                hc292b87_0  

Creating a Conda environment (required) #

A Conda environment is a mechanism for installing extra software tools and versions beyond the base Miniconda distribution in a controlled manner. When using the miniconda3 module, a Conda environment must be used to install extra software. The following command creates a new myjupyter environment:

[alice@dev1 ~]$ conda create -n myjupyter notebook
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /wynton/home/boblab/alice/.conda/envs/myjupyter

  added / updated specs:
    - notebook

The following packages will be downloaded:

    package                    |            build
    anyio-3.5.0                |  py311h06a4308_0         214 KB
    argon2-cffi-bindings-21.2.0|  py311h5eee18b_0          33 KB
    xz-5.4.2                   |       h5eee18b_0         642 KB
    yaml-0.2.5                 |       h7b6447c_0          75 KB
                                           Total:        67.6 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main 
  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu 
  zeromq             pkgs/main/linux-64::zeromq-4.3.4-h2531618_0 
  zlib               pkgs/main/linux-64::zlib-1.2.13-h5eee18b_0 

Proceed ([y]/n)? y

Downloading and Extracting Packages
jupyterlab_pygments- | 8 KB      | #################################### | 100% 
jupyter_client-7.3.5 | 194 KB    | #################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
# To activate this environment, use
#     $ conda activate myjupyter
# To deactivate an active environment, use
#     $ conda deactivate

[alice@dev1 ~]$ 

By default, the environment is created in your home directory under ~/.conda/. To create the environment at a specific location, see Managing environments part of the official Conda documentation. That section covers many useful topics such as removing a Conda environment, and creating an environment with a specific Python version.

Activating a Conda environment (required) #

After an environment is created, the next time you log in to a development node, you can set myjupyter (or any other Conda environment you’ve created) as your active environment by calling:

[alice@dev1 ~]$ module load CBI miniconda3
[alice@dev1 ~]$ conda activate myjupyter
(myjupyter) [alice@dev1 ~]$ jupyter notebook --version

(myjupyter) [alice@dev1 ~]$ 

Note how the command-line prompt is prefixed with (myjupyter); it highlights that the Conda environment myjupyter is activated. To deactivate an environment and return to the base environment, call:

(myjupyter) [alice@dev1 ~]$ conda deactivate
[alice@dev1 ~]$ jupyter notebook --version
jupyter: command not found

[alice@dev1 ~]$ 

We highly recommend configuring Conda environment to be automatically staged only on the local disk whenever activated. This results in your software running significantly faster. Auto-staging is straightforward to configure using the conda-stage tool, e.g.

[alice@dev1 ~]$ module load CBI miniconda3
[alice@dev1 ~]$ module load CBI conda-stage
[alice@dev1 ~]$ conda activate myjupyter
(myjupyter) [alice@dev1 ~]$ conda-stage --auto-stage=enable
INFO: Configuring automatic staging and unstaging of original Conda environment  ...
INFO: Enabled auto-staging
INFO: Enabled auto-unstaging

For the complete, easy-to-follow instructions, see the conda-stage documentation.

Once you have your Conda environment built, we recommend that you back up its core configuration. The process is quick and straightforward, and fully documented in the official Conda Managing environments documentation. For example, to back up the above myjupyter environment, call:

[alice@dev1 ~]$ conda env export --name myjupyter | grep -v "^prefix: " > myjupyter.yml
[alice@dev1 ~]$ ls -l myjupyter.yml
-rw-rw-r-- 1 alice boblab 3597 Jun 11 03:48 myjupyter.yml

This configuration file is useful:

To restore a backed up Conda environment from a yaml file, on the target machine:

  1. download the yaml file, e.g. myjupyter.yml

  2. make sure there is no existing environment with the same name, i.e. check conda env list

  3. create a new Conda environment from the yaml file

For example, assume we have downloaded myjupyter.yml to our local machine. Then we start by getting the name of the backed-up Conda environment and making sure it does not already exist;

{local}$ grep "name:" myjupyter.yml
name: myjupyter

{local}$ conda env list
# conda environments:
sandbox  /wynton/home/boblab/alice/.conda/envs/sandbox
base     /path/to/anaconda3

When have confirmed that there is no name clash, we can restore the backed up environment on our local machine using:

{local}$ conda env create -f myjupyter.yml 

This will install the exact same software versions as when we made the backup.

Warning: This is not a fool-proof backup method, because it depends on packages to be available from the package repositories also when you try to restore the Conda environment. To lower the risk for failure, keep your environments up to date with the latest packages and test frequently that your myjupyter.yml file can be restored.

We can also use myjupyter.yml to update an existing environment. The gist is to edit the file to reflect what we want to be updated, and then run conda env update .... See Managing environments part of the official Conda documentation for exact instructions.

Conda revisions #

If you updated or installed new packages in your Conda environment and need to roll back to a previous version, it is possible to do this using Conda’s revision utility. To list available revisions in the current Conda environment, use:

[alice@dev1 ~]$ CONDA_STAGE=false conda activate myjupyter
(myjupyter) [alice@dev1 ~]$ conda list --revisions
2023-06-11 03:46:03  (rev 0)
    +_libgcc_mutex-0.1 (defaults/linux-64)
    +_openmp_mutex-5.1 (defaults/linux-64)
    +anyio-3.5.0 (defaults/linux-64)
    +argon2-cffi-21.3.0 (defaults/noarch)

2023-06-11 03:52:03  (rev 1)
    +conda-pack-0.7.0 (conda-forge/noarch)
    +python_abi-3.11 (conda-forge/linux-64)

To roll back to a specific revision, say revision zero, use:

(myjupyter) [alice@dev1 ~]$ conda install --revision 0

Warning: This only works with packages installed using conda install. Packages installed via python3 -m pip install will not be recorded by the revision system. In such cases, we have to do manual backup snapshots (as explained above).

Appendix #

If you previously have installed Conda yourself, there is a risk that it installed itself into your ~/.bashrc file such that it automatically activates the ‘base’ environment whenever you start a new shell, e.g.

{local}$ ssh alice@log1.wynton.ucsf.edu
alice1@log1.wynton.ucsf.edu:s password: XXXXXXXXXXXXXXXXXXX
(base) [alice@log1 ~]$ 

If you see a (base) prefix in your prompt, then you have this set up and the Conda ‘base’ environment is active. You can verify this by querying conda info as:

[alice@dev1 ~]$ conda info | grep active
conda info | grep active
     active environment : base
    active env location : /wynton/home/boblab/alice/miniconda3

This auto-activation might sound convenient, but we strongly recommend against using it, because Conda software stacks have a great chance to cause conflicts (read: wreak havoc) with other software tools installed outside of Conda. For example, people that have Conda activated and then run R via module load CBI r, often report on endless, hard-to-understand problems when trying to install common R packages. Instead, we recommend to activate your Conda environments only when you need them, and leave them non-activated otherwise. This will give you a much smoother day-to-day experience. To clarify, if you never installed Conda yourself, and only used module load CBI miniconda3-py39/23.3.1-0-py39, then you should not have this problem.

To reconfigure Conda to no longer activate the ‘base’ Conda environment by default, call:

(base) [alice@dev1 ~]$ conda config --set auto_activate_base false
(base) [alice@dev1 ~]$ 

Next time you log in, the ‘base’ environment should no longer be activated by default.

If you want to completely retire you personal Conda installation, and move on to only using module load CBI miniconda3, you can uninstall the Conda setup code that were injected to your ~/.bashrc file by calling:

[alice@dev1 ~]$ conda init --reverse
no change     /wynton/home/boblab/alice/miniconda3/condabin/conda
no change     /wynton/home/boblab/alice/miniconda3/bin/conda
no change     /wynton/home/boblab/alice/miniconda3/bin/conda-env
no change     /wynton/home/boblab/alice/miniconda3/bin/activate
no change     /wynton/home/boblab/alice/miniconda3/bin/deactivate
no change     /wynton/home/boblab/alice/miniconda3/etc/profile.d/conda.sh
no change     /wynton/home/boblab/alice/miniconda3/etc/fish/conf.d/conda.fish
no change     /wynton/home/boblab/alice/miniconda3/shell/condabin/Conda.psm1
no change     /wynton/home/boblab/alice/miniconda3/shell/condabin/conda-hook.ps1
no change     /wynton/home/boblab/alice/miniconda3/lib/python3.9/site-packages/xontrib/conda.xsh
no change     /wynton/home/boblab/alice/miniconda3/etc/profile.d/conda.csh
modified      /wynton/home/boblab/alice/.bashrc

==> For changes to take effect, close and re-open your current shell. <==

[alice@dev1 ~]$