Examples not working?

SBGrid Software Collection #

Access to SBGrid software on Wynton HPC is limited to paying members of the SBGrid Consortium.

For bug reports related to SBGrid software, please use the SBGrid’s Report Software Bug form The Wynton sysadmins do not fix problems in SBGrid software.

SBGrid is a collection of hundred of programs for structural biology. It is kept up to date by the SBGrid team without needing intervention by the Wynton admins. Each lab using SBGrid must pay a yearly fee. This covers usage by anyone in the lab, both on Wynton HPC and any other Linux or macOS computer they own. The annual price depends on how many other labs from UCSF join. Many UCSF labs are already members. See the SBGrid website for information on how to join.

Verify access to SBGrid #

The SBGrid software stack is installed on Wynton HPC, but is available only to paying members, which are listed on the SBGrid Member Labs webpage. As a Wynton HPC user part of a paying lab, you will have access to SBGrid on the Wynton HPC cluster from being part of the sbgrid Unix group. You can verify that this is get case by running:

[alice@dev2 ~]$ groups
boblab sbgrid

If sbgrid is not part of your output, then you currently do not have access to the SBGrid software on Wynton HPC. If you think this is a mistake, please contact us so we can add you.

Enable SBGrid #

SBGrid is available for interactive use on the development nodes and for use in job scripts on compute nodes. It is not available on the login or data-transfer nodes.

In order to use SBGrid software, the SBGrid environment must be enabled. To enable the SBGrid, in the shell or in a job script, call source /programs/sbgrid.shrc, e.g.

[alice@dev2 ~]$ source /programs/sbgrid.shrc
********************************************************************************
                  Software Support by SBGrid (www.sbgrid.org)
********************************************************************************
 Your use of the applications contained in the /programs  directory constitutes
 acceptance of  the terms of the SBGrid License Agreement included  in the file
 /programs/share/LICENSE.  The applications  distributed by SBGrid are licensed
 exclusively to member laboratories of the SBGrid Consortium.
              Run sbgrid-accept-license to remove the above message.  
********************************************************************************
 SBGrid was developed with support from its members, Harvard Medical School,    
 HHMI, and NSF. If use of SBGrid compiled software was an important element     
 in your publication, please include the following reference in your work:      
                                                                                      
 Software used in the project was installed and configured by SBGrid.                   
 cite: eLife 2013;2:e01456, Collaboration gets the most out of software.                
********************************************************************************
 SBGrid installation last updated: 2023-04-06
 Please submit bug reports and help requests to:       <bugs@sbgrid.org>  or
                                                       <https://sbgrid.org/bugs>
            For additional information visit https://sbgrid.org/wiki
********************************************************************************
********************************************************************************

Some SBGrid programs do not run on older compute nodes #

Many SBGrid programs run only on “modern” CPUs. Broadly speaking, there are four generations of CPUs, on the cluster - x86-64-v1, x86-64-v2, x86-64-v3, and x86-64-v4 - and SBGrid often requires x86-64-v3 compute nodes or newer. If your job fails with an obscure error such as:

Illegal instruction (core dumped)

 *** caught illegal operation ***
address 0x2b3a8b234ccd, cause 'illegal operand'

it most likely ended up on a x86-64-v1 or x86-64-v2 compute node, while the program required something newer. To avoid this from happening, specify the x86-64-v=<level> resource to request a compute node with x86-64-v3 or newer. Either specify command-line option -l x86-64-v=3 when you submit the job, or add it is as an SGE declaration in your script:

#$ -l x86-64-v=3  ## Script requires a CPU compliant with x86-64-v3 or newer

In the unlikely case that the SBGrid documentation says a program is compatible with pre-Haswell CPUs, it can run on also x86-64-v2 CPUs, and in some cases even x86-64-v1. If so, you can relax the x86-64 level requirement.

All our development nodes, including the GPU ones, have x86-64-v3 or newer CPUs.

SBGrid programs with GPU support #

If you are using SBGrid programs with GPU support, please note that SBGrid programs are compiled for specific versions of CUDA. Sometimes SBGrid provide different builds for multiple CUDA versions. For example, the same RELION version is available for different CUDA versions. Because of this, you have to make sure you load a corresponding CUDA environment module, e.g. module load cuda/10.1.

As of 2022-09-16, most default versions of GPU-compatible SBGrid programs are not compiled against CUDA 11.0.2, or greater. However, note the newest NVIDIA A40 GPUs on the qb3-atgpu* compute nodes require the use of programs compiled against CUDA 11.0.2, or greater, which means those compute nodes may not be compatible with the SBGrid program you want to run. You may need to specify a beta version of the SBGrid programs, or avoid the qb3-atgpu* nodes. See the SBGrid documentation for your specific program.