Cluster Specifications

Overview

Compute nodes
449
Physical cores
12572
GPUs
148 GPUs on 42 GPU nodes (48/11 GPUs/nodes are communal and 100/31 GPUs/nodes are prioritized for GPU contributors)
RAM
48-1512 GiB/node
Local scratch
0.1-1.8 TiB/node
Global scratch
615 TiB
User home storage
500 GiB/user (383 TiB in total)
Group storage
5.6 PB
Number of accounts
1272
Number of projects
371

Summary of Compute Environment

Feature Login Nodes Transfer Nodes Development Nodes Compute Nodes
SSH access from outside of cluster ✓ (2FA if outside of UCSF) ✓ (2FA if outside of UCSF) no no
SSH access from inside of cluster no
Outbound access Within UCSF only: SSH and SFTP HTTP/HTTPS, FTP/FTPS, SSH, SFTP, Globus Via proxy: HTTP/HTTPS, GIT+SSH(*) no
Network speed 1 Gbps 10 Gbps 1 Gbps 1,10,40 Gbps
Core software Minimal Minimal Same as compute nodes + compilers and source-code packages CentOS and EPEL packages
modules (software stacks) no no
Global file system
Job submission no
Purpose Submit and query jobs. SSH to development nodes. File management. Fast in- & outbound file transfers. File management. Compile and install software. Prototype and test job scripts. Submit and query jobs. Version control (clone, pull, push). File management. Running short and long-running job scripts.

(*) GIT+SSH access on development nodes is restricted to git.bioconductor.org, bitbucket.org, gitea.com, github.com / gist.github.com, gitlab.com, cci.lbl.gov, and git.ucsf.edu.

All nodes on the cluster runs CentOS 7 which is updated on a regular basis. The job scheduler is SGE 8.1.9 (Son of Grid Engine) which provides queues for both communal and lab-priority tasks.

Details

Login Nodes

The cluster can be accessed via SSH to one of two login nodes:

  1. log1.wynton.ucsf.edu
  2. log2.wynton.ucsf.edu

Data Transfer Nodes

For transferring large data files, it is recommended to use one of the dedicate data transfer nodes:

  1. dt1.wynton.ucsf.edu
  2. dt2.wynton.ucsf.edu

which both has a 10 Gbps connection - providing a file transfer speed of up to (theoretical) 1.25 GB/s = 4.5 TB/h. As for the login nodes, the transfer nodes can be accessed via SSH.

Comment: You can also transfer data via the login nodes, but since those only have 1 Gbps connections, you will see much lower transfer rates.

Development Nodes

The cluster has development nodes for the purpose of validating scripts, prototyping pipelines, compiling software, and more. Development nodes can be accessed from the login nodes.

Node Physical Cores RAM Local /scratch CPU GPU
dev1 72 384 GiB 0.93 TiB Intel Gold 6240 2.60GHz  
dev2 32 512 GiB 1.1 TiB Intel Xeon E5-2640 v3 2.60GHz  
dev3 32 512 GiB 1.1 TiB Intel Xeon E5-2640 v3 2.60GHz  
gpudev1 32 128 GiB 0.82 TiB Intel Xeon E5-2640 v3 2.60GHz NVIDIA Tesla K80

Comment: Please use the GPU development node only if you need to build or prototype GPU software.

Compute Nodes

The majority of the compute nodes have Intel processors, while a few have AMD processes. Each compute node has a local /scratch drive (see above for size), which is either a hard disk drive (HDD), a solid state drive (SSD), or even a Non-Volatile Memory Express (NVMe) drive. Each node has a tiny /tmp drive (4-8 GiB).

The compute nodes can only be utilized by submitting jobs via the scheduler - it is not possible to explicitly log in to compute nodes.

File System

Scratch Storage

The Wynton HPC cluster provides two types of scratch storage:

There are no per-user quotas in these scratch spaces. Files not added or modified during the last two weeks will be automatically deleted on a nightly basis. Note, files with old timestamps that were “added” to the scratch place during this period will not be deleted, which covers the use case where files with old timestamps are extracted from tar.gz file. (Details: tmpwatch --ctime --dirmtime --all --force is used for the cleanup.)

User and Lab Storage

Each user may use up to 500 GiB disk space in the home directory. It is not possible to expand user’s home directory. Research groups can add additional storage space under /wynton/group by either mounting their existing storage or purchase new.

Network

The majority of the compute nodes are connected to the local network with 1 Gbps and 10 Gbps network cards while a few got 40 Gbps cards.

The cluster itself connects to NSF’s Pacific Research Platform at a speed of 100 Gbps - providing a file transfer speed of up to (theoretical) 12.5 GB/s = 45 TB/h.