|Feature||Login Nodes||Transfer Nodes||Development Nodes||Compute Nodes|
|Accessible via SSH from outside of cluster||✓ (2FA if outside of UCSF)||✓ (2FA if outside of UCSF)||no||no|
|Accessible via SSH from within cluster||✓||✓||✓||no|
|Outbound access||Within UCSF only: SSH and SFTP||HTTP/HTTPS, FTP/FTPS, SSH, SFTP, Globus||Via proxy: FTP, HTTP/HTTPS, GIT+SSH(*)||no|
|Network speed||1 Gbps||10 Gbps||1 Gbps||1,10,40 Gbps|
|Core software||Minimal||Minimal||Same as compute nodes + compilers and source-code packages||CentOS and EPEL packages|
|modules (software stacks)||no||no||✓||✓|
|Global file system||✓||✓||✓||✓|
|Purpose||Submit and query jobs. SSH to development nodes. File management.||Fast in- & outbound file transfers. File management.||Compile and install software. Prototype and test job scripts. Submit and query jobs. Version control (clone, pull, push). File management.||Running short and long-running job scripts.|
(*) GIT+SSH access on development nodes is restricted to git.bioconductor.org, bitbucket.org, gitea.com, github.com / gist.github.com, gitlab.com, cci.lbl.gov, and git.ucsf.edu.
The cluster can be accessed via SSH to one of two login nodes:
For transferring large data files, it is recommended to use one of the dedicate data transfer nodes:
which both has a 10 Gbps connection - providing a file transfer speed of up to (theoretical) 1.25 GB/s = 4.5 TB/h. As for the login nodes, the transfer nodes can be accessed via SSH.
Comment: You can also transfer data via the login nodes, but since those only have 1 Gbps connections, you will see much lower transfer rates.
The cluster has development nodes for the purpose of validating scripts, prototyping pipelines, compiling software, and more. Development nodes can be accessed from the login nodes.
||72||384 GiB||0.93 TiB||Intel Gold 6240 2.60GHz|
||48||512 GiB||0.73 TiB||Intel Xeon E5-2680 v3 2.50GHz|
||48||256 GiB||0.73 TiB||Intel Xeon E5-2680 v3 2.50GHz|
||32||128 GiB||0.82 TiB||Intel Xeon E5-2640 v3 2.60GHz||NVIDIA Tesla K80|
Comment: Please use the GPU development node only if you need to build or prototype GPU software.
The majority of the compute nodes have Intel processors, while a few have AMD processes. Each compute node has a local
/scratch drive (see above for size), which is either a hard disk drive (HDD), a solid state drive (SSD), or even a Non-Volatile Memory Express (NVMe) drive. Each node has a tiny
/tmp drive (4-8 GiB).
The compute nodes can only be utilized by submitting jobs via the scheduler - it is not possible to explicitly log in to compute nodes.
The Wynton HPC cluster provides two types of scratch storage:
/scratch/- 0.1-1.8 TiB/node storage unique to each compute node (can only be accessed from the specific compute node).
/wynton/scratch/- 615 TiB storage (BeeGFS) accessible from everywhere.
There are no per-user quotas in these scratch spaces. Files not added or modified during the last two weeks will be automatically deleted on a nightly basis. Note, files with old timestamps that were “added” to the scratch place during this period will not be deleted, which covers the use case where files with old timestamps are extracted from a tar.gz file. (Details:
tmpwatch --ctime --dirmtime --all --force is used for the cleanup.)
/wynton/home: 383 TiB storage space
/wynton/group: 5600 TB (= 5.6 PB) storage space
Each user may use up to 500 GiB disk space in the home directory. It is not possible to expand user’s home directory. Research groups can add additional storage space under
/wynton/group by either mounting their existing storage or purchase new.
While waiting to receive purchased storage, users may use the global scratch space, which is “unlimited” in size with the important limitation that files older than two weeks will be deleted automatically.
Importantly, note that the Wynton HPC storage is not backed up. Users and labs are responsible to back up their own data outside of Wynton HPC.
The majority of the compute nodes are connected to the local network with 1 Gbps and 10 Gbps network cards while a few got 40 Gbps cards.
The cluster itself connects to NSF’s Pacific Research Platform at a speed of 100 Gbps - providing a file transfer speed of up to (theoretical) 12.5 GB/s = 45 TB/h.