Research groups can purchase additional storage on Wynton beyond the user-specific storage that is free to all.
All user accounts come with a quota of non-expandable 500 GiB of storage in /wynton/home or /wynton/protected/home, which is free of charge
Global /wynton/scratch and /wynton/protected/scratch (for PHI users) may be used for smaller, short-term project needs of a few TBs with the caveat that files older than two weeks are deleted automatically
Additional, long-term storage can be purchased in /wynton/group, /wynton/protected/group (for PHI users), or in /wynton/protected/project (for specific PHI projects) at $160/TB (one-time fee). Both will count towards a lab’s storage quota total.
After 5 years from purchase, when the warranty runs out, a small “maintenance fee” might be introduced
To purchase more storage, please contact storage@wynton.ucsf.edu
We are currently offering the ability for research labs to purchase additional storage at a one-time charge of $160/TB for RAID 6 equivalent storage (or $320/TB for mirrored RAID 6). In contrast to /wynton/home, purchased storage will be mounted on /wynton/group, /wynton/protected/group, or /wynton/protected/project and dedicated to the lab that bought it – no oversubscription will occur. The purchased storage will have similar performance to the rest of the BeeGFS infrastructure (e.g. /wynton/scratch, /wynton/protected/scratch, /wynton/home, and /wynton/protected/home).
Please note, storage in /wynton/group/, /wynton/protected/group, and /wynton/protected/project count towards a lab’s storage quota total.
Given prices of hard drives, the stated rate might seem high, but there are three mitigating factors. First, we have enabled ZFS compression, so the actual available space might be significantly more. Second, the price includes the cost of the networking, metadata servers, storage server, maintenance, and administration. Third, we have proven that the performance of our BeeGFS infrastructure is much higher than the typical NFS server (in some respects, the performance is more than an order of magnitude faster). In the future, if absolutely necessary, we may also charge a “maintenance fee” for storage after the initial 5-year hardware warranty expires, but nothing has been decided as of yet. Similarly, any future storage purchases may be priced differently than that described here, to reflect the situation present at that time.
There are some additional parameters:
Unlike nodes, the storage cannot be extracted from the system once it’s been brought online. Once a lab or a group has “bought in”, they will not be able to retrieve their portion of storage hardware if they choose to leave Wynton (unlike compute nodes/shares).
The storage is not available for mounting outside of the cluster.
The storage is not backed up.
/wynton/protected/group? #
A: Yes, but PHI data must not be stored in /wynton/group. We can create group folders for you under /wynton/protected for PHI data storage and collaboration. If you choose to have your current storage allocation updated to include PHI storage, you have three options:
We can create a group folder in both /wynton/group and /wynton/protected/group. Both folders will share the same quota. In this case, the same quota will be applied to two folders, so it can be tricky to track where the group’s quota usage is, PHI or non-PHI.
We can create a group folder in /wynton/group and a folder with a separate sub-quota subtracted from the group quota in /wynton/protected/group, for example, /wynton/protected/group/group-phi. This makes it easier to track separate PHI and non-PHI group quota usage.
If the group using the PHI data does not include the whole lab group membership, or includes members from outside of the group, then a folder in /wynton/protected/project can be created and the child quota subtracted from the parent group.
Only members of the lab group with Wynton PHI access will be able to access files stored under /wynton/protected.
/wynton/home or /wynton/protected/home? #
A: No, the home directory quotas are managed separately from group quotas and additional purchased group space cannot be applied to home directories.
We have an immediate need to provide for Wynton HPC storage expansion to meet the demands of various research groups on campus. Given the difficulty of predicting longer-term costs and issues, the current pricing is considered short-term and may change as we understand the evolving needs and operational realities of Wynton HPC. Thus, this pricing model will be re-evaluated at the beginning of the next fiscal year (July 1, 2019). This model is based on some assumptions that are important to understand:
Various components of the shared storage environment are considered “infrastructure” and are currently funded from the ongoing support provided by the campus. These components include the networking infrastructure, and management and metadata servers that are part of the overall storage infrastructure. We don’t know with certainty that these components will continue to be funded by the campus and this introduces additional uncertainty as to the future pricing for storage, beyond the current offering described here.
The $160/TB price is for non-mirrored, potentially non-geographically redundant storage. While we hope to always purchase storage servers in pairs, providing failover between servers, there is no guarantee that we will always be able to do that. If you wish to protect your data beyond the level of RAID 6 (allows for two failed disks), we suggest you consider purchasing mirrored storage which completely duplicates data using a separate set of (RAID 6) disk drives.
Any new storage will be added to our existing BeeGFS installation and will not use separate instances of BeeGFS (which would significantly increase the potential costs, if not in hardware, certainly in terms of personnel effort).
We assume that our last purchase represents a reasonable scalable unit of storage. This purchase provided 1,200 TB (raw) storage and two storage servers. At ~$112,500, this results in a cost of $93/TB (raw) or $136/TB after accounting for RAID-Z2 and BeeGFS filesystem costs. We also need to add two additional metadata servers at a cost of $20,000. Taken together, this results in the $160/TB price.
The current storage hardware increment is ~$160,000, which may result in some delay between the first contributions and an actual purchase, although there is already pent-up demand and hence we are trying to proceed with the purchase as quickly as possible.
Last updated: May 22, 2023