Although there is no hard quota in place on UCSF Box, the storage is limited and not really meant to store large data sets. There are reports that the UCSF Box maintainers have started to reach out to users who misuse it to store huge amounts of data. Please use it responsibly.
Below instructions work on data transfer nodes only. Attempts to use them on development nodes will result in “Access failed: 401 Unauthorized” errors.
It is possible to access UCSF Box using FTP over a secure SSL connection (FTPS). The below instructions works from the Wynton HPC data-transfer nodes as well as your local computer.
It is also possible for UCSF Users to access UCSF Box via the UCSF Wynton HPC Box Connector. For details on how to utilize the UCSF Wynton HPC Box Connector see the Globus File Transfers page on our website.
In order to do access UCSF Box as describe below, you need to:
With a UCSF Box-specific password (see above), you can use, for instance, the lftp or curl tools to access to your UCSF Box account. Start by logging in to one of the data-transfer nodes, either directly from outside or via a login node, e.g.
[alice@log2 ~]$ ssh dt2
alice1@dt2:s password: XXXXXXXXXXXXXXXXXXX
[alice@dt2 ~]$
Then, verify that your UCSF Box setup is correct by logging into the root of your UCSF Box folder using your UCSF Box-specific password (not your Wynton HPC password):
[alice@dt2 ~]$ lftp --user alice.aliceson@ucsf.edu ftps://ftp.box.com
Password: XXXXXXXX <== UCSF Box password here!
lftp alice.aliceson@ucsf.edu@ftp.box.com:~> ls
drwx------ 1 owner group 0 Jun 12 2014 Grant_R01.pdf
drwx------ 1 owner group 0 Sep 30 2016 Secure-alice.aliceson@ucsf.edu
lftp alice.aliceson@ucsf.edu@ftp.box.com:~> exit
[alice@dt2 ~]$
🛑 Never specify your password via a command-line argument! If you do, it will be visible to all other users via commands such as ps and htop.
When starting lftp as above, you need to manually enter your password, which can be tedious or even prevent automatic file transfers in batch scripts. A solution to this is to set up the FTPS credentials in ~/.netrc. Here is what it could look like:
[alice@dt2 ~]$ cat ~/.netrc
machine ftp.box.com
login alice.aliceson@ucsf.edu
password AliceSecretPwd2017
🛑 The ~/.netrc file must be kept private, otherwise its content could be readable to other users.
Since the password is fully visible in plain text, make sure to keep this file private at all times, otherwise users on the system can see all your credentials, i.e.
[alice@dt2 ~]$ chmod 600 ~/.netrc
[alice@dt2 ~]$ ls -l ~/.netrc
-rw------- 1 alice alice 72 Jul 3 15:10 /wynton/home/boblab/alice/.netrc
To verify that the automatic authentication works, try to log in again. You should no longer be prompted for your password - instead lftp gets it automatically from ~/.netrc. For example:
[alice@dt2 ~]$ lftp --user alice.aliceson@ucsf.edu ftps://ftp.box.com
lftp alice.aliceson@ucsf.edu@ftp.box.com:~> ls
drwx------ 1 owner group 0 Jun 12 2014 Grant_R01.pdf
drwx------ 1 owner group 0 Sep 30 2016 Secure-alice.aliceson@ucsf.edu
lftp alice.aliceson@ucsf.edu@ftp.box.com:~> exit
$
Note that curl also recognizes ~/.netrc credentials, e.g.
[alice@dt2 ~]$ curl --netrc -O ftps://ftp.box.com/Grant_R01.pdf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 15.6M 100 15.6M 0 0 1561k 0 0:00:10 0:00:10 --:--:-- 3918k
[alice@dt2 ~]$ ls -la Grant_R01.pdf
-rw-r--r-- 1 alice cluster 16453180 Jul 10 21:13 Grant_R01.pdf
To upload a file, we can do:
[alice@dt2 ~]$ curl --netrc --upload-file notes.txt ftps://ftp.box.com/