Using GridFTP

Setting Up Your Environment

Mira, Cetus, Vesta, and Cooley

Mira, Cetus, Vesta, and Cooley use softenv for managing your software environment. You may add the GridFTP client utilities to your environment by inserting the keyword +globus in the .soft file in your home directory (.soft.cooley on Cooley). This must be on a single line by itself, and can be placed anywhere in the file except the last line; the last line of the file must always be the keyword @default.

For example, a .soft file consisting of the GridFTP tools and the default environment would look like this:

+globus
@default

Theta

On Theta, modules is used to manage your software environment. You may add the GridFTP client utilities to your environment by running the following command:

module load globus

Your environment is immediately updated after issuing the above command, but it will only persist for your login session. To permanently add the Globus Toolkit to your environment, you can add the above command to your .bashrc (if your login shell is bash), or .cshrc (if your login shell is csh or tcsh).

Prerequisites for Your Home Site

Trusting the ALCF CA

In order to use your ALCF MyProxy credentials to authenticate to both endpoints involved in your data transfer (i.e., ALCF and your home site), your system administrator may need to add the ALCF Certificate Authority (CA) files to your local GridFTP server's configuration, if this has not already been done.

Download the following tar.gz file containing ALCF CA files: 664c643b.tgz

  • Contents of archive:
    • ALCF CA Cert: 664c643b.0 (MD5 e790018fdae7419c732560a73a399321)
    • ALCF CA Signing Policy: 664c643b.signing_policy (MD5 ac0ed74fcbb8a6610b3c3ec057002057)
  • Your system administrator will need to install the above files on their GridFTP servers in /etc/grid-security/certificates

Information about our CA

ALCF has a local CA for the purpose of generating short-term certificates for GridFTP transactions. In order to perform GridFTP transfers, users must authenticate to the MyProxy server using the one-time password provided by their CryptoCard. The Registration Authority (RA) function for our CA is the identity verification we perform when issuing the CryptoCard to the user. Certificates are only issued to ALCF users with CryptoCards.

Performing GridFTP Transfers

  • If you are using our MyProxy server, first log in to the MyProxy server:
myproxy-logon -s myproxy.alcf.anl.gov
  • When prompted for a password, type in your CryptoCard PIN and generated password.

  • Note: proxy certificates are temporary; the default is 12 hours for the command above. You may request a shorter or longer certificate lifetime using the -t option followed by the number of hours desired, up to a maximum of 176 hours. Certificates may be renewed to avoid interruption during longer transfer jobs by running myproxy-logon again, before the certificate expires, and re-authenticating.

  • Perform your transfer using the globus-url-copy command:

globus-url-copy <arguments> gsiftp://miradtn.alcf.anl.gov/source_path_on_mira gsiftp://dest...
  • For example:
globus-url-copy -vb -p 4 -tcp-bs 4M gsiftp://miradtn.alcf.anl.gov/home/acherry/foo.tar gsiftp://gridftp.foobar.edu/home/ajc

This will issue a copy with verbose output (-vb), four data streams (-p 4), and a 4MB block size (-tcp-bs 4M). For very large files (over 100GB), you may add "-striped" to the globus-url-copy command to utilize multiple backend GridFTP servers and increase transfer bandwidth.  Striping is only available for Mira.  However, for files under 10GB, striping provides little benefit. For files between 10GB and 100GB, striping may provide some benefit, depending on the network. You may wish to experiment with transfer settings to determine the optimal settings for your particular transfer situation.

If you are running a transfer between GPFS filesystems internally at ALCF (e.g., between the /home and /projects filesystems on Mira), the "-p 4" may be omitted, since multiple streams offer little benefit on internal transfers.

Data transfers to/from ALCF should use one of the following URL prefixes:

For additional documentation on the usage of MyProxy, please refer to the MyProxy user documentation at http://grid.ncsa.uiuc.edu/myproxy/userguide.html

For additional documentation on GridFTP, please refer to the GridFTP user guide at http://www.globus.org/toolkit/docs/latest-stable/gridftp/user/#gridftpUser

Note: Use of file:// URLs (or paths without a transport specified) on the ALCF logins is highly discouraged, because it imposes excessive load that may affect other users. Please use gsiftp:// for the ALCF side of your file transfer when running globus-url-copy on the login nodes.