Transferring Data With Globus CLI

Installing Globus

You can install the Globus CLI by following the instructions found here.
Or you can use the RIS developed Dockerimage for Globus CLI. Globus CLI Dockerimage

Login Into Globus CLI

Once you have installed the Globus CLI or are using the Dockerimage, you will need to login. When using the dockerimage you will need to include the --no-local-server option and follow the directions it gives.

globus login

globus login --no-local-server

Globus CLI options

Below is a list of some of the options available when using the Globus CLI.

Globus `endpoint` Command

The endpoint command has many options, but the one that is most useful is the search option as you can use this option to search for the ID of an endpoint via name.

globus endpoint search "WashU"

An endpoint search using WashU as the search terms with bring up any endpoints that are related to that term. In this particular case, all the endpoints associated with WashU show up.

ID                                   | Owner              | Display Name
------------------------------------ | ------------------ | -------------------------------------
01f0ac4c-9570-11ea-b3c4-0ae144191ee3 | wustl@globusid.org | Wash U RIS storage1 dtn1
7e5cf228-8b33-11ea-bf85-0e6cccbb0103 | wustl@globusid.org | Wash U RIS storage1 dtn2
be749bfa-a36b-11e9-821c-02b7a92d8e58 | uwgs@globusid.org  | EEE WashU
8a8345c4-1593-11e8-b610-0ac6873fc732 | rcl2r@virginia.edu | MGI_WashU_vcf-files
93101176-23d6-11e8-b775-0ac6873fc732 | zhuda@msu.edu      | rrAD_from_WashU_New
1d354de0-5994-11ea-af4f-0201714f6eab | delapga@nih.gov    | WashU
f0d3e5ce-b634-11ea-9a3b-0255d23c44ef | tychele@wustl.edu  | WashU Turner Lab Incoming Globus Data
795c6697-6d04-11e5-ba46-22000b92c6ec | boliu@globusid.org | boliu#WashU-Galaxy
a912ce04-23eb-11e8-b776-0ac6873fc732 | u0129786@utah.edu  | 18-03-09_UW_Tristani_Data_Transfer
f8df3e58-f1e0-11e9-8a5c-0e35e66293c2 | winkleram@nih.gov  | outWashU

The ID is the information that will be utilized in the other options.

Globus `ls` Command

Once you have the ID of the endpoint you wish to use, you can use globus ls just like you use ls in linux/unix to display the files and directories in a directory.
The Wash U RIS storage1 dtn1/dtn2 endpoints are the storage1 endpoints and will display what is in your storage1 directory. The following command demonstrates how to do that.

globus ls 01f0ac4c-9570-11ea-b3c4-0ae144191ee3:/storage1/fs1/wustlkey/Active/

Globus `transfer` Command

Once you have determined the endpoints that you wish to use to transfer data, you can utilize the transfer command. You can transfer between the WashU endpoints to move data that way.
There are three ways you can transfer data.
- Transfer individual files.
- Transfer a directory, using the recursive -r option.
- Transfer files in bulk using the --batch option.
Transfering an invdividual file.

globus transfer 01f0ac4c-9570-11ea-b3c4-0ae144191ee3:/path/to/source/location/my_file.txt 7e5cf228-8b33-11ea-bf85-0e6cccbb0103:/path/to/destination/location/my_file.txt

Transfering a directory.

globus transfer -r 01f0ac4c-9570-11ea-b3c4-0ae144191ee3:/path/to/source/location/my_directory 7e5cf228-8b33-11ea-bf85-0e6cccbb0103:/path/to/destination/location/my_directory

Transfering using the --batch option
- You need to provide the directories that you are transferring between like with the transfer of a directory.
- You need a space separated text file with the location of the files in the source endpoint followed by the location in the destination endpoint. Each file must have it’s own line.
Sample1.txt Group1/Sample1.txt Sample2.txt Group1/Sample2.txt Sample3.txt Group1/Sample3.txt Sample4.txt Group2/Sample4.txt Sample5.txt Group2/Sample5.txt
- With the batch method, you can sort files from the source into separate directories in the destination or pull files from separate directories in the source into one directory in the destination.
Group1/Sample1.txt Sample1.txt Group1/Sample2.txt Sample2.txt Group2/Sample3.txt Sample3.txt Group2/Sample4.txt Sample4.txt Group2/Sample5.txt Sample5.txt
- You can provide the file of the files you wish to transfer to the command with the < option.
globus transfer --batch 01f0ac4c-9570-11ea-b3c4-0ae144191ee3:/path/to/source/location/my_directory 7e5cf228-8b33-11ea-bf85-0e6cccbb0103:/path/to/destination/location/my_directory < files_to_transfer.txt

Viewing Transfer and Task Information

Globus retains task information for 90 days. This is not controlled by RIS.

Commands in this section support the -F/--format json option for easier programmatic processing.

Transfer Status

Given a transfer task ID, information about the status of the task can be retrieved with task show:

[user@host cwd]$ globus task show 543e219c-01c1-11eb-81a3-0e2f230cc907
Label:                   None
Task ID:                 543e219c-01c1-11eb-81a3-0e2f230cc907
Is Paused:               False
Type:                    TRANSFER
Directories:             1
Files:                   3
Status:                  SUCCEEDED
Request Time:            2020-09-28T19:32:22+00:00
Faults:                  0
Total Subtasks:          6
Subtasks Succeeded:      6
Subtasks Pending:        0
Subtasks Retrying:       0
Subtasks Failed:         0
Subtasks Canceled:       0
Subtasks Expired:        0
Completion Time:         2020-09-28T19:32:27+00:00
Source Endpoint:         Globus Tutorial Endpoint 1
Source Endpoint ID:      ddb59aef-6d04-11e5-ba46-22000b92c6ec
Destination Endpoint:    Wash U RIS storage1 dtn2
Destination Endpoint ID: 7e5cf228-8b33-11ea-bf85-0e6cccbb0103
Bytes Transferred:       14
Bytes Per Second:        3

Transferred Filenames

The task info command can also produce a list of successfully transferred files when the -t/--successful-transfers option is used:

[user@host cwd]$ globus task show -t 543e219c-01c1-11eb-81a3-0e2f230cc907
Source Path         | Destination Path
------------------- | --------------------------------------------------------
/~/godata/file1.txt | /storage1/fs1/corcoran.william.p/Active/godata/file1.txt
/~/godata/file2.txt | /storage1/fs1/corcoran.william.p/Active/godata/file2.txt
/~/godata/file3.txt | /storage1/fs1/corcoran.william.p/Active/godata/file3.txt

This information is only available to the task owner (user), not RIS
This information is only available through the CLI and SDK, not the web UI

Event and Failure Logs

Timing and activity information (including errors) can be obtained with task event-list:

[user@host cwd]$ globus task event-list 543e219c-01c1-11eb-81a3-0e2f230cc907
Time                      | Code      | Is Error | Details
------------------------- | --------- | -------- | ---------------------------------------------------------------------------
2020-09-28T19:32:27+00:00 | SUCCEEDED |        0 | {"files_succeeded":3}
2020-09-28T19:32:27+00:00 | PROGRESS  |        0 | {"bytes_transferred":14,"duration":1.44,"mbps":0.0}
2020-09-28T19:32:25+00:00 | STARTED   |        0 | {"concurrency":2,"parallelism":4,"pipelining":20,"type":"GridFTP Transfer"}
2020-09-28T19:32:24+00:00 | SUCCEEDED |        0 | n/a
2020-09-28T19:32:24+00:00 | STARTED   |        0 | Starting sync delete

Further Globus Options

If you would like to learn more about the other options in the Globus CLI, you can find that information here.