Page Comparison

• Anchor
  id2 id2

  Installing Globus

• Anchor
  id3 id3

  Login Into Globus CLI

• Anchor
  id4 id4

  Globus CLI options

• Anchor
  id5 id5

  Globus endpoint Command

• Anchor
  id6 id6

  Globus ls Command

• Anchor
  id7 id7

  Globus transfer Command

• Anchor
  id8 id8

  Viewing Transfer and Task Information

  • Anchor
    id9 id9

    Transfer Status

  • Anchor
    id10 id10

    Transferred Filenames

  • Anchor
    id11 id11

    Event and Failure Logs

• Anchor
  id12 id12

  Further Globus Options

Installing Globus

• You can install the Globus CLI by following the instructions found here.

• Or you can use the RIS developed Dockerimage for Globus CLI. Globus CLI Dockerimage

Login Into Globus CLI

• Once you have installed the Globus CLI or are using the Dockerimage, you will need to login. When using the dockerimage you will need to include the --no-local-server option and follow the directions it gives.

globus login

globus login --no-local-server

Globus CLI options

• Below is a list of some of the options available when using the Globus CLI.

Globus endpoint Command

• The endpoint command has many options, but the one that is most useful is the search option as you can use this option to search for the ID of an endpoint via name.

globus endpoint search "WashU RIS"

• An endpoint search using WashU as the search terms with bring up any endpoints that are related to that term. In this particular case, all the endpoints associated with WashU show upMultiple endpoints will show up, but the one you want to use is RIS Storage1.

ID                                   | Owner                   | Display Name
------------------------------------ | ------------------ | ----- | --------------------------------
01f0ac4c-9570-11ea-b3c4-0ae144191ee3 | wustl@globusid.org | Wash U RIS storage1 dtn1
7e5cf228-8b33-11ea-bf85-0e6cccbb0103 | wustl@globusid.org | Wash U RIS storage1 dtn2
be749bfa-a36b-11e9-821c-02b7a92d8e58 | uwgs@globusid.org  | EEE WashU
8a8345c4-1593-11e8-b610-0ac6873fc732
7e92c283-7513-4d54-acdf-516f027bfeb2 | rcl2r@virginia.edu | MGI_WashU_vcf-files
93101176-23d6-11e8-b775-0ac6873fc732 | zhuda@msu.edu      | rrAD_from_WashU_New
1d354de0-5994-11ea-af4f-0201714f6eab | delapga@nih.gov    | WashU
f0d3e5ce-b634-11ea-9a3b-0255d23c44ef | tychele@wustl.edu  | WashU Turner Lab Incoming Globus Data
795c6697-6d04-11e5-ba46-22000b92c6ec | boliu@globusid.org | boliu#WashU-Galaxy
a912ce04-23eb-11e8-b776-0ac6873fc732 | u0129786@utah.edu  | 18-03-09_UW_Tristani_Data_Transfer
f8df3e58-f1e0-11e9-8a5c-0e35e66293c2 | winkleram@nih.gov  | outWashUwustl@globusid.org      | RIS Storage1

• The ID is the information that will be utilized in the other options. Please note that the correct RIS endpoint is the one named RIS Storage1. The others are being phased out.

Globus ls Command

• Once you have the ID of the endpoint you wish to use, you can use globus ls just like you use ls in linux/unix to display the files and directories in a directory.

• The Wash U RIS storage1 dtn1/dtn2 endpoints are the storage1 endpoints and will display what is in your storage1 directory. The following command demonstrates how to do that.

globus ls 01f0ac4c7e92c283-95707513-11ea4d54-b3c4acdf-0ae144191ee3516f027bfeb2:/storage1/fs1/wustlkey/Active/

Globus transfer Command

• Once you have determined the endpoints that you wish to use to transfer data, you can utilize the transfer command. You can transfer between the WashU endpoints to move data that way.

• There are three ways you can transfer data.

  • Transfer individual files.

  • Transfer a directory, using the recursive -r option.

  • Transfer files in bulk using the --batch option.

• Transfering an invdividual file.

globus transfer 01f0ac4c7e92c283-95707513-11ea4d54-b3c4acdf-0ae144191ee3516f027bfeb2:/path/to/source/location/my_file.txt 7e5cf2287e92c283-8b337513-11ea4d54-bf85acdf-0e6cccbb0103516f027bfeb2:/path/to/destination/location/my_file.txt

...

globus transfer -r 01f0ac4c7e92c283-95707513-11ea4d54-b3c4acdf-0ae144191ee3516f027bfeb2:/path/to/source/location/my_directory 7e5cf2287e92c283-8b337513-11ea4d54-bf85acdf-0e6cccbb0103516f027bfeb2:/path/to/destination/location/my_directory

• Transfering using the --batch option

  • You need to provide the directories that you are transferring between like with the transfer of a directory.

  • You need a space separated text file with the location of the files in the source endpoint followed by the location in the destination endpoint. Each file must have it’s own line.

  Code Block
  language python linenumbers false
  Sample1.txt Group1/Sample1.txt Sample2.txt Group1/Sample2.txt Sample3.txt Group1/Sample3.txt Sample4.txt Group2/Sample4.txt Sample5.txt Group2/Sample5.txt

  • With the batch method, you can sort files from the source into separate directories in the destination or pull files from separate directories in the source into one directory in the destination.

  Code Block
  language python linenumbers false
  Group1/Sample1.txt Sample1.txt Group1/Sample2.txt Sample2.txt Group2/Sample3.txt Sample3.txt Group2/Sample4.txt Sample4.txt Group2/Sample5.txt Sample5.txt

  • You can provide the file of the files you wish to transfer to the command with the < option.

  Code Block
  language python linenumbers false
  globus transfer --batch 01f0ac4c7e92c283-95707513-11ea4d54-b3c4acdf-0ae144191ee3516f027bfeb2:/path/to/source/location/my_directory 7e5cf2287e92c283-8b337513-11ea4d54-bf85acdf-0e6cccbb0103516f027bfeb2:/path/to/destination/location/my_directory < files_to_transfer.txt

Viewing Transfer and Task Information

Transfer Status

Given a transfer task ID, information about the status of the task can be retrieved with task show:

[user@host cwd]$ globus task show 543e219c-01c1-11eb-81a3-0e2f230cc907
Label:                   None
Task ID:                 543e219c-01c1-11eb-81a3-0e2f230cc907
Is Paused:               False
Type:                    TRANSFER
Directories:             1
Files:                   3
Status:                  SUCCEEDED
Request Time:            2020-09-28T19:32:22+00:00
Faults:                  0
Total Subtasks:          6
Subtasks Succeeded:      6
Subtasks Pending:        0
Subtasks Retrying:       0
Subtasks Failed:         0
Subtasks Canceled:       0
Subtasks Expired:        0
Completion Time:         2020-09-28T19:32:27+00:00
Source Endpoint:         Globus Tutorial Endpoint 1
Source Endpoint ID:      ddb59aef-6d04-11e5-ba46-22000b92c6ec
Destination Endpoint:    Wash U RIS storage1 dtn2Storage1
Destination Endpoint ID: 7e5cf2287e92c283-8b337513-11ea4d54-bf85acdf-0e6cccbb0103516f027bfeb2
Bytes Transferred:       14
Bytes Per Second:        3

Transferred Filenames

The task info command can also produce a list of successfully transferred files when the -t/--successful-transfers option is used:

...

Event and Failure Logs

Timing and activity information (including errors) can be obtained with task event-list:

[user@host cwd]$ globus task event-list 543e219c-01c1-11eb-81a3-0e2f230cc907
Time                      | Code      | Is Error | Details
------------------------- | --------- | -------- | ---------------------------------------------------------------------------
2020-09-28T19:32:27+00:00 | SUCCEEDED |        0 | {"files_succeeded":3}
2020-09-28T19:32:27+00:00 | PROGRESS  |        0 | {"bytes_transferred":14,"duration":1.44,"mbps":0.0}
2020-09-28T19:32:25+00:00 | STARTED   |        0 | {"concurrency":2,"parallelism":4,"pipelining":20,"type":"GridFTP Transfer"}
2020-09-28T19:32:24+00:00 | SUCCEEDED |        0 | n/a
2020-09-28T19:32:24+00:00 | STARTED   |        0 | Starting sync delete

Further Globus Options

• If you would like to learn more about the other options in the Globus CLI, you can find that information here.