RClone & WUSTL Box on Clusters

Owned by Mark Bober
Last updated: Jul 26, 2023
3 min read

Rclone can be used with the research clusters to access data in WUSTL Box.

  1. Start a Ondemand VNC session on either the ENGR or RIS clusters.

  2. In a web browser in that session:

    1. Download Rclone from here - https://rclone.org/downloads/ -you want the Linux Intel/AMD 64 bit ZIP file version.

  3. Open a terminal.

    1. cd ~/Downloads

    2. Unzip the rclone ZIP file you downloaded - the file name will vary as version numbers increase:

c. Make sure you have a ‘bin’ directory in your home dir, then copy the rclone binary to it. The error below means you already did have one, and does not mean there was an actual error:

Configure Rclone with WUSTL Box.

Overall instructions for using Rclone with box are here : https://rclone.org/box/ and we will walk through the steps below.

  1. Run rclone config. If you did not previously have a ~/bin directory as described above, you may need to execute it as ~/bin/rclone config. Enter in a friendly, short name for the service when prompted.



  2. Choose ‘box’ as the storage system. A long list of storage systems will print out; enter ‘box’ at the prompt:

     

  3. Leave the next prompts - client_id, client_secret, box_config_file, access_token, all blank.

  4. For box_sub_type, choose user. Do not edit the advanced configuration, then authorize using a web browser to complete the connection:

     

  5. A web browser will start, and open the Box login page. Choose SSO Sign in, then go through the regular prompts for entering your WUSTL Key password.

     

  6. After signing in, you will be prompted to authorize the connection with Rclone:

     

  7. Back in the terminal where you were running rclone, it will have received and configured the token. Type “y” to accept this remote config, then “q” to quit configuration.

     

Using Rclone on the Cluster

The most up to date documentation on usage will be at Rclone’s website; https://rclone.org.

Be sure to familiarize yourself with rclone commands in the Rclone documentation. You can copy to and from your ENGR home directory, RIS home directory, or RIS storage locations.

Some common commands include (all examples here assume you named the WUSTL Box remote “wustlbox”)

rclone ls --max-depth 1 wustlbox:/EIT/

…to get a listing of the files in the /EIT/ directory only, if one exists, non-recursively

rclone copy --max-depth=1 wustlbox:/EIT/ .

…copy, non recursively, files from the EIT folder to the current local folder

rclone sync -P wustlbox:/EIT/ .

…syncronize the contents of the local directory to the same contents as the WUSTL Box folder “EIT”. The source of the sync is listed first, the target second.

Always try this out with the --dry-run flag, as syncing the wrong direction can cause data loss!