Moving Data With Rclone

Moving Data With Rclone

storageN

  • The use of storageN within these documents indicates that any storage platform can be used.

  • Current available storage platforms:

    • storage1

    • storage2

    • storage3

What is Rclone?

From https://hub.docker.com/r/rclone/rclone:

Rclone (“rsync for cloud storage”) is a command line program

to sync files and directories to and from different cloud storage providers.

Overview

You will install rclone on your local computer. Through the command rclone config, you will create a credential file for rclone to connect to your WashU Box on your local computer. By copying the file to your home directory on RIS compute1 client, you will be able to access your Box storage through your rclone container on a compute1 exec node.

Prerequisites

  1. A WashU Box account

  2. A user account for RIS storageN and compute1 services

Building an Endpoint

Installation

  • For macOS users, run the following command to install rclone with Homebrew.

    > brew install rclone
  • For Windows users, download the relevant archive file from Rclone downloads for your environment. Then, extract the rclone.exe binary from the archive.

  • For Linux/BSD users, run the following command to install rclone.

    > curl https://rclone.org/install.sh | sudo bash

Configuration

Creating the configuration file for the connection to WUSTL Box

  • Open a terminal where rclone has been installed

  • Run rclone config to start the interactive process.

> rclone config No remotes found - make a new one n) New remote s) Set configuration password q) Quit config n/s/q>
  • Type n to setup a new remote connection. It will ask for the name for your new remote connection.

n/s/q> n name>
  • Type Box for example, as the name of your new remote connection. It will ask for the storage type.

name> Box Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value
  • Type box for the storage type.

Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / 1Fichier \ "fichier" 2 / Alias for an existing remote \ "alias" 3 / Amazon Drive \ "amazon cloud drive" 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, Tencent COS, etc) \ "s3" 5 / Backblaze B2 \ "b2" 6 / Box \ "box" 7 / Cache a remote \ "cache" 8 / Citrix Sharefile \ "sharefile" 9 / Dropbox \ "dropbox" 10 / Encrypt/Decrypt a remote \ "crypt" 11 / FTP Connection \ "ftp" 12 / Google Cloud Storage (this is not Google Drive) \ "google cloud storage" 13 / Google Drive \ "drive" 14 / Google Photos \ "google photos" 15 / Hubic \ "hubic" 16 / In memory object storage system. \ "memory" 17 / Jottacloud \ "jottacloud" 18 / Koofr \ "koofr" 19 / Local Disk \ "local" 20 / Mail.ru Cloud \ "mailru" 21 / Mega \ "mega" 22 / Microsoft Azure Blob Storage \ "azureblob" 23 / Microsoft OneDrive \ "onedrive" 24 / OpenDrive \ "opendrive" 25 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH) \ "swift" 26 / Pcloud \ "pcloud" 27 / Put.io \ "putio" 28 / QingCloud Object Storage \ "qingstor" 29 / SSH/SFTP Connection \ "sftp" 30 / Sugarsync \ "sugarsync" 31 / Tardigrade Decentralized Cloud Storage \ "tardigrade" 32 / Transparently chunk/split large files \ "chunker" 33 / Union merges the contents of several upstream fs \ "union" 34 / Webdav \ "webdav" 35 / Yandex Disk \ "yandex" 36 / http Connection \ "http" 37 / premiumize.me \ "premiumizeme" 38 / seafile \ "seafile" Storage> box ** See help for box backend at: https://rclone.org/box/ **
  • Leave blank for the following questions about: client_id, client_secret, box_config_file, access_token.

OAuth Client Id Leave blank normally. Enter a string value. Press Enter for the default (""). client_id> OAuth Client Secret Leave blank normally. Enter a string value. Press Enter for the default (""). client_secret> Box App config.json location Leave blank normally. Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`. Enter a string value. Press Enter for the default (""). box_config_file> Box App Primary Access Token Leave blank normally. Enter a string value. Press Enter for the default (""). access_token>
  • Type user for the option to delegate the connection role to rclone.

Enter a string value. Press Enter for the default ("user"). Choose a number from below, or type in your own value 1 / Rclone should act on behalf of a user \ "user" 2 / Rclone should act on behalf of a service account \ "enterprise" box_sub_type> user
  • Use the default values for the rest of the questions for: Edit advanced config? Use auto config? Then, It will provide you a link and wait for code.

Edit advanced config? (y/n) y) Yes n) No (default) y/n> Remote config Use auto config? * Say Y if not sure * Say N if you are working on a remote or headless machine y) Yes (default) n) No y/n> If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=##################### Log in and authorize rclone for access Waiting for code...
  • Open your browser to the link on your machine where rclone config has been running on.

  • Login to WashU Box with your credential. Approve the access on your Duo App.

  • Grant the access for rclone to connect to Box. Then, your will see the confirmation of the process. An email notification from box will be sent to you with the subject: Box login from “rclone”.

  • Close the browser. The configuration for rclone connection to Box will be displayed on your terminal. For example:

Got code -------------------- [Box] type = box box_sub_type = user token = {"access_token":"###########################","token_type":"bearer","refresh_token":"##############################################","expiry":"2020-12-11T12:45:22.744758-06:00"} -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d>
  • Type y if the configuration content looks OK. Then, you will see the new remote connection in the remotes list.

y/e/d> y Current remotes: Name Type ==== ==== Box box
  • Type q to finish the interactive process.

e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q> q