Transferring Data to OSN Storage Bucket

In order to transfer data from Andromeda or Unity to the OSN pod, you will have to use the command-line tool Rclone. Rclone is available on Unity whenever you access the cluster, while you have to load a module on Andromeda by using module load rclone/1.66.0.

Config

For both clusters, you have to edit the Rclone configuration file that is located here: ~/.config/rclone/rclone.conf. You can create it if you don’t have one already. In the Rclone configuration file, you will add the following information about the OSN bucket that was allocated to your lab.

~/.config/rclone/rclone.conf
nano ~/.config/rclone/rclone.conf

Enter this text, replace the OSN RW Bucket Access Key and OSN RW Bucket Secret Key with the keys from Hollie:

[osn-pod]
type = s3
provider = Ceph
access_key_id = `OSN RW Bucket Access Key`
secret_access_key = `OSN RW Bucket Secret Key`
endpoint = https://uri.osn.mghpcc.org
no_check_bucket = true 

Starting transfers

tmux new -s rclone-osn

### first one, need to make the raw_sequencing_data folder on osn:

cd /project/pi_hputnam_uri_edu/
rclone copy -P raw_sequencing_data/Past_genome osn-pod:hollie-putnam-lab-data/raw_sequencing_data/Past_genome

### check what's there:

rclone ls osn-pod:hollie-putnam-lab-data 

### copy rest of folders:

cd /project/pi_hputnam_uri_edu/raw_sequencing_data/
rclone copy -P <FolderToTransfer> osn-pod:hollie-putnam-lab-data/<FolderToTransfer>

Text from email:

Once you have edited/created the Rclone configuration file you can start doing transfers. The recommended way to do that is to start a screen session and then use the rclone copy command. There is an rclone sync command that synchronizes the content between a source and destination but it is not recommended because it will delete files in the destination folder when you delete them in your source folder.

Here is an example of command to copy a file from Unity/Andromeda to your OSN bucket called hollie-putnam-lab-data:

rclone copy -P <FileToTranfer> osn-pod:hollie-putnam-lab-data

The -P flag will show how the transfer progresses.

You can check if the transfer was successful with the following command:

rclone ls osn-pod:hollie-putnam-lab-data | grep <FileToTranfer>

If you want to copy the content of a directory you will have to specify the name of the directory in the destination path because Rclone copies the content but not the directory itself. Here is an example:

rclone copy -P <FolderToTransfer> osn-pod:hollie-putnam-lab-data/<FolderToTransfer>