How to use Rclone with Vultr Object Storage

Introduction

Rclone is a command-line program to manage files on cloud storage, with cloud equivalents to unix commands like rsync, cp, mv, mount, ls, and more. This guide describes how to configure rclone for Vultr Object Storage.

Prerequisites

• Install rclone for your platform, either manually, with Chocolatey on Windows, or Homebrew on macOS and Linux.
• Create a Vultr Object Storage bucket. This guide uses the example testbucket.

The rclone commands are cross-platform. This guide shows examples for macOS in zsh, which are compatible with Linux in bash. Windows users will make minor command adjustments for PowerShell or CMD.

Configuration

Locate your rclone configuration file location. This example path below comes from a macOS installation.

$ rclone config file
Configuration file doesn't exist, but rclone will use this path:
/Users/example/.config/rclone/rclone.conf

Create a configuration file.

$ nano /Users/example/.config/rclone/rclone.conf

Paste the following to the file. Replace access_key_id and secret_access_key with your values.

[vultr-ewr1]
type = s3
provider = Vultr
env_auth = false
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_SECRET_ACCESS_KEY
region =
endpoint = https://ewr1.vultrobjects.com
location_constraint =
acl = private
server_side_encryption =
storage_class =

The acl = line sets the default file permission.

• private : Owner gets full control. No one else has access. This permission is the default.
• public-read : Owner gets full control. Everyone has read access.

In general, use private unless you share files via URL. Read more about s3 permissions in the rclone documentation.

Testing

Create a test folder and change to it.

$ mkdir tmp
$ cd tmp

Assuming the tmp folder is empty, create three test files.

$ echo "hello rclone" > test1.txt
$ echo "hello rclone" > test2.txt
$ echo "hello rclone" > test3.txt

Copy one file to the bucket.

$ rclone copy test1.txt vultr:testbucket

List the bucket. Notice that test1.txt exists in the bucket and is 13 bytes.

$ rclone ls vultr:testbucket
   13 test1.txt

Delete test1.txt from the local system.

$ rm test1.txt

List the bucket. Notice that test1.txt still exists.

$ rclone ls vultr:testbucket
   13 test1.txt

Sync the entire folder to the bucket.

$ rclone sync . vultr:testbucket

List the bucket. Notice that rclone synced the bucket with the local system by removing test1.txt and adding the other two files.

$ rclone ls vultr:testbucket
   13 test2.txt
   13 test3.txt

Sync Two Object Storage Instances

In addition to cloning local files to object Storage, rclone can work with two object storage buckets, or locations. If you have multiple instances with different keys, declare them in rclone.conf with different labels. They can also have different permissions.

Here's an example of two different object storage instances, having different keys and different permissions. The vultr-private configuration does not allow public access, while the vultr-public configuration uses a different set of keys in a different object storage instance, and allows public web access. The keys and acl= lines are different between the configurations.

[vultr-private]
type = s3
provider = Vultr
env_auth = false
access_key_id = PRIVATE_KEY
secret_access_key = SECRET_PRIVATE_KEY
region =
endpoint = https://ewr1.vultrobjects.com
location_constraint =
acl = private
server_side_encryption =
storage_class =

[vultr-public]
type = s3
provider = Vultr
env_auth = false
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_SECRET_ACCESS_KEY
region =
endpoint = https://ewr1.vultrobjects.com
location_constraint =
acl = public-read
server_side_encryption =
storage_class =

To sync the private instance to the public instance, you'd use rclone like this:

$ rclone sync vultr-private:my_bucket vultr-public:my_public_bucket

Troubleshooting

If you encounter HTTP error codes 429, 503, or 504, you may have hit the Vultr Object Storage rate limit. If you encounter this issue, we recommend the following command line options.

• --tpslimit 100
  • Limit transaction per second to 100.
• --transfers 1
  • This prevents parallel file transfers.

More Information

Please refer to the rclone documentation for more information.