Syncing files to google cloud storage

I'm currently working on a solution for a project where we will be handling mass uploads from our customer base. Thousands of image files for automotive parts.

I came across an app called rclone, that lets you upload files to all sorts of cloud providers. It's open source, and can be found on github here

It supports syncing to and from the following locations.
* Amazon Drive
* Amazon S3
* Backblaze B2
* Box
* Ceph
* Dreamhost
* Dropbox
* FTP
* Google Cloud Storage
* Google Drive
* HTTP
* Hubic
* Memset Memstore
* Microsoft Azure Blob Storage
* Microsoft OneDrive
* Minio
* OVH
* Openstack Swift
* Oracle Cloud Storage
* QingStor
* Rackspace Cloud Files
* SFTP
* Wasabi
* Yandex Disk
* The local filesystem

This tutorial will show how to configure rclone to upload and one way sync a folder of your local filesystem to a google cloud bucket of your choosing.

Head on over to IAM & Admin within cloud console to create a service account. The example files in this scenario are web accessible image files, reading and writing shares requires storage admin access for your new service account only.

download the key in json format, rename the file for easier use. I'll call it rclone.json.

Head over to rclone's site, and download the rclone zip file.

Once downloaded, unzip, rename and place the folder somewhere to live. It might also be a good idea to place the key in the root of this folder for easier setup.

Set up remote

rclone config

Enter N for new remote

Name the remote, I'll name mine cloud-public

Select 8 for Google Cloud Storage (this is not Google Drive)

Leave the Application Client Id, Application Client Secret, and Project Number all blank.

Next, we need to enter the path to our JSON key file we created under the IAM & Admin of your Google Cloud Platform project. In this example I placed the key, named rclone.json within the rclone folder, so just enter rclone.json (or whatever you called yours, use the full path if not in same folder.

Under object_acl in my case, public_read is recommended for serving the files. If you just hit enter here, it defaults to a more secure setup where nobody but the owner can read.

bucket_acl will set bucket permissions of files written, for my use I selected 4 or public_read, again hitting enter here will default to the more secure owner read/write only.

Next we pick Location, It's where your Google Cloud Storage files are stored in Googles networks. I select 11 or Iowa

Next we pick the storage class of the objects saves, head on over the the Google Cloud Calculator and you can see right away the storage prices vary, If you'r not serving large files, you will probably be okay with Durable Reduced Reliability or even Regional. In my case I chose Regional.

Once complete, you will see a screen like below:

Remote config  
--------------------
[cloud-public]
client_id =  
client_secret =  
project_number =  
service_account_file = rclone.json  
object_acl = publicRead  
bucket_acl = publicRead  
location = us-central1  
storage_class = REGIONAL  
--------------------
y) Yes this is OK  
e) Edit this remote  
d) Delete this remote  
y/e/d>  

If it all looks good, hit Y, and then Q to quit.

syncing folders is super easy, to sync the local foo folder

Windows

cd C:\rclonedir  
rclone -v sync C:\foo remote:<bucketname>/<folder/  

Linux

cd <rclonedir>  
./rclone -v sync /mnt/foo/ remote:<bucketname>/<folder>/

So as you see above, other than the executable file that you grab from the rclone website, it works exactly the same, no matter if the client machine is Windows, Linux, or Apple.

I had planned on writing a custom app to do this for me, but this seems to fit the bill nicely. Hopefully it helps someone else needing to move files to and from cloud storage providers. It even moves files between two different cloud providers! I cant wait to give it a try.

Other than that, I noticed you can make one config, password protect it, and then just change out the key file for the user.