Syncing files to google cloud storage
I'm currently working on a solution for a project where we will be handling mass uploads from our customer base. Thousands of image files for automotive parts.
I came across an app called rclone, that lets you upload files to all sorts of cloud providers. It's open source, and can be found on github here
It supports syncing to and from the following locations.
- Amazon Drive
- Amazon S3
- Backblaze B2
- Google Cloud Storage
- Google Drive
- Memset Memstore
- Microsoft Azure Blob Storage
- Microsoft OneDrive
- Openstack Swift
- Oracle Cloud Storage
- Rackspace Cloud Files
- Yandex Disk
- The local filesystem
This tutorial will show how to configure rclone to upload and one way sync a folder of your local filesystem to a google cloud bucket of your choosing.
Head on over to IAM & Admin within cloud console to create a service account. The example files in this scenario are web accessible image files, reading and writing shares requires storage admin access for your new service account only.
download the key in json format, rename the file for easier use. I'll call it rclone.json.
Head over to rclone's site, and download the rclone zip file.
Once downloaded, unzip, rename and place the folder somewhere to live. It might also be a good idea to place the key in the root of this folder for easier setup.
Set up remote
N for new remote
Name the remote, I'll name mine
Google Cloud Storage (this is not Google Drive)
Application Client Id,
Application Client Secret, and
Project Number all blank.
Next, we need to enter the path to our JSON key file we created under the IAM & Admin of your Google Cloud Platform project. In this example I placed the key, named
rclone.json within the rclone folder, so just enter
rclone.json (or whatever you called yours, use the full path if not in same folder.
object_acl in my case,
public_read is recommended for serving the files. If you just hit enter here, it defaults to a more secure setup where nobody but the owner can read.
bucket_acl will set bucket permissions of files written, for my use I selected
public_read, again hitting enter here will default to the more secure owner read/write only.
Next we pick Location, It's where your Google Cloud Storage files are stored in Googles networks. I select
Next we pick the storage class of the objects saves, head on over the the Google Cloud Calculator and you can see right away the storage prices vary, If you'r not serving large files, you will probably be okay with
Durable Reduced Reliability or even
Regional. In my case I chose Regional.
Once complete, you will see a screen like below:
Remote config -------------------- [cloud-public] client_id = client_secret = project_number = service_account_file = rclone.json object_acl = publicRead bucket_acl = publicRead location = us-central1 storage_class = REGIONAL -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d>
If it all looks good, hit
Y, and then
Q to quit.
syncing folders is super easy, to sync the local foo folder
cd C:\rclonedir rclone -v sync C:\foo remote:<bucketname>/<folder/
cd <rclonedir> ./rclone -v sync /mnt/foo/ remote:<bucketname>/<folder>/
So as you see above, other than the executable file that you grab from the rclone website, it works exactly the same, no matter if the client machine is Windows, Linux, or Apple.
I had planned on writing a custom app to do this for me, but this seems to fit the bill nicely. Hopefully it helps someone else needing to move files to and from cloud storage providers. It even moves files between two different cloud providers! I cant wait to give it a try.
Other than that, I noticed you can make one config, password protect it, and then just change out the key file for the user.