Cloud Backup - My Experience So Far

[jameson_uk]

THESE ARE MY NOTES ON BACKING UP DATA TO GOOGLE CLOUD.   THIS MAY COST YOU MONEY SO PLEASE ENSURE YOU READ THROUGH EVERYTHING AND ARE HAPPY DOING THIS AT YOUR RISK

 

I have a 500GB+ music collection that I have carefully tagged and organised and after loosing a hard disk recently I started looking at a decent backup solution.

My first through was a standard cloud storage option but the costs of getting 500GB storage get prohibitively expensive

• Google Drive - 2TB (£7.99/month or £79.99/year)
• Dropbox Plus (2TB) (£9.99/month or £95.88/year)
• Microsoft One Drive - 1TB but only as part of Micrsoft 365 (£5.99/month £59.99/year)

Of these only the One Drive offering looks remotely tempting as it includes Office.

 

Having used some of the cloud storage options for work I started looking for a more cost effective option.

I did not want or need fancy synchronisation, a GUI or anything like that just somewhere to backup files to.

 

Now the Cloud options for true backups started looking much much better.    The cloud solutions for backup storage are based on files being stored but rarely accessed (ideal as the whole point of this is I should only ever need to restore from the cloud if something catastrophic happens).

 

So Microsoft, Amazon and Google all have similar offerings and have a free tier but I have used Google Cloud at work so started there.

Google offer a free trial which lasts for 12 months and comes with $300 credit which gives you plenty of room to play with and you won't be charged anything until you upgrade to a full account.

 

So you can simply register for the free trial https://cloud.google.com/free signing in with a Google account.

I can't remember whether you are guided through creating a project or not.   If not you need to create a project which will hold everything together.

When you get to the console you can navigate to the storage menu where you will an option to create a bucket (somewhere to store files).

The name of this needs to be globally unique (an easy way to do this is included the project name in the bucket name).

 

When you create the bucket you should see a monthly cost estimate on the right hand side so you can see what difference each of the options make.

For the cheapest costs you want to chose

Choose where to store your data

• Location Type - Region
• Location - US-WEST1, US-CENTRAL1, or US-EAST1 regions (These have additional free tier operations but 

Choose a default storage class for your data

• Archive

Choose how to control access to objects

• Doesn't really make a difference but uniform is sufficient for our needs

Advanced Settings

• Leave as defaults

 

Now the storage costs for 500 GB work out at £0.49 per month which is obviously quite a lot cheaper than any of the options above.

It isn't quite that simple as you also get charged for network operations.  In our use case the main thing is going to be our initial storing of files so lets say we add 17k files so if we set 17k Class A Operations our monthly cost jumps to a massive £1.17.  This however is only a one off cost of adding the files and all of this will be more than covered by the free tier credits.

 

So I then started looking at getting this up and running.    I am sure there are better ways of doing this and this can certainly be streamlined but this is relatively simple and seems to work well.

 

Google provide a docker image with all their utilities included so lets pull the smallest, most basic image via the command line

docker pull google/cloud-sdk:alpine

We can then test this works and has connectivity

docker run --rm google/cloud-sdk:alpine gcloud version

as long as this output the version numbers we can then move to store our google login credentials in a container called google-config

docker run -ti --name gcloud-config google/cloud-sdk:alpine gcloud auth login

this will output a URL you need to copy and paste into a browser.   You will need to log into your Google account and this will output a code that you need to copy and paste back into the command that should still be waiting for you.

 

Now I want to backup my local share (/mnt/user/music) to my bucket

docker run --rm -ti --volumes-from gcloud-config -v "/mnt/user/music:/music google/cloud-sdk:alpine gsutil -m rsync -r /music gs://<BUCKET_NAME>

which does a few things

• --rm - deletes the docker container after we use it
• --ti - creates an interactive shell
• --volumes-from gcloud-config - mounts the data from the google-config container we created above and contains our Google credentials
• -v "/mnt/user/music":/music - This makes my share (/mnt/user/music) appear in the container under /music
• google/cloud-sdk:alpine gsutil -m rsync -r /music gs://<BUCKET_NAME>
  • runs a Google rsync operation syncing /music to your bucket
  • -m makes it run in parallel
  • -r makes it process sub-folders

 

I have left this running for about two days now and it is nearly finished copying 480 GB.   My Class A Operation costs are running at £0.32 and I still have £242.42 credit remaining on my trial.   I will get charged for storage later (looking at £0.49 each month) but again this won't touch the free credit.

 

Once I have this all backed up I will see what the cost of running an rsync (with no changes) comes out as.   I am not sure which operations rsync calls under the covers but I am hoping it should be pretty low.

 

There is a warning here.    The whole point of using archive class storage is that you don't access it.    You will get charged a fair amount to download everything (I think it was in the region of £20 to download 500GB) which you would only need to spend in a disaster recovery situation.  There is also a minimum storage term of 365 days so if you delete files you will be charged as if they had been there for 365 days.

 

So just to reitterate where I started

THESE ARE MY NOTES ON BACKING UP DATA TO GOOGLE CLOUD.   THIS MAY COST YOU MONEY SO PLEASE ENSURE YOU READ THROUGH EVERYTHING AND ARE HAPPY DOING THIS AT YOUR RISK

[Harro]

Wonder if you had considered Backblaze. They seem to be a cheap solution.

[jameson_uk]

3 minutes ago, Harro said:

Wonder if you had considered Backblaze. They seem to be a cheap solution.

I hadn't but that is still $6 a month.   I think using archive storage on Google Cloud should come out about $1.10 a month (probably slightly less) and is effectively free for a year

[Harro]

So after a year, how much data do you figure will be saved in the Google archive? And will the price then be tiered or flat fee type?

[jameson_uk]

12 hours ago, Harro said:

So after a year, how much data do you figure will be saved in the Google archive? And will the price then be tiered or flat fee type?

My collection isn't growing that much so it will probably still be around 500 GB.

The free storage is 5GB but I am not sure whether this covers archive class buckets or not.   Charges are then per GB.

Still at $1.10 a month it is quite cheap. For 2TB the cost is a out $2.45 for storage plus the operations. I need to check how many operations rsync invokes but I am hoping that after the original upload (which would be covered by the trial) it should just be new and updated files and the cost would then be minimal.   What would hit you is downloading 2TB from archive class storage is going to cost $102.40.    The use case of this storage is that you don't access it regularly but even at this rate over 12 months you would be looking at $131.80 if you did download everything (12x $2.45 + $102.40).   I can't see Google Drive pricing in US but I am guessing 2TB is $9.99?   If so that would be £119.88 over the 12 months so would be cheaper but if you look over 24 months and downloaded once you would be looking at $161.20 vs $239.76 so the longer you have the archive without using it the cheaper it is.

Not sure if you could get away with signing up for a new trial after 12 months and doing an other upload or not (I think they get payment details off you when you sign up to prevent this but I guess as long as you enter a different card it might work). For the cost and hassle of uploading everything again I am not sure it is with it though.

Edited July 21, 2020 by jameson_uk
Updated for clarity

[Energen]

I think Harro was thinking about B2 Backblaze, which appears to be a different product/package and cheaper.  https://www.backblaze.com/b2/cloud-storage-pricing.html

 

Either way this thread is not about which cloud provider to use, so in that regard excellent write up on implementing Google Cloud.

 

As for pricing on Google, it appears that 2TB storage would be around $2.50/mo USD, and retrieval of same would be $103 USD.

 

If that's accurate than it would appear that even the B2 Backblaze is comparing to a different Google service, as storing 2TB seems to cost $10USD/month.

 

The only concern I would have with using Google is 1) privacy  -- what sort of encryption does it use on your files and are you really sure that Google/anyone can't access them.  And 2) Longevity.  Google is famous for shutting down services whenever people actually get used to using them.

 

jameson_uk I would be interested whether or not you could add any extra layers of encryption to your storage?

[Harro]

I was thinking about the OP and if that music data increased over time the cost per month increases. Whether that be icloud, google , backblaze, etc. So when does the cost of cloud storage become detrimental compared to say setting a 2TB drive up at another family members home and using sync?  

 

If this was done just a cost of hard drive and setup would be incurred, plus you have the advantage of increasing storage at your will.

Just my thoughts since a 500GB music storage seems it could be accomplished without a cloud base, but to each their own.

Edited July 21, 2020 by Harro

[Energen]

3 hours ago, Harro said:

So when does the cost of cloud storage become detrimental compared to say setting a 2TB drive up at another family members home and using sync?

You are making multiple assumptions that don't really matter.  If someone wants cloud backup then that's the only conversation.

 

1) What if you have no family members?

2) What if you have family without internet access?

3) How do you ensure that the backup drive will always be plugged in, turned on, and accessible?

4) How do you ensure network connectivity, router ports, firewall access, etc so that #3 applies?

 

The list goes on.  Sure it's an offsite backup if you can make everything work but that's not what is being talked about.  If you want cloud storage (which has access, redundancy, security, etc) you expect that there will be a recurring cost.. it doesn't matter if there are cheaper ways to do it.

[Harro]

This will be my last reply. 

I can see you would like to troll me for an argument, when I was only voicing other options that could be taken into account. I wasn't trying to convince OP to change his strategy on backup. As I had said to each their own.

[jameson_uk]

6 hours ago, Energen said:

As for pricing on Google, it appears that 2TB storage would be around $2.50/mo USD, and retrieval of same would be $103 USD.

 

If that's accurate than it would appear that even the B2 Backblaze is comparing to a different Google service, as storing 2TB seems to cost $10USD/month.

This is very specific Google Cloud Storage (Archive).   This is there for data that will be there for at least a year and will likely never be accessed.   Standard storage for 2TB would be $40.96 a month so you would be far better off with any of the standard cloud storage offerings.

Quote

 

The only concern I would have with using Google is 1) privacy  -- what sort of encryption does it use on your files and are you really sure that Google/anyone can't access them.  And 2) Longevity.  Google is famous for shutting down services whenever people actually get used to using them.

 

jameson_uk I would be interested whether or not you could add any extra layers of encryption to your storage?

 

This is enterprise grade stuff used by lots of big companies.   It is encrypted and locked down by default (in fact I don't believe there is a way to turn encryption off).   My understanding is that you can use a customer generated key for encryption which would not be accessible to Google and only you would have the key for decryption.

 

With Gmail and other customer offerings, they are free.   Google makes money by using your data.   With Google Cloud you are paying for the service and it is aimed at companies not individuals so there are no privacy concerns.

 

This storage is the basis of most cloud offerings and there is zero chance it would be shutdown.   They could change the conditions on arhcive storage but they are competing with Amazon and Microsoft who both have similar options and similar costs so it is unlikely.

[jameson_uk]

4 hours ago, Harro said:

I was thinking about the OP and if that music data increased over time the cost per month increases. Whether that be icloud, google , backblaze, etc. So when does the cost of cloud storage become detrimental compared to say setting a 2TB drive up at another family members home and using sync?  

 

If this was done just a cost of hard drive and setup would be incurred, plus you have the advantage of increasing storage at your will.

Just my thoughts since a 500GB music storage seems it could be accomplished without a cloud base, but to each their own.

Indeed this is a viable option but it isn't just the cost of a hard drive.   You need to have a box to run it on, electricity to keep the box on when you want to rsync and bandwith contention.   Add on top of that the complexity of setting up firewall rules (or the potential to get things wrong and open your servers up to the world) and in general maintenance of a sever environment it isn't plug and play.

 

This could be simplified by having a couple of USB hard drives and rsyncing to them.   Send one off to a family member, next week do the same with the second drive and the family member could return the first drive.

 

If I had another geek in my family I would probably go down this route but I wouldn't really want to be trying to remotely support my dad with a server issue.

 

With a cloud provider you are removing any infrastructure and support issues.

• They provide massive scale redundancy.   If the remote hard drive failed you would loose your backup.   With a cloud provider this is effectively not possible.
• They have massive bandwidth available and tools that allow you parallelise operations much easier.
• They provide massive scalability.   You could store PB of data if you wanted (it would cost a lot) without doing anything differently.
• They have security built in.   Your data is in secure data centres. Access is tightly controlled. You can grant / revoke access easily
• You don't need to worry about upgrades or vulnerabilities.   You are dealing with the cloud at a level of abstraction which means you don't need to worry about what version it is running on whether their servers have been patched for the latest zero day vulnerability.

Archive storage at $0.0012 per GB / month seems relatively cheap for this.

 

Looking at NewEgg I see over there a 2TB Ironwolf HDD costs $82.99.   Based on the following assumptions

1. You get 12 months free as part of your trial
2. You never need to restore from your backup
3. The HDD doesn't fail

You are looking at $2.46 per month. So It would take 45.74 months (82.99/2.46 + 12) for the cloud charges to be the same as the drive (assuming it is going into an existing server and somehow electricity is free).   

 

Obviously this is fairy tale marketing and the whole point is that you might need to download this all one day.   You also have to pay for network operations, there are other limitations (minimum storage periods etc.) that come into play in the real world.   Even so I think you are looking at nearly four years before the HDD has paid for itself and what is the lifespan of a HDD nowadays?

[jameson_uk]

So I have been running daily rsync and no costs have shown up at all (getting CRC and timestamps is obviously a free operation).

This means I have been charged 1 Class A Operation for each file I uploaded  (17,511 for which I would have been charged £0.71)

[jameson_uk]

So for the whole month of August I would have been billed £0.44 + TAX (~ £0.53 = $0.71) but this is obviously more than covered by the $250 trial credit.

 

Obviously the hit is if you ever want to download your files but as a true backup / DR solution this seems to be a simple and cheap option.

[sir_storealot]

Thank you for sharing your experience with this. This is definitely a very good and cheap way to get a reliable long-term backup.

[jameson_uk]

I do plan to do a test of pulling the ~480MB back just to see how much it would cost. I will post an update when I find time to do this

[Energen]

So one thing I have considered is that there is no file name encryption/obfuscation, correct?  Does not seem like that is a feature of any mainstream backup providers. 

[questionbot]

On 7/20/2020 at 11:33 PM, jameson_uk said:

<snip>

I did something similar before I got into unraid... this was a linux solution and basically it was using RClone... it is a cli app that can sync to pretty much anything. One of its great things is that it can sync to pretty much anything. So you can sync your collection to google drive but split it over 15GB free accounts. The end solution is seemless, it just looks like a normal directory with all your files.. but the syncing is split. You can even split across different services. Just something to consider if you are poor and want free storage.

[jameson_uk]

4 hours ago, questionbot said:

I did something similar before I got into unraid... this was a linux solution and basically it was using RClone... it is a cli app that can sync to pretty much anything. One of its great things is that it can sync to pretty much anything. So you can sync your collection to google drive but split it over 15GB free accounts. The end solution is seemless, it just looks like a normal directory with all your files.. but the syncing is split. You can even split across different services. Just something to consider if you are poor and want free storage.

Interesting but does this mean that to store 500 GB you would need to setup 34 different Google accounts?   Presumably this is manual

[questionbot]

uhhh yeah.. I mean a 500GB cloud is a different deal lol... I use it for a 100GB over 10 accounts and while it was a pain to setup, once it was up it was seamless... I was actually just posting this as a interesting side note. Also eveyone hates paying for internet stuff lol...

Edited September 18, 2020 by questionbot

[jameson_uk]

On 7/20/2020 at 2:33 PM, jameson_uk said:

There is a warning here.    The whole point of using archive class storage is that you don't access it.    You will get charged a fair amount to download everything (I think it was in the region of £20 to download 500GB) which you would only need to spend in a disaster recovery situation.  There is also a minimum storage term of 365 days so if you delete files you will be charged as if they had been there for 365 days.

 

On 9/15/2020 at 9:31 PM, jameson_uk said:

I do plan to do a test of pulling the ~480MB back just to see how much it would cost. I will post an update when I find time to do this

 

I finally got around to testing this and turns out I miscalculated.   I didn't read the docs properly and the downloading (egress) cost are on top of the standard egress costs not instead of.   So egress if $.012 per GB and egress from archive storage is is an additional $0.05 per GB.

 

So I would have actually been charged £61.69 (plus tax). (~$80 before tax).

Obviously this is more than covered by the trial fees but is obviously a lot more than I had first calculated.

 

I guess the question is how often do you need to invoke this kind of disaster recovery?

 

Storage is working out at £0.44 (+Tax) a month so over two years you are looking at £12.67 for storage including tax.  If you then assume you have to download once over that period that is an additional £61.69 (£74.03 inc tax) so the total over two years would be £86.70 (£3.61 a month).

 

If you work out for 2TB (2,048 GB) then you are looking at:

Storage 2,048 * $0.0012 = $2.46 per month ($58.98 over two years)

but you are looking at downloads of 2,048 * $0.17 = $348.16

 

So that comes to $407.14 which doesn't look that great against the likes of a standard Google One 2TB plan.   (The actual cost is slightly higher as you have to pay for network operations but that is a one of and is a few $)

Table code doesn't appear to be enabled on these forums so attaching this as an image

So based on 2TB storage (and using the entire 2TB) you if you restore once in 7 years you would break even, two restores would take 14 years.    Obviously if you never have to restore then you are making a significant saving from day 1.

 

I did come close to loosing my whole array once (the PSU blew and started to smoulder) but I have never had a need to restore in several years (but I know the pain it would cause if I ever had to).

 

As Ned Flanders says, insurance is gambling....    UnRaid already gives you some redundancy if a single drive fails but I guess I am looking at protecting the scenario of something like a fire taking out the whole server.   The main question then is how important is the data to you and what risk level you attach to loosing it.

 

I will experiment as to whether you can essentially jump trials (The simplest way to do this would just be to sign up for a new trail every year and start from scratch but I am not sure whether (a) there is any form of checking you are not creating multiple trials (You do have to enter a credit card number IIRC) and (b) whether you can essentially swap the costs of downloading or storage to another account without needing to upload again.