I really need to get a grip on my backups and decide on a cloud provider as well, hoping you helpfull people can aid me with this.

 

My array is 104tb with a little over 80tb used, I am not looking to back all this up, the only thing I am worried about is family photos, RAW photos and footage, familys phone backups etc, this comes at the moment to a bit over 2tb. Whilst I would love to just backup all 80tb to another server I dont see the point in spending so much money on more drives, at least just now.

 

At the moment I have the data I truly care about on my array, I have resillio sync one way syncing the data to an external drive on another computer in the same home and I then have it also syncing to another unraid machine I built with a couple of drives in it to basically serve as a backup in another location.

 

So far how flawed is my logic? I feel like by using resillio sync I might be going down a slippery path but so far it has ran for a few months *knock on wood* without issue.

 

Is there anything else I could be doing to improve this locally? Is there another way I should sync the data at least locally in my home to the other external drive? Is there anything wrong with me syncing to the other machine in another location?

 

My next thought is that I am not truly following the 3-2-1 method, I also need a cloud backup, I have 1tb in Onedrive, and a family Google plan but I feel for this much data which will likely grow as time goes on I need something with more space, what would you guys recomend? I am based in the UK if that makes any difference.

 

2025 is the year I get my backups in order...hopefully  

 

It's great that you're prioritizing backups and are on the path to a solid strategy! Let's review your current setup and refine it based on the 3-2-1 backup principle:

 

Current Setup Analysis

The 3-2-1 Backup Rule:

3 copies of your data (production data + 2 backups).

2 different storage types (e.g., local drive and cloud).

1 copy offsite (cloud or remote server).

 

 

Your Current Strategy

Primary Data Location:

On your Unraid array (~2TB of critical data stored).

 

Local Backup (Onsite):

Data synced via Resilio Sync to an external drive on another computer.

Valid approach but consider these:

Risk of corruption propagation: If your primary data gets corrupted or infected (e.g., ransomware), Resilio Sync might propagate that to the backup. Ensure you implement versioning.

 

Offsite Backup:

Data synced via Resilio Sync to another Unraid machine in another location.

This fulfills the "offsite" requirement. However, the same risk of corruption propagation applies.

 

Cloud Backup:

Not yet implemented but planned (1TB OneDrive and Google family plan).

This is critical to complete the 3-2-1 rule.

 

##################

 

Recommendations to Refine Your Strategy:

 

Local Backups

 

Current Tool (Resilio Sync):

While Resilio Sync is convenient, it is more of a file replication tool than a true backup solution.

 

Enhancements:

Enable versioning in Resilio Sync to maintain multiple versions of files, protecting against accidental deletions or corruption.

 

Alternative Backup Tools:

 

Consider True Backup Solutions for local backups:

Duplicati: Open-source, allows encrypted and versioned backups to local or cloud storage.

UrBackup: A great network backup tool with incremental backup support.

These tools can coexist with Resilio Sync if you prefer its ease of syncing.

 

Offsite Backups

 

Your Current Remote Unraid Server:This is a strong offsite backup. Ensure it:

Runs regular health checks on the drives to avoid silent corruption.

Has snapshots or versioning (e.g., Btrfs or ZFS snapshots) to guard against corruption propagation.

 

Cloud Backup

 

Cloud backup is essential for long-term resilience. For 2TB+ growing data, consider:

Backblaze B2: Affordable, scalable, and popular with Unraid users. Offers $5/TB/month.

Amazon Glacier Deep Archive: Ultra-low-cost storage for infrequently accessed data.

iDrive: An excellent all-in-one backup solution, offering plans with fixed quotas.

 

Implementation Tips:

 

Use rclone to automate cloud backups. It integrates well with Unraid and supports multiple cloud providers.

Ensure backups are encrypted before uploading (both locally and during transit).

 

#############

Example 3-2-1 Backup Setup

Primary Data Location:

Your Unraid array.

Local Backup:

Use Duplicati or UrBackup to back up critical data to:

External drive on another computer (onsite).

Secondary Unraid machine (offsite).

Cloud Backup:

Use rclone or Duplicati to sync critical data to Backblaze B2, Amazon Glacier, or iDrive.

 

Extra Tips

Periodic Testing:
Periodically test backups by restoring a sample to ensure data integrity and verify your process.

Growth Management:
Since your data will grow, choose scalable solutions like Backblaze B2 or Amazon Glacier that allow for incremental growth at a manageable cost.

Automation:
Automate backups with tools that notify you of errors (e.g., via email alerts).

Redundancy for Critical Data:
For highly critical data (e.g., family photos), keep an additional offline backup (e.g., a USB drive stored in a safe location).

 

Next Steps

Implement a cloud backup strategy (e.g., rclone with Backblaze B2 or Duplicati).

Introduce a true backup tool for local and offsite backups to reduce reliance on Resilio Sync.

Review your setup periodically (quarterly or biannually).

 

 

 

I would recommend using the plugin appdata backup.
Maybe getting a USB Disk Drive and unsigned disk to run backups from time to time.

 

Thanks! Its something thats been in the back of my mind for a while to sort.

 

I have settled on duplicati for my onsite backup - reason being it is much easier to setup to backup from array to an unassinged device / local computer compared to urbackup (could not figure it out)

 

So on my main onsite backup I now have a resilio sync local backup as well as a duplicati one running, duplicati has smart retention in place, it seemed like the best choice but I may be wrong, if so let me know your thoughts if you wouldnt mind?

 

I still have resillio sync running locally, I am not entirely sure what versioning is as it does not stand out in the gui - doing some reading online I can see some people refering to a .SyncArchive folder and that this should persist for however long the advanced setting "sync_trash_ttl" is set to - some questions:

 

1. Is my above understanding correct and I need to turn this on?

2. What is a good number to set this to?

3. Is it even worth continuing to use this locally now that I have duplicati setup and running locally?

 

Once above is sorted that should hopefully sort out my onsite backup.

 

Onto my offsite backup:

 

Is resillio sync enough for this IF my above understanding of versioning is correct? If so I will set this to the recomended and continue to let it chug along as it is.

 

If not I could do another backup from duplicati to my offsite machine as well, just not sure on the method - is sftp still safe or do you recomend another way?

 

To your second post on the app data backup tool, I am currently using that to backup my flash drive, dockers, app data etc.

 

I am going to invest in backblaze b2 at the end of the month I think, I like the idea of being able to resotre direct from disk providing they still offer this, I am also going to invest in another HDD, stick it in an external enclosure and do a manual backup of my important files maybe every 3 months, not sure where I will keep it though possibly at work, at that rate I would be as well as just using an old NUC running linux to do another backup with duplicati and save lugging the disk around.

 

Edit - Sorry one more question, my volumes for my backup are set at the defualt 50MByte, any issue with with this or should it be changed to something else?

Edited January 22, 20251 yr by vindictive-voyeur2295
Another question

First backup in Dupicati seems to have completed sucesfully:

Source files - 1.9TB

Backup size - 1.4TB

 

Does above sound correct, are backups slightly compressed, is it showing I have no clue about backup software so far  

 

I also got  1 warning:

 

2025-01-22 04:19:16 +00 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-BackendQuotaNear]: Backend quota is close to being exceeded: Using 1.458 TB of 120.000 GB (78.881 GB available) * Apprenatly this is an issue with Duplicati looking on there forums below is taken from there:

"Not sure why this part of the code was changed, but right now it reports the space from /, so whatever disk it “thinks” is mounted there is reported."

 

This part was wrote prior to reading duplicatis forum - Dont understand above as there is plenty space on the drive the backup is being sent to as its an 8tb drive.

Edited January 22, 20251 yr by vindictive-voyeur2295
Extra Info

Conclusion/Summary:

 

Your current strategy is robust, and seems well implemented.... with Duplicati handling both onsite and offsite backups. To optimize your setup:

Consider increasing Duplicati's volume and block sizes to improve performance for large backups.

Use Resilio Sync for real-time synchronization needs, but rely on Duplicati for comprehensive backups and versioning.

Regularly monitor your backup destinations to ensure adequate storage space and verify the integrity of your backups.

 

So Let's review your current backup strategy and address your questions to ensure a robust 3-2-1 backup approach.

 

Onsite Backup:

You've implemented Duplicati for onsite backups, utilizing its smart retention feature. This is a solid choice, as Duplicati offers efficient backup management with customizable retention policies.

Regarding your use of Resilio Sync locally, it's important to note that while Resilio Sync facilitates real-time file synchronization, its versioning capabilities are somewhat limited. By default, Resilio Sync stores previous versions of files in the .sync/Archive directory for 30 days on desktops. This duration can be adjusted using the sync_trash_ttl parameter in the advanced settings.

 

Review/See more here: https://help.resilio.com/hc/en-us/articles/204754239-Using-Archive-for-file-versioning-and-restoring-deleted-files
 

Addressing Your Questions:

Quote

Is my understanding correct, and do I need to turn this on?

Yes, your understanding is correct. Resilio Sync's versioning is managed through the .sync/Archive folder. To ensure this feature is active, verify that the "Store deleted files in Archive" option is enabled in the folder preferences. You can adjust the retention period by setting the sync_trash_ttl parameter to your desired number of days.

 

Quote

What is a good number to set this to?

Default...
The default is 30 days, which is suitable for most users. However, depending on your storage capacity and recovery needs, you can adjust this period. Setting it to zero will retain versions indefinitely, but be cautious as this can consume significant storage space over time.

 

Quote

Is it even worth continuing to use this locally now that I have Duplicati set up and running locally?

Since Duplicati provides comprehensive backup solutions with advanced retention policies, maintaining Resilio Sync locally might be redundant. However, if you require real-time synchronization across devices, Resilio Sync can complement Duplicati by ensuring immediate file consistency, while Duplicati handles scheduled backups and versioning.

 

###########

Next lets review:
Offsite Backup
For offsite backups, relying solely on Resilio Sync's versioning may not be sufficient due to its limited versioning capabilities. Implementing Duplicati for offsite backups would enhance your data protection strategy. SFTP is a secure method for transferring backups offsite, and Duplicati supports SFTP, making it a viable option for your needs.

 

Additional Considerations:

Duplicati Volume Size:

The default volume size in Duplicati is 50MB. For large backups, increasing the volume size can improve performance. Users with substantial data sets have reported better performance with volume sizes ranging from 250MB to 1GB. Adjusting the deduplication block size from the default 100KB to a larger size (e.g., 1MB) can also enhance performance, especially for large backups.

As seen here: https://forum.duplicati.com/t/backing-up-2tb-50mb-remote-volume-size-okay/11912

 

Backup Compression

Duplicati applies compression to backups, which can result in smaller backup sizes compared to the original data. The extent of compression depends on the data types being backed up. For instance, text files compress well, while already compressed files like videos or images may see little to no reduction in size.

 

Warning Message

The warning regarding backend quota being close to exceeded is a known issue in Duplicati, where it may misreport available space. If your destination drive has ample space, this warning can typically be disregarded. However, it's advisable to monitor your backup destination to ensure sufficient space is available.

 

 

Thanks for this - sorry for the delay I have had to deal with a CPU that decided to fail this week in my backup server, but all working again.

 

Duplicati is now running on both my onsite and offsite machine with smart retention in place, volume size in duplicati is now 500mb, I went to change the deduplication block size to 1MB but when I clicked the option it showed the default is 1MB already.

 

I have not changed anything in resilio sync yet as I am deciding wheather I want to keep them running or not, I think I may turn them off, at least at my onsite machine, having them accesible on my offsite machine is handy since I spend more time on it than my home one right now.

 

Seems the final thing for me now is to get backblaze setup at the end of the month.