Hi all, I need some suggestions for workflow.

 

I'm a photographer, and before my recent NAS + unRAID build I have been using external hard drives for archiving, initially unloading memory cards to a folder in my Macbook Pro internal drive, offloading to a master archive as needed, then using Carbon Copy Cloner to duplicate that external drive to a second external drive for redundancy. Now that I have my NAS w/unRAID built, I need help on best workflow suggestions. Here's my goal:

 

• I want to use the unRAID NAS as primary photo archive storage (also have 1 parity drive in use, but I know that's not "backup")
• I SMB over wifi LAN from my MBP to the server to access the archive (I don't plan to use things like Lightroom or any library-building type software with the *archive* files, that's specifically done locally on my MBP before copying over to the array) for viewing and copying/downloading as needed.
• I need some sort of cloning method within unRAID to create a true backup to the array. Initially I had thought "maybe there is an unRAID version of Carbon Copy Cloner that would work via Unassigned Devices" but naturally there is not, at least not yet. I need a software solution that can clone the photoarchive share (or if not cloning the share itself, all the folders and files within it) WITH the same sort of functionality that I had within Carbon Copy Clone (ie, incremental-only updates, meaning it will only scan for changes and only spend time copying over additions and modifications instead of cloning every sector and byte)
• To create my third copy in 3-2-1 to send to the safe deposit box I can easily just plug both externals into my MBP and use CCC as usual and clone one to the other

 

Does anyone have good suggestions for accomplishing this? Ideally within an app since I'm not the GREATEST with command line?

 

Thanks!

There are quite a few backup threads on the forum.  A lot to wrap your head around, as there are several different programs available that use a graphical interface to control what/where/when/how data is backed up.  And probably an equal number of opinions on which is best.  This is the Internet after all.  😆

 

Personally, I have found just using a simple bash script which runs periodically, using the User Scripts plugin works fine for me.  It isn't that difficult to set up, and l learned some about my server in the process.  The Linus rsync command does all of the work, and there is enough info in the forum (and on the Internet) that made it easy enough to figure out.

 

Below is the script I use.  I have it also send me messages on Telegram (which I use for monitoring my server and other network services).  You can exclude that if you wish.  The rsync settings I use make an uncompressed mirror image of the folders listed, and only transfer/update the data which has been changed.  For this script, I backup to a 8TB external drive.  I swap out the drive every month or so with another, which I keep in an offsite location:

 

#!/bin/bash

# Telegram variables - From Github - h/t to RealLukeManning
TOKEN=Token_Goes_Here
CHAT_ID=Chat_ID_Goes_Here
URL="https://api.telegram.org/bot$TOKEN/sendMessage"

# Send message backup is starting
MESSAGE="Starting sync to USB2 - $(date)"
curl -s -X POST $URL -d chat_id=$CHAT_ID -d text="$MESSAGE" > /dev/null

# Start Music backup
USB2STAT=$(rsync -ah -p --delete-during --stats "/mnt/disks/Samsung_Flash_Drive_FIT/UNRAID_Backups" "/mnt/user/Backup/Ryzen Home desktop" "/mnt/user/Backup/appdata backup" "/mnt/disks/EXT_USB_FAN_2")

if [[ $? -eq 0 ]]; then
  # Send message that all backup is COMPLETE
  MESSAGE="Backup to USB2 is COMPLETE - $(date)"
  curl -s -X POST $URL -d chat_id=$CHAT_ID -d text="$MESSAGE%0A%0A$USB2STAT" > /dev/null
  echo $MESSAGE
  echo $USB2STAT
else
  # Send message that backup FAILED
  MESSAGE="***ERROR*** Backup to USB2 FAILED - $(date)"
  curl -s -X POST $URL -d chat_id=$CHAT_ID -d text="$MESSAGE%0A%0A$USB2STAT" > /dev/null
  echo $MESSAGE
  echo $USB2STAT
fi

 

This is the messages sent to my Telegram account.  This week, it parsed 5.12TB of data to mirror, but only needed 65.52GB to update, which only required less than 12 minutes to complete:

 

 

 

@ConnerVT I appreciate the effort in that response.

Do you just then go manually run the script whenever you feel like doing a backup/sync?

2 minutes ago, thirtythreemangos said:

@ConnerVT I appreciate the effort in that response.

Do you just then go manually run the script whenever you feel like doing a backup/sync?

One can automate running the backup via a script by either linking it within Unassigned Devices to the drive being plugged in, or alternatively at a scheduled time by running it using the User Scripts plugin.

 

Rsync looks super handy. But when I went to this page I just got super overwhelmed at the prospect of actually ensuring all the right options were (or weren't) actually going to be utilized and employed. I dig some digging into Carbon Copy Cloner and figured out a way for it to use an SMB volume (if that's even the best way to refer to that) after creating a new User on unRAID that has read/write permissions. CCC was able to save the username/password to access that volume/share through my Macbook Pro and over the network. Tested the functionality and it actually works! But I imagine its going to be super slow, so the initial clone sounds like a nightmare. I'm gonna figure out a way to use Rsync (or even just like Krusader?) for a local-to-local clone when plugging the external drive directly into the server, then for subsequent backups that can just be done plugged into the MBP and through the network. Hopefully that ends up working  But at least with this method I retain the SafetyNet feature and the userfriendliness that CCC gives over just command line approach.

Edited June 10, 20242 yr by thirtythreemangos

1 hour ago, itimpi said:

1 hour ago, thirtythreemangos said:

@ConnerVT I appreciate the effort in that response.

Do you just then go manually run the script whenever you feel like doing a backup/sync?

One can automate running the backup via a script by either linking it within Unassigned Devices to the drive being plugged in, or alternatively at a scheduled time by running it using the User Scripts plugin.

 

^ This.  My script is run weekly via the User Scripts plugin.  This plugin gives a place to create/edit scripts, and assign them a schedule as to when they will run.  I run this script weekly (as it is mostly a backup of other backup files I have on my server).  I back up onto a 8TB external USB drive, which are managed by the Unassigned Devices plugin.

 

I know, a lot to learn when you first fire up a new Unraid system for the first time.  But the benefits of Unraid are many.  As it is a long maintained application with many people using it, most questions, issues and wants have been already asked by others before we got here.  Add that the community is pretty friendly and helpful, you usually don't need to invent the wheel - just need to build from the work of others.