Cache: Prefer & Cache: Only with Backup [One Way sync]

[Hexenhammer]

Hello,

I would like to request a feature that probably was requested before just in different wording.

 

Please consider adding One or Two Cache Options, Cache: Prefer and Cache: Only with Backup [one way sync: Cache to Array]

When Mover invokes it will create a 1:1 copy of Cached Share on the array, leaving the cache intact and active.

Next time when mover runs it will always threat the cache as the main source and delete the old data

Basically its Internal option for the same function CA Backup plugin does [that plugin is specific to system folders]

 

If two copies of same data is confusing for the unRaid, then the backupod data on the array could be saved as zero compression zip.

 

Why this is needed:

Main usage is 10Gb networked remote drive on cache for max speed, maybe even RAID0 [or whatever chosen by user]

On remote PC you have access to fast storage that you can run games from, emulators, software, video encoding whatever you want [unRaid pools have many types of raid].

And This option will auto backup your files to the slower array.

 

 

 

[trurl]

User shares are simply the combined top level folders on array and pools. To backup something from cache, it must be saved in some user share that doesn't duplicate the existing share on cache.

1 hour ago, Hexenhammer said:

If two copies of same data is confusing for the unRaid, then the backupod data on the array could be saved as zero compression zip.

Exactly.

 

This functionality seems out-of-scope for the way cache options for user shares work with mover. You can use a script to make a backup, but backup isn't really what the cache options are about.

[-Daedalus]

To be fair, this has been requested before, and I know I'm not the only one that's implemented this as a script.

The standard answer is it's not an option because of how mover runs, but I don't think it's too much of an issue to add this functionality, with a big red warning to use array-only shares (or even just don't allow the destination share to be selected at all if it's set to use cache)

[trurl]

2 hours ago, -Daedalus said:

use array-only shares

There is no such thing.

 

11 hours ago, trurl said:

User shares are simply the combined top level folders on array and pools.

 

[Hexenhammer]

5 hours ago, -Daedalus said:

To be fair, this has been requested before, and I know I'm not the only one that's implemented this as a script.

The standard answer is it's not an option because of how mover runs, but I don't think it's too much of an issue to add this functionality, with a big red warning to use array-only shares (or even just don't allow the destination share to be selected at all if it's set to use cache)

Can you share the script please?

 

[Hexenhammer]

2 hours ago, trurl said:

There is no such thing.

 

 

 

Share that is only on the array and not on the pool?

[itimpi]

16 hours ago, Hexenhammer said:

Basically its Internal option for the same function CA Backup plugin does [that plugin is specific to system folders]

Not really as that plugin makes the backup to a different share.

[Hexenhammer]

3 minutes ago, itimpi said:

Not really as that plugin makes the backup to a different share.

Yep that will work great, when mover runs or some specific script that runs after mover it will create a a subfolder in CacheBackup share, subfolder will be named after the Cache:Only share

 

Cache Only Share: Games

goes into CacheBackup/Games located for example on Disk 3+ Disk 6

 

And every period of time it will delete [if it can update just the changed files it can be faster, but i have no idea if its possible] the CacheBackup/Games and re-create it based on Cache Only share.

 

 

 

[-Daedalus]

11 hours ago, trurl said:

There is no such thing.

 

 

 

8 hours ago, Hexenhammer said:

 

Share that is only on the array and not on the pool?

 

This is what I meant.

 

8 hours ago, Hexenhammer said:

Can you share the script please?

 

 

Sure. You'll need to change the names of shares obviously, and I take no responsibility for your server catching fire, etc.

 

#!/bin/bash
logger "SSD Pool > Array backup started"

 

# Backup things from SSD pool to array disks
rsync -avh --del --stats \
--exclude 'backup/' \
--exclude 'downloads/' \
--exclude 'movies/' \
--exclude 'tv/' \
--log-file=/mnt/user/backup/ssd-pool/backup-log.txt \
/mnt/ssd/ /mnt/user0/backup/ssd-pool/

logger "SSD Pool > Array backup finished"

 

[Hexenhammer]

11 minutes ago, -Daedalus said:

 

 

This is what I meant.

 

 

Sure. You'll need to change the names of shares obviously, and I take no responsibility for your server catching fire, etc.

 

#!/bin/bash
logger "SSD Pool > Array backup started"

 

# Backup things from SSD pool to array disks
rsync -avh --del --stats \
--exclude 'backup/' \
--exclude 'downloads/' \
--exclude 'movies/' \
--exclude 'tv/' \
--log-file=/mnt/user/backup/ssd-pool/backup-log.txt \
/mnt/ssd/ /mnt/user0/backup/ssd-pool/

logger "SSD Pool > Array backup finished"

 

 

Can you explain a bit what it means?

 

I see exclusions, thats what, excluded shares on the cache Pool?

SSD Pool is the name of the cache it backups? If mine is Gaming, then i should replace it for Gaming?

What -del does? it deletes something

 

P.S. sorry iif these are dumb and basic questions im new unraid and linux user and never used rsync.

BTW is there a GUI for RSYNC? I need something EASY and dumb friendly to backup to Google drive

I have a windows backup tool that is amazing, paid for license upgrades 3 times over the years its called 'insynchq', very easy to us, has tons of features and simple gui, also supports many cloud services.

But for it to work i need to keep my gaming PC on  to do the backup or run a VM with windows

 

Thanks

 

[-Daedalus]

Yeah sure, my bad!

 

#!/bin/bash
logger "SSD Pool > Array backup started"

The inner-workings of very first line aren't super-important, just know that it should be at the start of every script you make for now.

This first bit just sends a message to the unRAID syslog "SSD Pool...."

 

# Backup things from SSD pool to array disks
rsync -avh --del --stats \

This next bit is just a comment, saying what it's doing.

The next line is the start of the rsync command. In general, it looks like:

rsync -someOptionsHere /path/to/source/directory /path/to/destination/directory.

 

So with this one, the options are a, v, and h.

a = archive. Basically what you want. Copy any new files from source, to destination, overwriting if the file is newer, but don't delete anything from either side. Also keep all the ownership, permissions, etc.

v = verbose. Just log what it's actually doing while it's doing it, in case something screws up and I need to look at it later.

h = human-readable. Just so it's in KB/MB/GB instead of bytes.

 

Then the next bits are more options, but they don't have single-letter shorthand that can be combined together, so you have to specify them separately.

--del = delete from source directory after copy to destination is finished. Because the whole point is we want stuff off the cache here.

--stats = more info about what it did at the end of the operation, again for logs.

 

The very last thing is the \ symbol. This basically says, continue the command, but on the next line. Just for my own readability.

 

--exclude 'backup/' \
--exclude 'downloads/' \
--exclude 'movies/' \
--exclude 'tv/' \
--log-file=/mnt/user/backup/ssd-pool/backup-log.txt \
/mnt/ssd/ /mnt/user0/backup/ssd-pool/

 

These continue the options like before, this time to exclude certain directories from the copy, becuase I don't care about them much, so it's not the end of the world if they get lost. I mostly made this script to keep my documents and pictures a little more safe.

 

The second-last line is an option to write all the logging we've been doing to a file. In this case "backup-log.txt" on the SSD.

And the very last line specifies our source and destination. So my cache is called "ssd" so we have "/mnt/ssd" as the source, and I want it to go to a share called "backup" and a folder in there called "ssd-pool".

 

The important bit here though is that we don't want it going to the user share (/mnt/user/backup) because this includes the cache and the spinning disks, so the files wouldn't actually move, or might even get corrupted, so we specify "user0" (this is a short-hand for "the share, but exclude cache").

 

Hopefully all that made sense!

Edited January 7, 2023 by -Daedalus

[Hexenhammer]

3 hours ago, -Daedalus said:

Yeah sure, my bad!

 

#!/bin/bash
logger "SSD Pool > Array backup started"

The inner-workings of very first line aren't super-important, just know that it should be at the start of every script you make for now.

This first bit just sends a message to the unRAID syslog "SSD Pool...."

 

# Backup things from SSD pool to array disks
rsync -avh --del --stats \

This next bit is just a comment, saying what it's doing.

The next line is the start of the rsync command. In general, it looks like:

rsync -someOptionsHere /path/to/source/directory /path/to/destination/directory.

 

So with this one, the options are a, v, and h.

a = archive. Basically what you want. Copy any new files from source, to destination, overwriting if the file is newer, but don't delete anything from either side. Also keep all the ownership, permissions, etc.

v = verbose. Just log what it's actually doing while it's doing it, in case something screws up and I need to look at it later.

h = human-readable. Just so it's in KB/MB/GB instead of bytes.

 

Then the next bits are more options, but they don't have single-letter shorthand that can be combined together, so you have to specify them separately.

--del = delete from source directory after copy to destination is finished. Because the whole point is we want stuff off the cache here.

--stats = more info about what it did at the end of the operation, again for logs.

 

The very last thing is the \ symbol. This basically says, continue the command, but on the next line. Just for my own readability.

 

--exclude 'backup/' \
--exclude 'downloads/' \
--exclude 'movies/' \
--exclude 'tv/' \
--log-file=/mnt/user/backup/ssd-pool/backup-log.txt \
/mnt/ssd/ /mnt/user0/backup/ssd-pool/

 

These continue the options like before, this time to exclude certain directories from the copy, becuase I don't care about them much, so it's not the end of the world if they get lost. I mostly made this script to keep my documents and pictures a little more safe.

 

The second-last line is an option to write all the logging we've been doing to a file. In this case "backup-log.txt" on the SSD.

And the very last line specifies our source and destination. So my cache is called "ssd" so we have "/mnt/ssd" as the source, and I want it to go to a share called "backup" and a folder in there called "ssd-pool".

 

The important bit here though is that we don't want it going to the user share (/mnt/user/backup) because this includes the cache and the spinning disks, so the files wouldn't actually move, or might even get corrupted, so we specify "user0" (this is a short-hand for "the share, but exclude cache").

 

Hopefully all that made sense!

 

 

Hi,

yes it was very detailed. Thank you

So in my case i just need to remove the -del [If im using the cache and just want it to be backed up, not removed]

 

In the last command, can i point it to single file to copy [like virtual hard drive image]

 

Is something like this ok? to backup virtual HDD Stright to HDD1 in the array

 

 

#!/bin/bash
logger "SSD Pool > Array backup started"

 

# Backup things from SSD pool to array disks
rsync -avh --stats \
--log-file=/mnt/Disk1/backup/backup-log.txt \
/mnt/ssd/fileio.img /mnt/Disk1/backup/

 

 

Or this one for a share that mounted on Windows 11 and I run old games and emulators from [basiclly use as remote storage]

 

#!/bin/bash
logger "SSD Pool > Array backup started"

 

# Backup things from SSD pool to array disks
rsync -avh --stats \
--log-file=/mnt/Disk2/backup/backup-log.txt \
/mnt/gaming/ /mnt/Disk2/backup/gaming

 

[-Daedalus]

Second example looks spot on.

 

For the first one, I believe rsync always expects a directory, but I think if you change the desination to "/mnt/disk1/backup/fileio.img" it should be alright.

 

I'd advise running the above with the "--dry-run" option, so that you can verify what will happen. If you're happy, remove that and kick it off.