Introduce "Cache Copier/Syncer" as addition to Mover


Recommended Posts

Hi,

 

I would love for an additional "cache copier/syncer"  as an addition to the existing mover. What exactly do I mean by that:

 

Say I only want to use one large SSD cache drive (instead of 2 or more to get an array-like redundancy, since large SSDs are still somewhat pricey and, of course, take up space and ports). I would like to have all my docker/vm/often used data ond the cache drive, but I would also care to have that cache-data backed up on the standard array/data drives on a regular basis. Hence, I would like for a nightly backup to the array to occur, but not with moving, but copying/syncing instead. Ideally, the share which is used for this would be read-only to all users (a "cache-sync-share", if you will), so only the syncer can write to it (root definitely shouldn't, but could). However, the users/root CAN use it to restore the data from this share (in case of SSD failure), or, of course, delete it entirely,

 

This would be very helpful, because I could save on SSDs without sacrificing redundancy. I personally do not care about the performance of the syncer, because this job would occur during low-to-none-load times.

 

I also know that I can probably build this very feature on my own, using the "cache: only" setting +  rsync/rclone or something similar (most basic: cp -uvr + crontab + remove data-singles without cache-equivalent), but I really feel like this is something many people would benefit from.

 

Maybe this request will get a few "+1"s, which would be nice, but otherwise I would also be happy to just script-kiddie implement this on my own ;).

 

 

Link to comment
  • 7 months later...
On 5/14/2021 at 11:38 AM, RT87 said:

I also know that I can probably build this very feature on my own, using the "cache: only" setting +  rsync/rclone or something similar (most basic: cp -uvr + crontab + remove data-singles without cache-equivalent), but I really feel like this is something many people would benefit from.

Is CA Backup maybe the solution here in combination with luckyBackup or any other sync application for you other important data?

Link to comment

+1

 

But I'm thinking about slightly different logic. In share setting add checkbox "Sync cache with main pool" (or smth like this) when cache mode either Only or Prefer. In normal operation data are read/write to cache and synced with main pool when Mover is operating. Obviously this will lead to misunderstanding of word "move", so might be added another action "Sync" similar to "Mover" with it's own button, settings, etc (in settings might be checkbox to schedule syncing alongside with mover). When cache drive fails then data operation remains normal, but read/writes should be manually redirected to main pool. I think automatic redirection is not possible because of possible data consistency damage by process which will try to use data from main pool while they state based on data from cache pool. To restore normal operation user might change cache pool (in case of additional cache pools) or replace cache drive in current cache pool. In both cases syncing must be performed right away to eliminate uncertainty of where to write/read from data.

 

I think this logic is the best from user perspective. You setup couple of settings and in case of failure just replace failed drive. Just like main pool operation. IMHO this feature will be very helpful because some of the users don't want to protect data on cache drive with another cache drive when main pool does the same.

Link to comment
  • 1 month later...
On 1/2/2022 at 8:09 AM, ich777 said:

Is CA Backup maybe the solution here in combination with luckyBackup or any other sync application for you other important data?

For now I'm using Vorta, but sure, any reliable backup solution will do...  but still, I can't be the "only one" with this use-case, so an "official" way would be beneficial.

Link to comment
35 minutes ago, RT87 said:

For now I'm using Vorta, but sure, any reliable backup solution will do...  but still, I can't be the "only one" with this use-case, so an "official" way would be beneficial.

But what about CA Backup, if you configure it like this:

grafik.png.7fbf47a61dc3d4c5825bbd9902e23887.png

 

It should be what you are trying to accomplish and you could also exclude folders, use a compression if you want to, set schedules,...

I know it is not designed for this but it should do the job just fine.

 

I really don't think that Unraid was designed for that use case, I also totally understand why you want this, I also have such disks that I backup frequently through luckyBackup with a schedule and snapshots.

Link to comment

I guess both pretty much do the same thing, featurewise, I just went with Vorta because it uses borg and it's something I can use more flexible, i.e. it's not unraid specific.

 

With "official" I meant somehintg along the lines of "another cache use-type", just plain and simple, no versioning or anything, just hold-on-cache-and-sync-to-array-via-cronjob

Link to comment
10 minutes ago, RT87 said:

 

With "official" I meant somehintg along the lines of "another cache use-type", just plain and simple, no versioning or anything, just hold-on-cache-and-sync-to-array-via-cronjob

That introduces a whole can of worms, because of the way Unraid merges the file systems across all array and pool drives. Multiple files with the same name and path on different drives show up as a single file in the user share filesystem, with only one actually being the target of reads and writes.

Link to comment
23 hours ago, JonathanM said:

That introduces a whole can of worms, because of the way Unraid merges the file systems across all array and pool drives. Multiple files with the same name and path on different drives show up as a single file in the user share filesystem, with only one actually being the target of reads and writes.

It wouldn't need to be the same path, I could simply backup the appdata cache-path to /mnt/user/appdata_cache_backup or somthing similar.

Link to comment
11 hours ago, trurl said:

That is what the CA Appdata Backup plugin lets you do.

well almost, I still would like to just have it run with the mover-job. Just as a regular no-plugin-or-config-required setup, simply by using a new pool-usage-type "prefer-cache-but-sync-to-array-during-moving"
 

I'm a little worried that people will neglect this because they may not fully realise the danger of running everything on your cache without a cache-array/backup. But maybe I'm a little off here, I don't know, as I said, I'm happy with my solution, I just think a more elegant way might be nice.

Link to comment
6 hours ago, RT87 said:

would like to just have it run with the mover-job

More flexible to have it run as its own thing with its own schedule.

 

Looks like Mover Tuning plugin will let you add scripts to run before/after mover. Seems to be a work in progress, I don't use it.

 

 

Link to comment
  • 3 weeks later...
  • 5 months later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.