Gymnae

I'm still facing the issue, that mover with this plugin won't move nor sync files. I have two ZFS pools, one primary with a single NVME and a secondary, which acts as long term storage. The NVME is primary, the second is secondary storage. Mover should sync all files smaller or equal to 98 MB and move all files larger than 98 MB from primary to secondary Thresholds: Moving: 70% Free-up: 60% Fill-up to: 60% I played with these numbers to no change. I am running one central configuration and not a per-share config. I tried that before, but also did not help. Mover won't sync nor move new files that should be covered by the configuration. That extends to small files, which should be synced, but also to bigger files, which should be moved. I can verify this by running rsync manually and see what is synced between primary and secondary. Realizing this gives me the impressions that without running rsync manually, I do not have parity protection when this plugin controls mover. ca.mover.tuning-debug-20251031-0843.zip

Nope, the move direction is cache to array. It happened again this night. It's always the same amount: I attached a fresh debug log. To note: I tried adjusting the moving threshold between 80% and 5% - the mover still took the same action and moved the same amount. As an unrelated side-note: Since the "server" is sitting in a room close to my kids room, the 3hrs moving action every night appears to wake up the kids, because the enterprise HDDs are f'n loud. Interesting how configurations of a NAS has effect on sleep patterns. (fun) ca.mover.tuning-debug-20250903-0915.zip

It seems like my config leads to a loop. Every time the mover runs, it moves the same amount of 360 something GB, but I know that I don't have that much ingress on cache to move. Something makes it move data back and forth, causing unnecessary writes and reads. For now, I need to disable mover. ca.mover.tuning-debug-20250901-1121.zip

Agreed, in the case of configuring syncing and moving based on file size there should not be thresholds. That would lower complexity. The total available space as well as the minimum free space are managed from upstream unraid share configuration.

Oh, interesting, I might be confused then. I'd like to set it globally across all shares in the global setting, that seems to make sense. Could you give me an example working values for sync for the first three percentage values (move threshold / free down / fill up)? Should sync file size minimum file size for move be equal? I'm trying the the following now

With my config even small files were moved, while my intention was to keep all files below the sync threshold on primary (1st ZFS dataset) and sync copies to secondary (2nd ZFS dataset) while moving only files above the configured threshold to secondary

Does this feature affect the entire share, meaning that all files of the share will reside then on both primary and secondary, or does it obey configured skipped files types and skipped file lists? Aka would this feature enable the use case of having files with certain parameters like age, size, file type, path etc. moved to secondary but keep others on both primary and secondary?

With master storage I meant that if a file gets deleted or manipulated on secondary storage, then the state of the file on primary storage is restored to secondary storage during a sync. Thus you avoid collisions or undeletion. Yet having read your exchange above, I concur that convoluting plugins with feature creep creates confusion. This superb plugin could focus on tuning the vanilla mover with additional tweaks. But functionality that extends beyond the scope of mover behavior might best become part of an extra plugin A separate sync plugin might be a good idea, it could contain the functionality which we discussed above: Offers users to define affected shares, file types, file sizes, regex, paths etc. to be covered by the plugin Plugin then controls on which storage data covered by the plugin configuration gets accessed, i.e. stop spinning up disks to access small files present on fast primary storage (e.g. cache) but also parity protected on slow storage (array) Controls which storage acts as master storage for affected files, see above what I mean by that, aka one location replicates its state to the second location no matter what happens on that second location - this is required because we work with copies instead of vanilla moved files Tells mover to create copies of data covered by the user configuration for parity protection on a secondary storage (e.g. array or different ZFS pool) Supersedes mover if mover wants to move files off a storage but the configuration of the sync plugin is against it Offers bidirectional sync to achieve desired state also for per-existing data from before plugin installation, but with the caveat that the mater storage location will define the state of files affected from now on. i.e. after activating a configuration and the plugin and following the initial sync, changes made to files falling under the configuartion in the secondary location will be overwritten by sync operations of the plugin Could be a complementary plugin to mover tuner

Create parity protection for all files primarily stored on a fast storage medium which should only be accessed by consumers (SMB/NFS mounts, docker shares) on this fast storage medium Continuous synchronization with backfill, aka making sure that files residing at a configured path, match of a regex, part of a whitelist, share, size etc. are synced to a designated storage (sync target), where they are accessed by consumers. This sync should be independent of modification time and trump mover configuration, aka be higher in priority, if sync settings contradict mover settings. All changes should occur on the sync target and propagate to sources. The sync target should therefore act as master data location, also propagating deletion.

Hi, thanks for your lightning fast reply. I removed the custom overrides to have one central configuration for the plugin. What I meant regarding syncing: Files, which should reside mainly on primary, are moved to secondary and live only there. No copy on primary. For example, a couple of .jpeg thumbnails were created two days ago and exist only on secondary. My suspicion is that something is amiss between mover, mover tuner and the ZFS datasets. I had to manually create datasets, because unraid only created them on secondary or vice versa. So, for example, a share was created as a ZFS dataset with unraid, but only created on secondary. So I created its sibling on primary. I am not sure if this could be a problem, the GUI does not give any indication that it is a problem. It seems random, where files end up, and it does mean that the disks spin up for files a few kilobytes in size, when they should only spin up when files beyond a configured threshold and age are accessed via a user share.

Hi there, thank you for maintaining this plugin, which has become the corner stone of my attempt to keep disks spun down and utilize fast NVME storage where useful. My scenario is comparable to what I read in the thread a couple of times: Access modified files to a certain size only on fast storage Access files of certain file types only on fast storage Keep all files in sync between fast storage and spinning disks In my case I don't use an array, but two ZFS pools. One is called "vault" and includes two spinning disks plus an SATA cache, the other is called "pioneer" and consists out of only one 2TB NVME. Example: I'm using a photo library and want to access all thumbnails and smaller files from "pioneer", but bigger files from "vault", like movies. So the disks keep spun down. What I see: Even with mover tuner configured to obey points 1 - 3 of my scenario and even with an ignore files txt in place, unraid moves smallish files to "vault", meaning there's no copy of them on "pioneer". Thus I am not sure, if mover tuner has an effect or if something's amiss in my setup. I attached a fresh debug zip ca.mover.tuning-debug-20250724-1524.zip

I did that as well, but the behavior described also appears without this plugin, hence this post

Hi there, I'm a bit confused about the right settings. I searched the Forum and read the FAQ. My scenario: Unraid 7.0.0 Two ZFS pools, no traditional array: "Pioneer" with one 2TB NVME "Vault" with two mirrored 20 TB HDD and one 1TB (L2ARC) cache SSD Both pools hold ZFS datasets of the same name Both pools have a pre-allocated 20% dataset for keeping space usage below 80% Mover has the following setting: Move from primary ("Pioneer") to secondary ("Vault") All shares are user shares All shares are accessed via NFS, with the user share mounted by the consumers The plugin Mover tuner has the following settings (the current version as of this posting has no effect on achieving this goal, hence this post): Move when primary is above 75% Move until primary is below 60% Don't move based on age Move to secondary based on size Size is set to 242GB Sync primary to secondary (for "parity") The goal Keep files below a size threshhold on primary until a limit of allocated volume space is used, only move bigger files until that limit is reached Use mover tuner to move files based on size (this is a feature offered by the plugin Mover tuner, but I feel the issue at hand is independent of the plugin) Mover tuner is not a must, in fact I would rather achieve this with a vanilla install of Unraid Keep the HDDs of "Vault" spun down when Files below a certain size threshhold are accessed Files which are too big for "Pioneer" pool but reside on the L2ARC device of "Vault" Spin the disk up to access big files which have been moved to secondary and don't reside on the L2ARC device What I see All files, even small ones, are immediately moved to secondary - this is independent of whether or not the move tuner plugin is active The disks never spin down Are there some crucial settings I might be missing or have wrong? datengrab-diagnostics-20250227-1011.zip

Hi there, I'm a bit confused about the right settings. I searched the Forum and read the FAQ. My scenario: Unraid 7.0.0 with two ZFS pools, no traditional array: Pioneer with one 2TB NVME Vault with two mirrored 20 TB HDD and one 1TB Cache SSD Mover has the following setting: Move from primary ("Pioneer") to secondary ("Vault") Mover tuner has the following settings: Move when primary is above 75% Move until primary is below 60% Don't move based on age Move to secondary based on size Size is set to 242GB Sync primary to secondary (for "parity") All shares are accessed via NFS, with the user share mounted by the consumers The goal Keep as many files on primary, only move bigger files. Keep the disks spun down when, for example, Nextcloud access documents or photo sharing immich is accessing thumbnails and videos Spin the disk up to access big files which have been moved to secondary What I see: All files, even small ones, are immediately moved to secondary Are there some crucial settings I might be missing or have wrong?

Re-formatting the cache SSD seemed to have done the trick. Time Machine backups as well as other shares are moved without a hitch. I think it might have been my tweaking of the BTRFS partition with the same flags I used on my system drive. Now I kept it vanilla and will keep it this way