Jaybau

Perhaps the root cause is Unraid: Files should not continue to exist if they fail to transfer/generate.

Thank you...I believe this explains how I might have received my DFI integrity issue (files did not completely transfer, which I believe is caused by an Unraid defect for allowing/default 0 free space for a drive or the way Unraid pool handles 0 free space). I do wish DFI had the option to regenerate hashes for the suspect files.

Thank you. I didn't know a high temperature would be a failure (array is functioning). I'll see if moving the drive in my case will result in a lower temperature.

I frequently get array healthy report failures. But I see no other error messages, and my array seems to functional as expected. So I'm not sure what the failure is, or what is unhealthy. Unraid Status: 18-08-2022 00:20 Notice [TOWER] - array health report [FAIL] Array has 4 disks (including parity & cache) tower-diagnostics-20220818-0709.zip

Frequent false positives is not acceptable to me if I can find a better alternative.

DFI is in the repository, and I see no mention of it being broken. However, I have recently received something about "BLAKE3 hash key has unexpected length" on good files. I know the files are good because I create other hash files not by this plugin. I don't trust this plugin 100%. I don't see any issues posted on Github. And support threads on Unraid is not the best place for support/issues (prefer Github). My DFI version is 2021.08.24e.

When there's an update to a docker, where can I go to see what has changed?

Yes, 100 MB.

With a "minimum free space" of 0 on a cache drive, what happens when beyond available free space? Any problems, or just an "out of space" warning message? Example scenarios: A) You transfer a file, and the drive goes to zero. B) You are performing a task at the OS level (e.g. building a docker container). C) Performing multiple file transfers/tasks that consume the storage at the same time.

Here is the recreated scenario: We are expecting the files being copied to use the array, which has enough storage. Other notes: 1) The cache drive says 86.0 KB free, which is below the minimum limit of 100MB. 2) I'm surprised the guest OS doesn't recognize the file being copied is larger than available space, and prevent the transfer. I'm not sure if/how Unraid reports available storage to a guest OS. 3) The file copy action will keep repeating over and over, automatically retrying, and constant read/writes. tower-diagnostics-20220812-0924.zip

I ran into a different mess when with the default of zero (I don't think the default should be zero since zero is the most insane value...at least for me it was).

Is the default 0?

Thanks for pointing out there are two separate minimum free settings. For my scenario, I have both minimum amounts set for cache and user share, the same value.

When the cache drive becomes full, automatically invoke the Mover to move files from cache to array. I don't want to wait for the schedule to move files, nor manually move files. Automatic move can be an option.

When my cache is full, it doesn't switch to writing to the array. Even with minimum free space set.

I would like to see BTRFS device stats being displayed in the webGUI and receive notifications for errors: # btrfs device stats /mnt/disk1 [/dev/md1].write_io_errs 0 [/dev/md1].read_io_errs 0 [/dev/md1].flush_io_errs 0 [/dev/md1].corruption_errs 0 [/dev/md1].generation_errs 0 Thank you.

You don't trade any of Unraid's features with SnapRAID. With SnapRAID, it adds integrity+snapshot parity, multiple snapshots/parity if you want, and you can store the parity wherever you want. Snapraid is an application, not a file system. There's zero risk to use for sync/scrubs. SnapRAID is pretty clever, and I don't think it would take someone clever to integrate with Unraid. You simply configure SnapRAID what to snapshot, where to store the snapshot, where to store the metadata. Then you just run syncs and scrubs. I haven't tested Snapraid with restores and making sure there's no conflict with Unraid parity or BTRFS or DFI. It's possible the other subsystems would need to get updated if I did a restore from Snapraid. But for now, at least I have some additional easy and cheap protection, if I ever need it.

Unraid has manual options/capability, but not well integrated, automated, stress free, or confidence inspiring when I'm reading posts. Also, XFS is the preferred/recommended/default filesystem, therefore a lot of users (especially new) are not going to opt for BTRFS. From my understanding you have to do a BTRFS scrub to detect a file error (manually or scheduled). If error (you get a notification), then you have to get the parity to report an error, and select to repair the data drive from parity. You also need to hope your parity didn't get overwritten. I'm also not sure what happens if there's been updated data to the same block as a corrupt file. This may be an area between the advantages/disadvantages between realtime and snapshot RAID. My guess is that Unraid relies on traditional backups for "snapshots", while SnapRAID and BTRFS can use a parity algorithm to accomplish the same goal via snapshots+parity. If my guess is correct, then restoring changing data can become complicated, but simple for static data such as movie/tv. Or parity report reports an error, then you have to do a full BTRFS scrub to determine if there's a problem with your data, then correct the problem with your parity. If BTRFS (or DFI) do not report a problem, then you have to fix/rebuild your parity. I have more experience experimenting with SnapRAID, and so far its very easy and a lot more intuitive. I have very little experience with BTRFS and its been too easy to the point of black box magic. Basically with SnapRAID, and with my little BTRFS experience, you just scan for hash/checksum errors, and if error you click a button to restore. With Unraid, lots of posts about "have parity error, what do I do?", and then a bunch of hopeful analysis steps. Synology Hybrid Raid is another option that integrates BTRFS, so you can have parity + integrity + snapshots.

I think a solution can be very simple with at least SnapRAID. I don't know if Unraid is saying that if you want BTRFS integrity+parity or SnapRAID, then find another platform, or if Unraid will change their mind and begin R&D. I don't know if Unraid doesn't see the need, doesn't see the value, too costly, or doesn't want to. Unraid is great, but having the functional of SnapRAID or BTRFS would be evolutionary. You could have BOTH realtime and snapshot RAID...easily. That's what I'm doing. Unfortunately it is not integrated with a GUI, schedule, configuration, notifications, updates, support, etc. SnapRAID is very powerful and extremely simple, and probably a solution most people would want. Snapshots can be safer too since there's no data and parity changes happening during a crash/outage. You can also put your parity anywhere. You can save multiple parity snapshots too. With SnapRAID, I plan on doing some sort of 3-2-1 backup strategy. If I'm misguided, please let me know.

Any concerns using chattr on the "user" folder (with the links/inode)? Or do I need to use chattr on the physical drive? Any issues with balance, moving, or other Unraid operations?

It's possible Unraid may need to evolve their parity algorithm or integrate other algorithm choices (snapraid should be simple).

https://forums.unraid.net/search/?q=parity error https://www.reddit.com/r/unRAID/search/?q=parity errors BTRFS, ZFS, SnapRAID have presumably solved this. Instead of comparing data to parity, compare to a known presumed reliable stored hash. I don't believe Unraid has metadata to perform this logic, therefore Unraid (nor the user) knows what to do.

+ chattr I frequently swap my user share read/write permissions for additions or maintenance, then swap back. This is specially true on my photo share. Would be nice if this feature was built-in and easy to use.

My thoughts... 1) Figure out if the parity error is parity or data. Don't leave users to figure out if they should restore the data from parity, repair parity, or recover from a backup. There's tools today to automate this. 2) Snapshot RAID (like SnapRAID) is phenomenal. I'm really liking that I can simply snapshot a folder, not an entire drive. I can also do snapshots at different intervals. With snapshots, I can also undelete a file. This detects checksum errors, does a "backup" via parity calculation, saves storage for multiple drives (via parity calculation, not compression), and can do a restore of a file, even at a point of time depending on your snapshot date. You can also store your parity offline, external, or cloud. I'm surprised this technology is not more popular or a plugin. <--- This really NEEDS to become a plugin! 3) Silent error detection (bitrot or whatever causes files to change). 4) I have frequent false positive with Dynamix File Manager plugin when doing manual checks. So now I'll use the plugin to give me the error, but then manually check my other checksums for verification. I get enough false parity errors and false DFI errors that I no longer consider them credible. How do I know? Because my external BLAKE3 checksums validate, so do BTRFS scrubs. 5) BTRFS scrubs only detect a problem, but do not correct. 6) Don't like that Unraid is weak at correcting problems. 7) Want something at the file level, not block level. Unraid + SnapRAID would be a great integrated platform.

When will Duplicacy get updated to v1.6.3?