Laurian Posted August 3, 2018 Share Posted August 3, 2018 I would love a different hashing algorithm: XXH64 https://github.com/Cyan4973/xxHash It is not only faster but that data is compatible with https://mediahashlist.org/mhl-specification/ (yes, that spec does MD5 too, but other tools using that spec use XXH64) Actually MHL file support would be useful too, to test if the files were not corrupted on copy to unRAID or to other remote disks. On macOS I use https://hedge.video/ for my media transfer and it creates such files. In the meantime I'll see what I can do with custom scripts and https://github.com/pomfort/mhl-tool Quote Link to comment
Vr2Io Posted August 3, 2018 Share Posted August 3, 2018 (edited) running at RAM speed limits ？I dont belive, even crc32 crc64 blake2 .... cant (base on not multithread) There are other factor limit the actual speed. Edited August 3, 2018 by Benson Quote Link to comment
fkick Posted August 13, 2018 Share Posted August 13, 2018 Hi When "Automatically protect new and modified files" is turned on, is the entire drive that the new file is added to scanned and recalculated, or just the new/modified file? Also, I'm noticing that the "build up-to-date" on a few of my drives is showing "O" even with the auto protect option turned on. Thanks! Quote Link to comment
Vr2Io Posted August 14, 2018 Share Posted August 14, 2018 12 hours ago, fkick said: Hi When "Automatically protect new and modified files" is turned on, is the entire drive that the new file is added to scanned and recalculated, or just the new/modified file? Also, I'm noticing that the "build up-to-date" on a few of my drives is showing "O" even with the auto protect option turned on. Thanks! Just the new/modified file hash for all array disks. For showing "O" fixing, select the disks and build. Quote Link to comment
fkick Posted August 14, 2018 Share Posted August 14, 2018 10 hours ago, Benson said: Just the new/modified file hash for all array disks. For showing "O" fixing, select the disks and build. Thanks Benson. It seems every day I'm getting the "O" on random drives in the array, even if no new files are added, and I'm getting a lot of reads on drives for hours in the background since enabling the "auto". Is there a way to see what the plugin is doing when run automatically in the gui? I know I can see manually run operations but can't see anything in the log regarding an auto run. Thanks Quote Link to comment
Vr2Io Posted August 14, 2018 Share Posted August 14, 2018 3 hours ago, fkick said: Thanks Benson. It seems every day I'm getting the "O" on random drives in the array, even if no new files are added, and I'm getting a lot of reads on drives for hours in the background since enabling the "auto". Is there a way to see what the plugin is doing when run automatically in the gui? I know I can see manually run operations but can't see anything in the log regarding an auto run. Thanks You may monitor "bunker" process for know more FIP doing what and other process such as "md5sum" "sha1sum" "b2sum" etc. Quote Link to comment
fkick Posted August 14, 2018 Share Posted August 14, 2018 1 hour ago, Benson said: You may monitor "bunker" process for know more FIP doing what and other process such as "md5sum" "sha1sum" "b2sum" etc. Thanks, Looking at my Processes that are running, I'm seeing Bunker scanning files and inotifywait running with the following: root 32155 21440 0 11:30 ? 00:00:00 /bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -Aq1 -b2 -f /boot/config/plugins/dynamix.file.integrity/saved/disks.export.20180814.new.hash /mnt/disk1/TV/File1.m4v root 32173 32155 3 11:30 ? 00:00:01 /usr/bin/b2sum /mnt/disk1/TV/File1.m4v root 32392 21440 0 11:30 ? 00:00:00 /bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -Aq1 -b2 -f /boot/config/plugins/dynamix.file.integrity/saved/disks.export.20180814.new.hash /mnt/disk1/TV/File2.m4v root 32723 21440 0 11:30 ? 00:00:00 /bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -Aq1 -b2 -f /boot/config/plugins/dynamix.file.integrity/saved/disks.export.20180814.new.hash /mnt/disk1/TV/File2.m4v root 32744 32723 3 11:30 ? 00:00:01 /usr/bin/b2sum /mnt/disk1/TV/File3.m4v root 19889 2 0 11:31 ? 00:00:00 [kworker/u8:1] root 21439 1 0 Aug13 ? 00:00:05 inotifywait -dsrqo /var/run/hash.pipe -e close_write --exclude ^/mnt/disk[0-9]+/(.*docker\.img$|.*\.AppleDB/|.*\.tmp$|.*\.nfo$|.*\.temp$|.*\.itdb$|.*\.DS_Store$|\.Trash\-99/|docker/|.*\.AppleDB/|.*\.DS_Store$) --format %w%f /mnt/disk1 /mnt/disk2 /mnt/disk3 /mnt/disk4 /mnt/disk5 /mnt/disk6 /mnt/disk7 /mnt/disk8 root 21440 1 0 Aug13 ? 00:01:25 /bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/watcher A -b2 (.*docker\.img$|.*\.AppleDB/|.*\.tmp$|.*\.nfo$|.*\.temp$|.*\.itdb$|.*\.DS_Store$|\.Trash\-99/|docker/|.*\.AppleDB/|.*\.DS_Store$) /mnt/disk1 /mnt/disk2 /mnt/disk3 /mnt/disk4 /mnt/disk5 /mnt/disk6 /mnt/disk7 /mnt/disk8 root 22768 21440 0 11:32 ? 00:00:00 /bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -Aq1 -b2 -f /boot/config/plugins/dynamix.file.integrity/saved/disks.export.20180814.new.hash /mnt/disk1/TV/File4.m4v root 22789 22768 3 11:32 ? 00:00:04 /usr/bin/b2sum /mnt/disk1/TV/File4.m4v root 23381 2 0 11:27 ? 00:00:00 [kworker/1:2] root 23931 2 0 11:27 ? 00:00:00 [kworker/3:2] root 24814 15807 0 11:19 ? 00:00:00 [afpd] And if I look at the integrity tab, it is showing Drive 1 as Up-todate though Bunker appears to be parsing it (with drive 6 and 7 not up to date, even though they were about 30 minutes ago and no media has been added or modified). See attached Thanks for your help! Quote Link to comment
fkick Posted August 15, 2018 Share Posted August 15, 2018 After doing some more testing, it looks like the setting "Save new hashing results to flash" is what is causing the the system to rescan every file every day, rather then just updating the previously exported hashes. Once I disabled this setting, I'm no longer getting the hours of reads on my drives. Looks like I'll just need to export my hashes manually if I want them on the flash key. Thanks! Quote Link to comment
Vr2Io Posted August 15, 2018 Share Posted August 15, 2018 (edited) Turn off some automatic option may solve some problem. I am not quite agree your founding, anyway I turn off all, so can't comment. ?? Edited August 15, 2018 by Benson Quote Link to comment
fkick Posted August 16, 2018 Share Posted August 16, 2018 It looks like I was mistaken with the "Save new hashing results to flash" being the cause of my issue. I woke this am to some drive over heating warnings as this plugin was again walking through every file on my array, even though all my builds were showing up to date (and the disk it was currently walking through had no media added to it). I did still have "Automatically protect new and modified files on", however, if this is supposed to only generate and check for hashes on new content, does this need to scan of every item in the array? Quote Link to comment
Vr2Io Posted August 16, 2018 Share Posted August 16, 2018 (edited) Previous reply should't all right, if new file write, then , if other file in same directory haven't hash will hash back. So, pls build for all disk, otherwise you will got disk busy some time. Besides, some member also report if mass file(s) writing, it also got trouble and can't avoid. For me, I turn off all auto, I usually not write much file in a time but with large size. I dont want FIP slow down the speed, so manual suitable for me. Edited August 16, 2018 by Benson Quote Link to comment
superderpbro Posted August 18, 2018 Share Posted August 18, 2018 Been using this for a while and curious. What happens when a file is renamed? Does it change the hash or make a new one? How does it know the difference between a file name change or say a text file edit and file corruption (since the hash would change if you edited a txt file)? Thanks :P Quote Link to comment
John_M Posted August 22, 2018 Share Posted August 22, 2018 On 8/18/2018 at 6:44 AM, superderpbro said: Been using this for a while and curious. What happens when a file is renamed? Does it change the hash or make a new one? How does it know the difference between a file name change or say a text file edit and file corruption (since the hash would change if you edited a txt file)? Since a file's name is separate from its content I would expect nothing to be done when the name is changed. The content remains unaltered; therefore so does the hash. The name is part of the file's metadata and editing it actually affects the parent directory. So if directories were hashed as well as files, then it's the directory that contains the file that would have to be rehashed. 1 Quote Link to comment
Maticks Posted September 1, 2018 Share Posted September 1, 2018 Thank you guys for making this plugin. Had two corrupted files on a hard drive that threw some read errors a while back. Discovered the two corrupted files and restored from another machine, works perfectly. Quote Link to comment
Vr2Io Posted September 1, 2018 Share Posted September 1, 2018 (edited) 4 hours ago, Maticks said: Thank you guys for making this plugin. Had two corrupted files on a hard drive that threw some read errors a while back. Discovered the two corrupted files and restored from another machine, works perfectly. Me too (Thanks), longtime haven't disk problem ( disk read error, but haven't drop disk out ), this tool help me identify which file corrupt. BTW, "checksum suit" plugin also can do that and I parallel run with FIP. I odd unRAID won't correction the read out from parity/array. Aug 31 11:45:32 kernel: md: disk3 read error, sector=11368765680 Aug 31 11:45:32 kernel: md: disk3 read error, sector=11368765688 Aug 31 11:45:32 kernel: BTRFS warning (device md3): csum failed root 5 ino 22234 off 6193057792 csum 0xddf17fc0 expected csum 0xff585686 mirror 1 Aug 31 11:45:35 kernel: sd 10:0:2:0: [sde] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 Aug 31 11:45:35 kernel: sd 10:0:2:0: [sde] tag#0 Sense Key : 0x3 [current] [descriptor] Aug 31 11:45:35 kernel: sd 10:0:2:0: [sde] tag#0 ASC=0x11 ASCQ=0x0 Aug 31 11:45:35 kernel: sd 10:0:2:0: [sde] tag#0 CDB: opcode=0x88 88 00 00 00 00 02 a5 a1 a9 40 00 00 04 00 00 00 Aug 31 11:45:35 kernel: print_req_error: critical medium error, dev sde, sector 11368770584 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770632 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770640 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770648 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770656 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770664 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770672 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770680 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770688 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770696 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770704 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770712 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770720 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770728 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770736 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770744 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770752 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770760 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770768 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770776 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770784 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770792 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770800 Aug 31 11:45:35 kernel: md: disk3 read error, sector=11368770808 Aug 31 11:45:35 kernel: BTRFS warning (device md3): csum failed root 5 ino 22234 off 6196539392 csum 0x64d97971 expected csum 0x2ec1207e mirror 1 Aug 31 11:45:36 kernel: BTRFS warning (device md3): csum failed root 5 ino 22234 off 6183276544 csum 0x335a47b9 expected csum 0x2be09287 mirror 1 Aug 31 11:45:36 kernel: BTRFS warning (device md3): csum failed root 5 ino 22234 off 6183276544 csum 0x335a47b9 expected csum 0x2be09287 mirror 1 Aug 31 11:45:36 kernel: BTRFS warning (device md3): csum failed root 5 ino 22234 off 6183276544 csum 0x335a47b9 expected csum 0x2be09287 mirror 1 Edited September 1, 2018 by Benson Quote Link to comment
themaxxz Posted September 15, 2018 Share Posted September 15, 2018 Hi, I'd like to report a 'bug' of some sort. After skimming this thread, I thought this plugin could be usefull, so i installed it today on my unraid server (6.5.3). I decided to start slow and perform a disk by disk build. I first did a build on disk13, which only had 2 files. After that completed I started a build on disk3 with ~500 files. Currently still ongoing. (1 bunker process running related to disk3) I suddenly noticed the UI displayed a green check for all but one disk (disk9) under 'Build up-to-date'? I tracked back that it's populated from the file /boot/config/plugins/dynamix.file.integrity/disks.ini. After manually correcting (removing the not build disks) this file, the status in the UI reflects the correct situation. While trying to understand how this could happen, I was able to reproduce the issue, by clicking on the 'hash files' link on Tools/Integrity page. After clicking on this link all but one disk9 were re-added again to disks.ini file. What should be the correct format of this disks.ini file? Quote Link to comment
ChaOConnor Posted October 7, 2018 Share Posted October 7, 2018 First off, thanks for this pluggin. I installed it because I have a lot of photos on my unRaid share and I noticed a few of them weren't opening properly. I suspected maybe the files were corrupt, so a bit late to the party, I installed the pluggin and ran a build to establish a baseline. After a week, I got a notification there were like 400 errors found using BLAKE2. My question is if you have a good hash for what a file should be, can you use that to recreate what the file and restore the original? If so, how would you go about this? If you can't do what I described above, what can you do besides restore from a backup? How should I be using this pluggin to protect files on the server verse an off-server backup? Thanks! Quote Link to comment
itimpi Posted October 8, 2018 Share Posted October 8, 2018 The checksum does not provide any capability to correct errors. The idea is that it identifies files that appear to be corrupt so that they can be restored from backups. Quote Link to comment
jammer Posted October 27, 2018 Share Posted October 27, 2018 Hi there, This is my first post as I started to trial Unraid a few weeks ago. Thanks for this plugin. I'm encountering some weirdness that I hope the community can help me understand. I am notified of SHA256 hash errors after the daily integrity check after backing up photos from a Windows PC over SMB. I enabled "Automatically protect new files". This is in the log: SHA256 hash key mismatch, is corrupted SHA256 hash key mismatch, /Jimmy Samsung/20160518_215300.jpg is corrupted SHA256 hash key mismatch, bbean cruise/00013.MTS is corrupted SHA256 hash key mismatch, ion/IMG_0408.JPG is corrupted SHA256 hash key mismatch, ion/IMG_0815.JPG is corrupted SHA256 hash key mismatch, thering at Janice/DSC_8135.JPG is corrupted SHA256 hash key mismatch, up shots/DSC_7719.JPG is corrupted SMART attributes are good (two new WD Red 4TB). Parity check (no auto correct) is fine too. When manually checking SHA256 hash values for the above files on Unraid and of the source files, they come back identical. Subsequent integrity checks show the same errors so it doesn't seem like a one-time read error. Questions: 1. What could cause the plugin to report these as errors even though the files seem to be written correctly? 2. Is it normal for the file paths to be cut off? I had to search for the names of the files to find their paths. 3. What could cause the file name to be missing entirely (i.e. the first entry)? Any help would be much appreciated! Quote Link to comment
jserio Posted November 6, 2018 Share Posted November 6, 2018 Running Unraid 6.6.3. I have my VM (Windows 10) set up with GPU and USB card passthrough. Also, the OS is running on a SATA controller which is also passed through. Everything works beautiful. Except when I enable File Integrity Control (in Settings). I get terrible mouse lag/stutter every few seconds. I tried with both SHA2 and Blake2 and the problem persists. Anyone else have this problem? Any ideas? Quote Link to comment
mbc0 Posted November 6, 2018 Share Posted November 6, 2018 What client are you using to connect to the VM? Quote Link to comment
jammer Posted November 15, 2018 Share Posted November 15, 2018 On 7/17/2018 at 9:16 PM, _Shorty said: Frankly, I don't think this thing can be trusted to do what it is intended to do. If it's flagging even one file as being corrupt, as in my case, when other file hashing tools come back and say both copies actually have the same hash then there's a problem somewhere with this tool. I wouldn't necessarily put too much stock into it saying you've got 295 corrupted files. If you just replaced the drive and consequently things were just rebuilt from parity I would be more inclined to think your data is just fine. I believe I'm experiencing the same issue as _Shorty. I tested by making a copy of a directory onto the same drive. Got hash errors on some files but the files are identical to the source (used other hashing tools and a binary compare using cmp). Did a hash export from the Dynamix Plugin and the stored hashes were indeed different from each other. _Shorty, did you end up finding a solution or alternative for file integrity? Anyone else? Quote Link to comment
Socrates Posted November 16, 2018 Share Posted November 16, 2018 I am sorry I am late to the party, but can someone share their settings for this plugin? I have installed it, but would like to set it up the ideal way. I am would like to get notified for any file corruption through bitrot or related before hand. Quote Link to comment
_Shorty Posted November 17, 2018 Share Posted November 17, 2018 Jammer, no, I haven't even bothered looking. It would be a nice additional safety net, for sure. But with the Dynamix one acting flaky I just turned it off. Not much point when it isn't properly doing what it should be. I haven't looked to see if there are any alternatives that perform a similar job. Quote Link to comment
bonienl Posted November 17, 2018 Author Share Posted November 17, 2018 19 minutes ago, _Shorty said: Not much point when it isn't properly doing what it should be. It works perfect for me. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.