Dynamix File Integrity plugin

[SantyC93]

After looking for a bit in the Bunker forum page I couldn't really find the solution I wanted (I'd like to run the FileIntegrity "automatic check" so that it uses my task settings and excluded folders), but I think I got something close enough.

 

This command to check the folder I need and export (one script for each folder):

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -A -l -f /mnt/user/logs/Folder1build.log -b3 /mnt/user/Folder1

 

And this one to verify them:

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -v -l -f /mnt/user/logs/Folder1check.log -n -b3 /mnt/user/Folder1

 

Just wanted to ask if there's somithing wrong with them since I'm pretty inexperienced with this stuff, for example the command does not save any log in the folder I specified.

 

If anyone knows what I did wrong and can help me fix this error or can tell me if I made any more mistakes I would be really grateful, ty

Edited April 7, 2023 by SantyC93
spelling

[Vr2Io]

11 hours ago, SantyC93 said:

After looking for a bit in the Bunker forum page I couldn't really find the solution I wanted (I'd like to run the FileIntegrity "automatic check" so that it uses my task settings and excluded folders), but I think I got something close enough.

 

This command to check the folder I need and export (one script for each folder):

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -A -l -f /mnt/user/logs/Folder1build.log -b3 /mnt/user/Folder1

 

And this one to verify them:

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -v -l -f /mnt/user/logs/Folder1check.log -n -b3 /mnt/user/Folder1

 

Just wanted to ask if there's somithing wrong with them since I'm pretty inexperienced with this stuff, for example the command does not save any log in the folder I specified.

 

If anyone knows what I did wrong and can help me fix this error or can tell me if I made any more mistakes I would be really grateful, ty

 

"-f" not use to export log, it use to specify where are the hash file. 

[SantyC93]

4 hours ago, Vr2Io said:

 

"-f" not use to export log, it use to specify where are the hash file. 

 

What would be the correct way to export the logs? Could you make an example pls?

Tyvm

[Vr2Io]

5 hours ago, SantyC93 said:

 

What would be the correct way to export the logs? Could you make an example pls?

Tyvm

 

You could export the result ( not detail log ) to a file, by add "&>test.log" at command ending.

 

cat test.log

Finished - verified 6 files, skipped 0 files. Found: 0 mismatches, 0 corruptions. Duration: 00:00:09. Average speed: 2.48 GB/s

 

Edited April 8, 2023 by Vr2Io

[SantyC93]

8 minutes ago, Vr2Io said:

 

You could export the result ( not detail log ) to a file, by add "&>test.log" at command ending.

 

cat test.log

Finished - verified 6 files, skipped 0 files. Found: 0 mismatches, 0 corruptions. Duration: 00:00:09. Average speed: 2.48 GB/s

 

 

That could be useful, but logs are necessary in the event of a corrupted file. If there's no way to export the logs I guess I can't use a script to fix my problem. Ty anyway

[Vr2Io]

47 minutes ago, SantyC93 said:

 

That could be useful, but logs are necessary in the event of a corrupted file. If there's no way to export the logs I guess I can't use a script to fix my problem. Ty anyway

If file corrupt, it should show out the details in output.

Edited April 8, 2023 by Vr2Io

[SantyC93]

7 hours ago, Vr2Io said:

If file corrupt, it should show out the details in output.

 

Ty! Worked great

 

After a bit of tests I ended up with this script:

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -v -l -n -b3 /mnt/user/Folder1 &>"Folder1_Verify_log_$(date +"%Y_%m_%d_%I_%M_%p").log"

/usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Bunker Verify" -d "Folder1 Verify complete" -i "normal"

mv "/root/Folder1_Verify_log_"* "/mnt/user/logs/"

 

it works perfectly when I manually run it, but if I run it in background or if I schedule the script the .log file does not get generated

Any idea on what could be causing the issue?

Edited April 8, 2023 by SantyC93

[Vr2Io]

2 hours ago, SantyC93 said:

but if I run it in background or if I schedule the script the .log file does not get generated

No such problem, does job still in running and not finish ?

 

[SantyC93]

7 hours ago, Vr2Io said:

No such problem, does job still in running and not finish ?

 

I'm sure the script finished (I tried on a very small folder so it takes few seconds).

I tried running it many times and the log file does not get generated if the script runs in the background.

Tried googling but can't find anything about this problem.

Edited April 9, 2023 by SantyC93

[Vr2Io]

You should set the full path for redirection

i.e. "/Folder1/Verify_log_$(date +"%Y_%m_%d_%I_%M_%p").log"

Edited April 9, 2023 by Vr2Io

[SantyC93]

1 hour ago, Vr2Io said:

You should set the full path for redirection

i.e. "/Folder1/Verify_log_$(date +"%Y_%m_%d_%I_%M_%p").log"

 

Adding the path fixed the problem

Tyvm!

 

This is the working script:

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -v -l -n -b3 /mnt/user/Folder1 &>"/mnt/user/logs/Folder1_Verify_log_$(date +"%Y_%m_%d_%I_%M_%p").log"
/usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Bunker Verify" -d "Folder1 Verify complete" -i "normal"

 

Edited April 9, 2023 by SantyC93

[Gray J]

Is the program actively ignoring excluded shares for anyone else?
I mean it is still checking shares that have been marked as excluded.

[Vr2Io]

1 hour ago, Gray J said:

Is the program actively ignoring excluded shares for anyone else?
I mean it is still checking shares that have been marked as excluded.

Exclude only means on add / build hash. If exclude path file's have hash attribute or the hash record file have it, then it will verify too.

You can use "clear" function to clear the hash attribute in exclude path.

Edited April 11, 2023 by Vr2Io

[SantyC93]

On 4/9/2023 at 11:00 PM, SantyC93 said:

 

Adding the path fixed the problem

Tyvm!

 

This is the working script:

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -v -l -n -b3 /mnt/user/Folder1 &>"/mnt/user/logs/Folder1_Verify_log_$(date +"%Y_%m_%d_%I_%M_%p").log"
/usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Bunker Verify" -d "Folder1 Verify complete" -i "normal"

 

 

Update:

Tested the script on a large folder and the exported log file exceeded 300MB, causing the log to only show this error message:

<b>Fatal error</b>:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 355982752 bytes) in <b>/usr/local/emhttp/plugins/dynamix.file.manager/include/Control.php</b> on line <b>121</b><br />

 

It seems to be caused by the PHP memory limit, which is set to 128MB by default.
Googling the problem I found many post on StackOverflow suggesting to increse the memory limit to solve the problem.

 

Anyone knows if it is possible to change the PHP memory limit in Unraid? And would it be a bad idea to increase it to 512MB?

Are there other ways to fix this problem without increasing the PHP memory limit?

 

Ty very much for the help

[T0rqueWr3nch]

On 3/19/2023 at 6:16 AM, bonienl said:

 

I don't see this happening on my system and need to investigate.

Currently busy with the Unraid 6.12 release, but will look into this later.

 

@bonienlI ran into the same problem when I attempted to implement this plugin with the "automatic" functionality today.

 

I believe I have figured out the problem (I actually found two problems; one I've already submitted the PR for) and I have the automatic functionality working as expected on my Unraid development server as we speak. I just wanted to do some further testing before I submit the additional PR.

 

tl;dr: Focus on your 6.12 release. You can take this off your mind. I'll submit the PR for your review. We appreciate your work!

[T0rqueWr3nch]

Pull requests in:

• Implement auto-hashing for cache-to-disk moves by Torqu3Wr3nch · Pull Request #79 · bergware/dynamix (github.com)
• Fix Automatic Checksums - Don't exclude everything when nothing is set to be excluded. by Torqu3Wr3nch · Pull Request #81 · bergware/dynamix (github.com)

[bonienl]

33 minutes ago, T0rqueWr3nch said:

Pull requests in:

Thanks, I will look into this when time permits.

Currently the focus is on Unraid 6.12, which should be released "soon"

 

[cpxazn]

Whenever I drag a folder into VLC to queue a playlist, for some reason it triggers rehashing for all files in that directory. I've seen at least 20 b3sum processes get kicked off by this which kills server performance. Is there a reason why the plugin thinks the files have been modified?

 

EDIT: This is with automatic protection enabled. b3sum processes will run even when parity check is running.

Edited May 17, 2023 by cpxazn

[T0rqueWr3nch]

On 5/17/2023 at 9:52 AM, cpxazn said:

Whenever I drag a folder into VLC to queue a playlist, for some reason it triggers rehashing for all files in that directory. I've seen at least 20 b3sum processes get kicked off by this which kills server performance. Is there a reason why the plugin thinks the files have been modified?

 

EDIT: This is with automatic protection enabled. b3sum processes will run even when parity check is running.

 

We're really only watching for a close_write, so the fact that you're seeing this implies that VLC is opening the file like it intended to write to it.

 

My bet is that it's related to this: VLC drop linux FS-Cache when playing (#27848) · Issues · VideoLAN / VLC · GitLab. Specifically this line: 

Quote

I found problem: taglib open file for writing when read metadata.

 

Try updating VLC and let us know how that goes.

 

Update (5/23/23): It doesn't appear this is fixed until 3.0.19 (current release is 3.0.18), so you may have to wait for VLC to release 3.0.19 or seek out an alpha/beta version.

Edited May 23, 2023 by T0rqueWr3nch

[Hobbes_Is_Real]

I just installed unRaid for the first time this last weekend.  Bitflips is a concern of mine.  Considering the warning at the top of the first post, is this safe to use with Btrfs encrypted?  I did read through at least a 1/3 of the 47 pages of this thread, but I am just want to make sure before I use it.

[itimpi]

3 hours ago, Hobbes_Is_Real said:

I just installed unRaid for the first time this last weekend.  Bitflips is a concern of mine.  Considering the warning at the top of the first post, is this safe to use with Btrfs encrypted?  I did read through at least a 1/3 of the 47 pages of this thread, but I am just want to make sure before I use it.

If you are using btrfs then you may not need this plugin as it has built-in checksumming for detection of bitrot.

[OmegaXis]

I have 3 disks that always refuse to export properly, even though the plugin says otherwise.  Is there something I am missing?

[isvein]

On 6/4/2023 at 2:06 AM, itimpi said:

If you are using btrfs then you may not need this plugin as it has built-in checksumming for detection of bitrot.

I wonder the same thing, but for an ZFS formatted array

[itimpi]

12 minutes ago, isvein said:

I wonder the same thing, but for an ZFS formatted array

 

In terms of having them as array drives then ZFS and BTRFS offer very similar capabilities. in protecting against bitrot.   An unresolved question at the moment is whether ZFS or BTRFS will prove more resilient in such a configuration if hardware problems are encountered, and if performance is the same (I expect this is likely).

[isvein]

So this plugin wont add much on zfs formatted array drives as it has the scrub and checksum already?
and if a drive goes down, we always have good old parity as usual