Verify Offline Backups with Checksums built from DYNAMIX FILE INTEGRITY PLUGIN

[wacko37]

I have posted on the "DYNAMIX FILE INTEGRITY PLUGIN" thread but got no answers, But I feel the scope of this question is much larger than the plugin itself which is why I'm posting here.

 

ZFS has got me asking so many questions about how I'm handling my data and utilising my hardware to the best of its ability. This has lead me down the path of using the DYNAMIX FILE INTEGRITY PLUGIN (DFI) to checksum my important data on my Unraid array and my offline backups (zfs is just not suitable for my home use case on main array)

 

I believe I fully understand how DFI works, by applying a checksum to the extend attributes of each file. Verification involves crossing checking the actual file against this checksum on a defined schedule. Any discrepancies found will be notified.

 

So....  if something gets corrupted I would need to manually replace the file from a backup. Now this is where I get stuck on how to proceed. 

 

How do I ensure my backups are not corrupted, during a  rsync from and to Unraid?

 

What I have been doing up until now is "rsync" every once in a while to an offline backup which is basically a large drive formatted in NTFS as I use only windows outside Unraid. This is done via "Unassigned Devices" "device script" and works flawlessly. But none of the backups are verified and I have no idea how to do this or proceed.

 

I'm aware that rsync has "--checksum" command but that would doubling the checksums...correct?

There is also the "--xattrs" command but will this transfer to NTFS intact and if so how would I then verify those checksums?

 

DFI has a way to export the checksums via corz compatible files but they are grouped via disks and not via shares which makes it very hard, and doing a verification elsewhere is more risk and extra work.

 

I greatly appreciate any help on how I can make this work or change altogether.

 

 

 

 

 

 

[Vr2Io]

You can ref. below post to use command line to perform different operation.

 

 

4 hours ago, wacko37 said:

How do I ensure my backups are not corrupted, during a  rsync from and to Unraid?

 

 

You can hash the destination file ( after writing finish to media and flush the ram cache ) base on the hash record, so no need hash the source again.

 

I won't use Windows to hold data file, source and backup hold in Unraid, all in same environment will be best efficiency.

 

[wacko37]

Firstly @Vr2Io thankyou for your time and expertise, (i've seen you reply to many of the questions in the DFI forum during my seach for answers)

 

Reading through the quoted posts from DFI now, thanks. (none of my search words brough me to the post mentioned)

 

I'm guessing your native language is not English but your doing a better job than me and that's all I speak....lol.

 

15 hours ago, Vr2Io said:

I won't use Windows to hold data file, source and backup hold in Unraid, all in same environment will be best efficiency.

 Are you saying NTFS cannot hold the "extended attributes" or are you recommending I don't use Windows at all?

 

I really do require this to work on NTFS, as this backup drive travels with me to work offshore and everything is Windows environment out there.

[Vr2Io]

4 hours ago, wacko37 said:

I'm guessing your native language is not English

Yes

 

4 hours ago, wacko37 said:

Are you saying NTFS cannot hold the "extended attributes"

Yes, for my understanding, NTFS can't handle extend attributes in usual way, but I may be wrong.

 

4 hours ago, wacko37 said:

are you recommending I don't use Windows at all?

I mean not recommend use Windows to handle those backup / hash job .... doing all in one environment will less headaches. Almost all my data store in Unraid and let it handle all job. Some storage set also in different file system, i.e. BTRFS and ZFS, I think this will improve resilience.

 

DFI store hash in two way, extend attributes and plain text file, if you want process file other then array, you can

 

** edit export text file, change the path content **

 

then DFI UI also serve as usual, but actually file were in different path.

 

This also my early method to handle verify / import / export for file which outside array,

 

After you learn command line, then you can write some script to handle all job.

Edited August 31, 2023 by Vr2Io

[wacko37]

1 hour ago, Vr2Io said:

Yes

 

Yes, for my understanding, NTFS can't handle extend attributes in usual way, but I may be wrong.

 

I mean not recommend use Windows to handle those backup / hash job .... doing all in one environment will less headaches. Almost all my data store in Unraid and let it handle all job. Some storage set also in different file system, i.e. BTRFS and ZFS, I think this will improve resilience.

 

DFI store hash in two way, extend attributes and plain text file, if you want process file other then array, you can

 

** edit export text file, change the path content **

 

then DFI UI also serve as usual, but actually file were in different path.

 

This also my early method to handle verify / import / export for file which outside array,

 

After you learn command line, then you can write some script to handle all job.

Ok.... I'm very close to getting this to work thanks to your help and @SantyC93 previous posts.

 

So..... good news is that NTFS can indeed retain the "extended attributes" via the ----xattrs rsync command. I created a test folder on a NTFS flash drive and manually ran the Bunker verification which completed correctly "/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -v -l -n -b3 /mnt/user/**** "

Brilliant now I know my use case will work!

 

So I have an rsync script that I use in Unassigned Devices that @dlandon helped perfect. I'm currently hacking this script to included the verification after the rsync has been complete.

I'm stuck with NO contents being provided in the logs. It creates the directory and file but no content. The "view device script log" in UD clearly shows the needed information (see attached)

 

I have also attached the script for viewing, the relevant line is 109, any input would be greatly appreciated as this is all above my knowledge base..... but I'm so close....lol

08.3-Test-Rsync.log.txt 08.3-Test-Rsync.sh

[dlandon]

This line (109) seems to have an issue with the directed output:

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -v -l -n -b3 "$dst_path" &>"$MOUNTPOINT"/bunker-logs/"$job_name"/log.`date '+%Y_%m_%d__%H_%M_%S'`.log >> $LOGFILE

 

This directs all output:

&>"$MOUNTPOINT"/bunker-logs/"$job_name"/log.`date '+%Y_%m_%d__%H_%M_%S'`.log

What are you trying do with that code?

 

Why not this?

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -v -l -n -b3 "$dst_path" >> $LOGFILE

 

All of the bunker output will go to the UD log.

[wacko37]

26 minutes ago, dlandon said:

What are you trying do with that code?

Firstly, thankyou for your input. To answer your question..... I dont really know.... But I do know what I'm trying to achieve. 

I need the full Bunker Log to be generated on the destination in a log file preferable separate to the rsync log file, exactly like rsync logs are created via this script.

 

29 minutes ago, dlandon said:

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -v -l -n -b3 "$dst_path" >> $LOGFILE

This indeed logs the verification within  UD "device Script Log" but does not generate a separate log file similar to what rsync generates. The Logs within UD is exactly what I require but in a log file.

[wacko37]

Ok, I have advanced somewhat in other areas of this script.

 

I want the ability to not Bunker verify if required, so I copied the on/off features for notifications in the script and implemented them for Bunker. And it works as desired.

 

Now to further test the script I have added more test folders of different file formats and everything is working….somewhat, But bunkers is verifying all files after each rsync folder is executed, causing more verification than is required. (hope that makes sense)

 

See logs and updated script attached.

08.4-Test-Rsync (1).sh 08.4-Test-Rsync.log.txt

[wacko37]

20 minutes ago, wacko37 said:

Now to further test the script I have added more test folders of different file formats and everything is working….somewhat, But bunkers is verifying all files after each rsync folder is executed, causing more verification than is required. (hope that makes sense)

Ok fixed that by repositioning the required code.

 

Its working but I cant help but feel like its a messy hack of a script now (probably because I don't fully know what I'm doing)

 

Still cant get the bunker log to write to the log file that it creates.

08.5-Test-Rsync.sh

[wacko37]

5 hours ago, dlandon said:

This line (109) seems to have an issue with the directed output:

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -v -l -n -b3 "$dst_path" &>"$MOUNTPOINT"/bunker-logs/"$job_name"/log.`date '+%Y_%m_%d__%H_%M_%S'`.log >> $LOGFILE

 

This directs all output:

&>"$MOUNTPOINT"/bunker-logs/"$job_name"/log.`date '+%Y_%m_%d__%H_%M_%S'`.log

What are you trying do with that code?

 

Why not this?

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -v -l -n -b3 "$dst_path" >> $LOGFILE

 

All of the bunker output will go to the UD log.

AAAHHHHH...... now I understand your post.

 

I worked out that the logs cannot be written in 2 places at once, once I removed  ">> $LOGFILE" I indeed got Content within my desired log file.

 

jesus.....so much time wasted and the answer was right there in front of me!

 

 

 

 

[wacko37]

The script is now good enough for me to use, although its definitely not perfect, its doing what I require.

 

Please use with caution as I have copied and pasted (hacked) quite a bit and edited where required to make it work. I am not trained in this field at all and its been a processes of trial and error.

 

BIG thanks to @cholzer who made the original script, it was then perfected by @dlandon to function correctly within UD some time ago.

 

Also thanks @Vr2Io@JonathanM for a push in the right direction and good advice, and lastly @bonienl for the DFI Plugin and Bunker!

 

Hope this helps someone in the future

 

Issues - 

 

1 - logs to be available/visible in both the generated log file and within the UD "device Script Log" cant get both to work simulationally

 

2 - When the generated log file is opened/viewed for the first time an error is generated in the syslog

 

UNRAID nginx: 2023/09/01 15:53:15 [error] 24436#24436: *1512682 open() "/usr/local/emhttp/plugins/dynamix.file.manager/javascript/ace/mode-log.js" failed (2: No such file or directory) while sending to client, client: [Redacted], server: [Redacted], request: "GET /plugins/dynamix.file.manager/javascript/ace/mode-log.js HTTP/2.0", host: "[Redacted]", referrer: "[Redacted] /Main/Browse?dir=%2Fmnt%2Fdisks%2FUSERS%2Fbunker-logs"
Sep  1 15:53:28 Wacko-UNRAID bunker: verified 425 files from /mnt/disks/USERS/Test. Found: 0 mismatches, 0 corruptions. Duration: 00:01:06. Average speed: 119 MB/s

                                      

0_-_TestUSBNTFS_-_SYNC--VERIFY.sh

[foo_fighter]

Thanks. Nice script.

 

A couple of  comments:

You may want to run bunker -a on the $src_path before running the rsync command.

bunker doesn't seem to really play nice with piping STDOUT to a log file. It keeps dumping "Scanning \-/-/"  (spinning bar) into the log file.

Your bunker -v command works for your case where everything is under /Test but it wouldn't work in general for different destination directories. Also it will verify Everything under /Test even though you only rsync'd /Test/Agoria and /Test/Peacock. For example if there's an existing /Test/HugeSizeDirectory, it would also try to verify that.