Dynamix File Integrity plugin


bonienl

Recommended Posts

Thanks for your observation.

 

I have seen in the past that RFS can have issues in V6 when writing extended attributes and once in a blue moon completely hangs. I am going to put a BIG warning that using this plugin in combination with RFS may lead to instability.

 

WARNING: USING THIS PLUGIN ON DISKS FORMATTED IN REISERFS MAY LEAD TO SYSTEM INSTABILITY. IT IS ADVISED TO USE XFS.

OK thanks, good to have an explanation for the behaviour.

Link to comment

Wow, that was close...I installed the plugin yesterday but did not activate any checksum generation. Looks like with 3 of my drives being ReiserFS still I better won't :P

 

There are chances it works properly, but nothing guaranteed. To play save I recommend to use it only on XFS formatted disks.

 

Link to comment

WARNING: USING THIS PLUGIN ON DISKS FORMATTED IN REISERFS MAY LEAD TO SYSTEM INSTABILITY. IT IS ADVISED TO USE XFS.

 

POO!  My Primary/low power server is still on REIERFS... was hoping to keep it REIERFS as the Backup Server uses XFS and it is nice knowing that the stable REIERFS recovery tools exist.  Perhaps THIS is what locked up the server and not the 100% CPU utilization when trying 3 disks at once?  Any risk of DATA loss with REIERFS now? 

 

I just finished triple pre-clearing another 4TB HGST for the hoped for dual parity.

 

Would the advised course of action be

1 - Disable automatic checking

2 - Switch out each disks filesystem to XFS as per http://lime-technology.com/forum/index.php?topic=37490.msg346739#msg346739  ... only using unBalance rather than screen rsync http://lime-technology.com/forum/index.php?topic=39707.0

3 - Remove empty disk from the array

4 - New Config 

5 - Build Fresh Parity

6 - Redo the Initial Integrity Build Per disk

 

Thanks

Link to comment

Nope... Sorry for the poorly worded panic posting.

 

I ran the File Integrity Plugin on a machine with all RFS disks

 

1 - Should I kill the plugin, convert to XFS, then rerun the plugin?

 

2 - Any risk of Data Loss on the RFS disks now that the plugin has been run?

Link to comment

Nope... Sorry for the poorly worded panic posting.

 

I ran the File Integrity Plugin on a machine with all RFS disks

 

1 - Should I kill the plugin, convert to XFS, then rerun the plugin?

 

2 - Any risk of Data Loss on the RFS disks now that the plugin has been run?

Whenever you convert from one filesystem to another, the disk needs to be reformatted. In that process you will loose the information stored in the extended attributes, as it goes with the filesystem.

 

The plugin doesn't touch the actual files, and running the plugin doesn't pose a risk to your data.

 

When the GUI hangs due to extended attributes not properly written under RFS, look for the process "setfattr" and do killall setfattr.

 

Link to comment

...

Would the advised course of action be

1 - Disable automatic checking

2 - Switch out each disks filesystem to XFS as per http://lime-technology.com/forum/index.php?topic=37490.msg346739#msg346739  ... only using unBalance rather than screen rsync http://lime-technology.com/forum/index.php?topic=39707.0

3 - Remove empty disk from the array

4 - New Config 

5 - Build Fresh Parity

6 - Redo the Initial Integrity Build Per disk

 

Thanks

3,4,5 don't necessarily have anything to do with converting depending on your specific situation. I was able to convert all my data disks to XFS without removing any drives or rebuilding parity just by moving things off each drive and onto others with free space, then formatting the empty, then repeating with others. There is a lot in that first thread you linked.
Link to comment

Whenever you convert from one filesystem to another, the disk needs to be reformatted. In that process you will loose the information stored in the extended attributes, as it goes with the filesystem.

 

The plugin doesn't touch the actual files, and running the plugin doesn't pose a risk to your data.

 

When the GUI hangs due to extended attributes not properly written under RFS, look for the process "setfattr" and do killall setfattr.

 

Fantastic Info here - also clears up a bit of ignorance on my part

 

SMALL feature request for future bumps... a flag indicating if an initial build and a flag indicating an export has been accomplished per disk.

 

Good idea, but I need to work out how to do this in a reliable way.

 

perhaps write a CSV file to the flash that is read each time the gui page is accessed?

Disk#, InitialBuild, InitialExport

 

...

Would the advised course of action be

2 - Switch out each disks filesystem to XFS as per http://lime-technology.com/forum/index.php?topic=37490.msg346739#msg346739  ... only using unBalance rather than screen rsync http://lime-technology.com/forum/index.php?topic=39707.0

3 - Remove empty disk from the array

4 - New Config 

5 - Build Fresh Parity

3,4,5 don't necessarily have anything to do with converting depending on your specific situation. I was able to convert all my data disks to XFS without removing any drives or rebuilding parity just by moving things off each drive and onto others with free space, then formatting the empty, then repeating with others. There is a lot in that first thread you linked.

 

Cheers, I am going through these links now!  LOTS of info here!

 

I do not have enough room on the current array to free up a disk

I will need to add a fresh XFS disk to the array in order to move files

I do not want to leave the new disk in the array as it will have a home as a 2nd parity drive (Hopefully soon).

 

Link to comment

SMALL feature request for future bumps... a flag indicating if an initial build and a flag indicating an export has been accomplished per disk.

 

Have created the following, see the attached picture.

 

For all disks it will show whether an initial build and/or an initial export was done. A green check means yes, while an orange circle means no.

 

Useful ?

disk-status.png.447962ef022eff1bec7fd0247ff7710f.png

Link to comment

SMALL feature request for future bumps... a flag indicating if an initial build and a flag indicating an export has been accomplished per disk.

 

Have created the following, see the attached picture.

 

For all disks it will show whether an initial build and/or an initial export was done. A green check means yes, while an orange circle means no.

 

Useful ?

 

Yes, I think it looks nice and adds value.

Link to comment

For those interested, version 2016.01.06a is available now, see release notes.

 

Just updated and saw that my disks have the orange circle. To clarify, will they always be a circle upon a refresh reboot of the server?

 

No, once a build or export is done, this will change into a check symbol. This information is kept on flash in the file /boot/config/plugins/dynamix.file.integrity/disks.ini

 

Mine looks as follows:

disk11=build,export
disk3=build,export
disk2=build,export
disk8=build,export
disk9=build,export
disk4=build,export
disk10=build,export
disk6=build,export
disk5=build,export
disk7=build,export
disk1=build,export

You can "trick" the system by creating/editing this file yourself. E.g. when you know all disks are built and exported, create the file with the proper entries.

 

 

Link to comment

I'm getting checksum errors reported on Macintosh .DS_Store files. It seems possible that when I add files to a folder on my unRAID share using the OS X Finder the .DS_Store file within that folder is being updated but its checksum isn't, which results in the file corruption messages.

 

unRAID file corruption: 07-01-2016 09:49
Notice [LAPULAPU] - bunker check command
Found 6 files with BLAKE2 hash key corruption

 

Syslog snippet:

 

Jan  7 03:52:47 Lapulapu bunker: warning: is missing
Jan  7 03:52:48 Lapulapu bunker: warning: is missing
Jan  7 03:58:18 Lapulapu bunker: warning: is missing
Jan  7 03:59:04 Lapulapu bunker: warning: is missing
Jan  7 04:24:40 Lapulapu bunker: warning: is missing
Jan  7 04:25:03 Lapulapu bunker: warning: is missing
Jan  7 05:48:36 Lapulapu bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/L_Public/.DS_Store is corrupted
Jan  7 05:48:36 Lapulapu bunker: warning: is missing
Jan  7 05:48:36 Lapulapu bunker: warning: is missing
Jan  7 05:52:11 Lapulapu bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/L_Public/Shared Videos/.DS_Store is corrupted
Jan  7 05:53:22 Lapulapu bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/L_Public/Shared Videos/DVD Rips/.DS_Store is corrupted
Jan  7 06:53:48 Lapulapu bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/L_Public/Shared Videos/DVD Rips/3 MKVs/.DS_Store is corrupted
Jan  7 07:08:10 Lapulapu kernel: mdcmd (68): spindown 0
Jan  7 07:08:11 Lapulapu kernel: mdcmd (69): spindown 2
Jan  7 08:12:44 Lapulapu bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/L_Public/Shared Videos/EyeTV Archive/.DS_Store is corrupted
Jan  7 09:15:00 Lapulapu emhttp: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin checkall
Jan  7 09:45:16 Lapulapu bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/L_Public/Shared Videos/MakeMKV/.DS_Store is corrupted
Jan  7 09:49:16 Lapulapu bunker: checked 49978 files from source /boot/config/plugins/dynamix.file.integrity/disk1.export.hash. Found: 0 mismatches (updated), 6 corruptions. Duration: 06:36:46 Average speed: 118. MB/s

 

Perhaps the lines containing "is missing" are a clue.

 

Is there a way of telling the plugin to exclude .DS_Store files?

Link to comment

Over the last few weeks my server has been getting random kernel warnings. I have been trying to figure out why they are happening all of the sudden and I think they have something to do with this plugin. After updating to the latest plugin I started running the initial build and export on both of my disks. This kernel warning happened right after I started that process. Does any of this log mean anything too you?

 

Jan  6 21:55:02 Pithos kernel: ------------[ cut here ]------------
Jan  6 21:55:02 Pithos kernel: WARNING: CPU: 3 PID: 24488 at arch/x86/kernel/cpu/perf_event_intel_ds.c:315 reserve_ds_buffers+0x10e/0x347()
Jan  6 21:55:02 Pithos kernel: alloc_bts_buffer: BTS buffer allocation failure
Jan  6 21:55:02 Pithos kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables kvm_intel kvm vhost_net vhost macvtap macvlan tun xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod ahci r8169 i2c_i801 libahci mii
Jan  6 21:55:02 Pithos kernel: CPU: 3 PID: 24488 Comm: qemu-system-x86 Not tainted 4.1.13-unRAID #1
Jan  6 21:55:02 Pithos kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H77 Pro4-M, BIOS P2.00 08/06/2013
Jan  6 21:55:02 Pithos kernel: 0000000000000009 ffff880357b039a8 ffffffff815f12b0 0000000000000000
Jan  6 21:55:02 Pithos kernel: ffff880357b039f8 ffff880357b039e8 ffffffff8104775b ffff88044f395ec0
Jan  6 21:55:02 Pithos kernel: ffffffff8101fd9b 0000000000000000 0000000000000000 0000000000010e10
Jan  6 21:55:02 Pithos kernel: Call Trace:
Jan  6 21:55:02 Pithos kernel: [<ffffffff815f12b0>] dump_stack+0x4c/0x6e
Jan  6 21:55:02 Pithos kernel: [<ffffffff8104775b>] warn_slowpath_common+0x97/0xb1
Jan  6 21:55:02 Pithos kernel: [<ffffffff8101fd9b>] ? reserve_ds_buffers+0x10e/0x347
Jan  6 21:55:02 Pithos kernel: [<ffffffff810477b6>] warn_slowpath_fmt+0x41/0x43
Jan  6 21:55:02 Pithos kernel: [<ffffffff8101fd9b>] reserve_ds_buffers+0x10e/0x347
Jan  6 21:55:02 Pithos kernel: [<ffffffff8101ab5c>] x86_reserve_hardware+0x141/0x153
Jan  6 21:55:02 Pithos kernel: [<ffffffff8101abb2>] x86_pmu_event_init+0x44/0x240
Jan  6 21:55:02 Pithos kernel: [<ffffffff810a7b77>] perf_try_init_event+0x42/0x74
Jan  6 21:55:02 Pithos kernel: [<ffffffff810ad260>] perf_init_event+0x9d/0xd4
Jan  6 21:55:02 Pithos kernel: [<ffffffff810ad61c>] perf_event_alloc+0x385/0x4f7
Jan  6 21:55:02 Pithos kernel: [<ffffffffa01c1553>] ? stop_counter+0x2f/0x2f [kvm]
Jan  6 21:55:02 Pithos kernel: [<ffffffff810ad7bc>] perf_event_create_kernel_counter+0x2e/0x12c
Jan  6 21:55:02 Pithos kernel: [<ffffffffa01c166e>] reprogram_counter+0xc0/0x109 [kvm]
Jan  6 21:55:02 Pithos kernel: [<ffffffffa01c1739>] reprogram_fixed_counter+0x82/0x8d [kvm]
Jan  6 21:55:02 Pithos kernel: [<ffffffffa01c1921>] reprogram_idx+0x4a/0x4f [kvm]
Jan  6 21:55:02 Pithos kernel: [<ffffffffa01c2183>] kvm_handle_pmu_event+0x66/0x87 [kvm]
Jan  6 21:55:02 Pithos kernel: [<ffffffffa01ee2dc>] ? vmx_invpcid_supported+0x1b/0x1b [kvm_intel]
Jan  6 21:55:02 Pithos kernel: [<ffffffffa01ab2dc>] kvm_arch_vcpu_ioctl_run+0x4f9/0xeb0 [kvm]
Jan  6 21:55:02 Pithos kernel: [<ffffffffa01f0773>] ? __vmx_load_host_state.part.53+0x125/0x12c [kvm_intel]
Jan  6 21:55:02 Pithos kernel: [<ffffffffa01a665b>] ? kvm_arch_vcpu_load+0x139/0x143 [kvm]
Jan  6 21:55:02 Pithos kernel: [<ffffffffa019dff1>] kvm_vcpu_ioctl+0x169/0x48f [kvm]
Jan  6 21:55:02 Pithos kernel: [<ffffffffa019de88>] ? vcpu_put+0x3b/0x3b [kvm]
Jan  6 21:55:02 Pithos kernel: [<ffffffff8110c316>] do_vfs_ioctl+0x367/0x421
Jan  6 21:55:02 Pithos kernel: [<ffffffff81114033>] ? __fget+0x6c/0x78
Jan  6 21:55:02 Pithos kernel: [<ffffffff8110c409>] SyS_ioctl+0x39/0x64
Jan  6 21:55:02 Pithos kernel: [<ffffffff815f69ae>] system_call_fastpath+0x12/0x71
Jan  6 21:55:02 Pithos kernel: ---[ end trace cf2394a39c569833 ]---
Jan  6 21:57:02 Pithos bunker: added 3 files from /mnt/disk2 with mask *. Duration: 00:04:54 Average speed: 46.6 MB/s
Jan  6 21:57:27 Pithos bunker: added 8 files from /mnt/disk1 with mask *. Duration: 00:05:20 Average speed: 93.0 MB/s
Jan  6 21:57:59 Pithos emhttp: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
Jan  6 21:58:11 Pithos emhttp: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
Jan  6 21:59:41 Pithos bunker: exported 42588 files from /mnt/disk1 with mask *. Duration: 00:00:31
Jan  6 22:00:24 Pithos emhttp: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
Jan  6 22:01:51 Pithos bunker: exported 34208 files from /mnt/disk2 with mask *. Duration: 00:02:41

Link to comment

There's something I may have misunderstood, so I'm running my tests again, regarding my .DS_Store problem.

 

Can I just check something, please? Under the File Integrity Control tab, am I right in thinking that the Check button initiates a check of the files against their exported checksums and not against the checksums stored in the extended attributes and it isn't a manual equivalent of the scheduled daily/weekly check, which does use the extended attributes? If I am right, then what is the result of clicking the Check button without ever clicking the Export button? Could that explain the "bunker: warning: is missing" messages in my syslog?

 

I've stopped adding files to my shares for the time being and I've run Build again, and then Export. My disks now have both Initial Build and Initial Export green checks. I'm running Check at the moment and hope to see no errors this time.

 

Nice plugin, btw :) Many thanks.

 

EDIT: The manual check has completed with no errors reported. I'll use the server as normal and see if any errors are reported after the scheduled check on Sunday.

 

Link to comment

Over the last few weeks my server has been getting random kernel warnings. I have been trying to figure out why they are happening all of the sudden and I think they have something to do with this plugin. After updating to the latest plugin I started running the initial build and export on both of my disks. This kernel warning happened right after I started that process. Does any of this log mean anything too you?

What format are the disks?  Recently a warning was added that problems might be encountered with the plugin if Reiserfs is being used for the disks.

Link to comment

This utility only reports file corruption but does not have any means to repair it. This action is with the user.

 

So what options are available for us to fix bitrot/whatever?

Restore from backup.

 

Or Squid has a checksum plugin that also makes par2 files but I haven't tried that functionality yet.

Link to comment

Over the last few weeks my server has been getting random kernel warnings. I have been trying to figure out why they are happening all of the sudden and I think they have something to do with this plugin. After updating to the latest plugin I started running the initial build and export on both of my disks. This kernel warning happened right after I started that process. Does any of this log mean anything too you?

bts buffer is the branch trace store buffer, google indicates there was a patch last year to reduce the frequency of such logs & whether it happens or not is a function of memory fragmentation. I would think that seeing this is just a sign that this plugin is working your CPU hard. Any other problems noted or just that warn?

Link to comment

 

Over the last few weeks my server has been getting random kernel warnings. I have been trying to figure out why they are happening all of the sudden and I think they have something to do with this plugin. After updating to the latest plugin I started running the initial build and export on both of my disks. This kernel warning happened right after I started that process. Does any of this log mean anything too you?

What format are the disks?  Recently a warning was added that problems might be encountered with the plugin if Reiserfs is being used for the disks.

 

All disks are XFS

Link to comment

 

Over the last few weeks my server has been getting random kernel warnings. I have been trying to figure out why they are happening all of the sudden and I think they have something to do with this plugin. After updating to the latest plugin I started running the initial build and export on both of my disks. This kernel warning happened right after I started that process. Does any of this log mean anything too you?

bts buffer is the branch trace store buffer, google indicates there was a patch last year to reduce the frequency of such logs & whether it happens or not is a function of memory fragmentation. I would think that seeing this is just a sign that this plugin is working your CPU hard. Any other problems noted or just that warn?

 

That is the only relevant info in the log.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.