mattkhan Posted January 6, 2016 Share Posted January 6, 2016 Thanks for your observation. I have seen in the past that RFS can have issues in V6 when writing extended attributes and once in a blue moon completely hangs. I am going to put a BIG warning that using this plugin in combination with RFS may lead to instability. WARNING: USING THIS PLUGIN ON DISKS FORMATTED IN REISERFS MAY LEAD TO SYSTEM INSTABILITY. IT IS ADVISED TO USE XFS. OK thanks, good to have an explanation for the behaviour. Quote Link to comment
taalas Posted January 6, 2016 Share Posted January 6, 2016 Wow, that was close...I installed the plugin yesterday but did not activate any checksum generation. Looks like with 3 of my drives being ReiserFS still I better won't Quote Link to comment
bonienl Posted January 6, 2016 Author Share Posted January 6, 2016 Wow, that was close...I installed the plugin yesterday but did not activate any checksum generation. Looks like with 3 of my drives being ReiserFS still I better won't There are chances it works properly, but nothing guaranteed. To play save I recommend to use it only on XFS formatted disks. Quote Link to comment
landS Posted January 6, 2016 Share Posted January 6, 2016 WARNING: USING THIS PLUGIN ON DISKS FORMATTED IN REISERFS MAY LEAD TO SYSTEM INSTABILITY. IT IS ADVISED TO USE XFS. POO! My Primary/low power server is still on REIERFS... was hoping to keep it REIERFS as the Backup Server uses XFS and it is nice knowing that the stable REIERFS recovery tools exist. Perhaps THIS is what locked up the server and not the 100% CPU utilization when trying 3 disks at once? Any risk of DATA loss with REIERFS now? I just finished triple pre-clearing another 4TB HGST for the hoped for dual parity. Would the advised course of action be 1 - Disable automatic checking 2 - Switch out each disks filesystem to XFS as per http://lime-technology.com/forum/index.php?topic=37490.msg346739#msg346739 ... only using unBalance rather than screen rsync http://lime-technology.com/forum/index.php?topic=39707.0 3 - Remove empty disk from the array 4 - New Config 5 - Build Fresh Parity 6 - Redo the Initial Integrity Build Per disk Thanks Quote Link to comment
bonienl Posted January 6, 2016 Author Share Posted January 6, 2016 Not completely clear to me what you want to achieve. Are you asking for the steps how to convert from RFS to XFS on your existing disks ? Quote Link to comment
landS Posted January 6, 2016 Share Posted January 6, 2016 Nope... Sorry for the poorly worded panic posting. I ran the File Integrity Plugin on a machine with all RFS disks 1 - Should I kill the plugin, convert to XFS, then rerun the plugin? 2 - Any risk of Data Loss on the RFS disks now that the plugin has been run? Quote Link to comment
bonienl Posted January 6, 2016 Author Share Posted January 6, 2016 Nope... Sorry for the poorly worded panic posting. I ran the File Integrity Plugin on a machine with all RFS disks 1 - Should I kill the plugin, convert to XFS, then rerun the plugin? 2 - Any risk of Data Loss on the RFS disks now that the plugin has been run? Whenever you convert from one filesystem to another, the disk needs to be reformatted. In that process you will loose the information stored in the extended attributes, as it goes with the filesystem. The plugin doesn't touch the actual files, and running the plugin doesn't pose a risk to your data. When the GUI hangs due to extended attributes not properly written under RFS, look for the process "setfattr" and do killall setfattr. Quote Link to comment
trurl Posted January 6, 2016 Share Posted January 6, 2016 ... Would the advised course of action be 1 - Disable automatic checking 2 - Switch out each disks filesystem to XFS as per http://lime-technology.com/forum/index.php?topic=37490.msg346739#msg346739 ... only using unBalance rather than screen rsync http://lime-technology.com/forum/index.php?topic=39707.0 3 - Remove empty disk from the array 4 - New Config 5 - Build Fresh Parity 6 - Redo the Initial Integrity Build Per disk Thanks 3,4,5 don't necessarily have anything to do with converting depending on your specific situation. I was able to convert all my data disks to XFS without removing any drives or rebuilding parity just by moving things off each drive and onto others with free space, then formatting the empty, then repeating with others. There is a lot in that first thread you linked. Quote Link to comment
bonienl Posted January 6, 2016 Author Share Posted January 6, 2016 SMALL feature request for future bumps... a flag indicating if an initial build and a flag indicating an export has been accomplished per disk. Good idea, but I need to work out how to do this in a reliable way. Quote Link to comment
landS Posted January 6, 2016 Share Posted January 6, 2016 Whenever you convert from one filesystem to another, the disk needs to be reformatted. In that process you will loose the information stored in the extended attributes, as it goes with the filesystem. The plugin doesn't touch the actual files, and running the plugin doesn't pose a risk to your data. When the GUI hangs due to extended attributes not properly written under RFS, look for the process "setfattr" and do killall setfattr. Fantastic Info here - also clears up a bit of ignorance on my part SMALL feature request for future bumps... a flag indicating if an initial build and a flag indicating an export has been accomplished per disk. Good idea, but I need to work out how to do this in a reliable way. perhaps write a CSV file to the flash that is read each time the gui page is accessed? Disk#, InitialBuild, InitialExport ... Would the advised course of action be 2 - Switch out each disks filesystem to XFS as per http://lime-technology.com/forum/index.php?topic=37490.msg346739#msg346739 ... only using unBalance rather than screen rsync http://lime-technology.com/forum/index.php?topic=39707.0 3 - Remove empty disk from the array 4 - New Config 5 - Build Fresh Parity 3,4,5 don't necessarily have anything to do with converting depending on your specific situation. I was able to convert all my data disks to XFS without removing any drives or rebuilding parity just by moving things off each drive and onto others with free space, then formatting the empty, then repeating with others. There is a lot in that first thread you linked. Cheers, I am going through these links now! LOTS of info here! I do not have enough room on the current array to free up a disk I will need to add a fresh XFS disk to the array in order to move files I do not want to leave the new disk in the array as it will have a home as a 2nd parity drive (Hopefully soon). Quote Link to comment
bonienl Posted January 6, 2016 Author Share Posted January 6, 2016 SMALL feature request for future bumps... a flag indicating if an initial build and a flag indicating an export has been accomplished per disk. Have created the following, see the attached picture. For all disks it will show whether an initial build and/or an initial export was done. A green check means yes, while an orange circle means no. Useful ? Quote Link to comment
archedraft Posted January 6, 2016 Share Posted January 6, 2016 SMALL feature request for future bumps... a flag indicating if an initial build and a flag indicating an export has been accomplished per disk. Have created the following, see the attached picture. For all disks it will show whether an initial build and/or an initial export was done. A green check means yes, while an orange circle means no. Useful ? Yes, I think it looks nice and adds value. Quote Link to comment
bonienl Posted January 6, 2016 Author Share Posted January 6, 2016 For those interested, version 2016.01.06a is available now, see release notes. Quote Link to comment
archedraft Posted January 6, 2016 Share Posted January 6, 2016 For those interested, version 2016.01.06a is available now, see release notes. Just updated and saw that my disks have the orange circle. To clarify, will they always be a circle upon a refresh reboot of the server? Quote Link to comment
bonienl Posted January 6, 2016 Author Share Posted January 6, 2016 For those interested, version 2016.01.06a is available now, see release notes. Just updated and saw that my disks have the orange circle. To clarify, will they always be a circle upon a refresh reboot of the server? No, once a build or export is done, this will change into a check symbol. This information is kept on flash in the file /boot/config/plugins/dynamix.file.integrity/disks.ini Mine looks as follows: disk11=build,export disk3=build,export disk2=build,export disk8=build,export disk9=build,export disk4=build,export disk10=build,export disk6=build,export disk5=build,export disk7=build,export disk1=build,export You can "trick" the system by creating/editing this file yourself. E.g. when you know all disks are built and exported, create the file with the proper entries. Quote Link to comment
landS Posted January 7, 2016 Share Posted January 7, 2016 On my high power server, just re-built and re-exported - this took less than 1 minute for 3 nearly full 4 TB disks... likely no need for earlier version users to manually create the file. Now all Green Checks, very nice! Quote Link to comment
John_M Posted January 7, 2016 Share Posted January 7, 2016 I'm getting checksum errors reported on Macintosh .DS_Store files. It seems possible that when I add files to a folder on my unRAID share using the OS X Finder the .DS_Store file within that folder is being updated but its checksum isn't, which results in the file corruption messages. unRAID file corruption: 07-01-2016 09:49 Notice [LAPULAPU] - bunker check command Found 6 files with BLAKE2 hash key corruption Syslog snippet: Jan 7 03:52:47 Lapulapu bunker: warning: is missing Jan 7 03:52:48 Lapulapu bunker: warning: is missing Jan 7 03:58:18 Lapulapu bunker: warning: is missing Jan 7 03:59:04 Lapulapu bunker: warning: is missing Jan 7 04:24:40 Lapulapu bunker: warning: is missing Jan 7 04:25:03 Lapulapu bunker: warning: is missing Jan 7 05:48:36 Lapulapu bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/L_Public/.DS_Store is corrupted Jan 7 05:48:36 Lapulapu bunker: warning: is missing Jan 7 05:48:36 Lapulapu bunker: warning: is missing Jan 7 05:52:11 Lapulapu bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/L_Public/Shared Videos/.DS_Store is corrupted Jan 7 05:53:22 Lapulapu bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/L_Public/Shared Videos/DVD Rips/.DS_Store is corrupted Jan 7 06:53:48 Lapulapu bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/L_Public/Shared Videos/DVD Rips/3 MKVs/.DS_Store is corrupted Jan 7 07:08:10 Lapulapu kernel: mdcmd (68): spindown 0 Jan 7 07:08:11 Lapulapu kernel: mdcmd (69): spindown 2 Jan 7 08:12:44 Lapulapu bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/L_Public/Shared Videos/EyeTV Archive/.DS_Store is corrupted Jan 7 09:15:00 Lapulapu emhttp: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin checkall Jan 7 09:45:16 Lapulapu bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/L_Public/Shared Videos/MakeMKV/.DS_Store is corrupted Jan 7 09:49:16 Lapulapu bunker: checked 49978 files from source /boot/config/plugins/dynamix.file.integrity/disk1.export.hash. Found: 0 mismatches (updated), 6 corruptions. Duration: 06:36:46 Average speed: 118. MB/s Perhaps the lines containing "is missing" are a clue. Is there a way of telling the plugin to exclude .DS_Store files? Quote Link to comment
archedraft Posted January 7, 2016 Share Posted January 7, 2016 Over the last few weeks my server has been getting random kernel warnings. I have been trying to figure out why they are happening all of the sudden and I think they have something to do with this plugin. After updating to the latest plugin I started running the initial build and export on both of my disks. This kernel warning happened right after I started that process. Does any of this log mean anything too you? Jan 6 21:55:02 Pithos kernel: ------------[ cut here ]------------ Jan 6 21:55:02 Pithos kernel: WARNING: CPU: 3 PID: 24488 at arch/x86/kernel/cpu/perf_event_intel_ds.c:315 reserve_ds_buffers+0x10e/0x347() Jan 6 21:55:02 Pithos kernel: alloc_bts_buffer: BTS buffer allocation failure Jan 6 21:55:02 Pithos kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables kvm_intel kvm vhost_net vhost macvtap macvlan tun xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod ahci r8169 i2c_i801 libahci mii Jan 6 21:55:02 Pithos kernel: CPU: 3 PID: 24488 Comm: qemu-system-x86 Not tainted 4.1.13-unRAID #1 Jan 6 21:55:02 Pithos kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H77 Pro4-M, BIOS P2.00 08/06/2013 Jan 6 21:55:02 Pithos kernel: 0000000000000009 ffff880357b039a8 ffffffff815f12b0 0000000000000000 Jan 6 21:55:02 Pithos kernel: ffff880357b039f8 ffff880357b039e8 ffffffff8104775b ffff88044f395ec0 Jan 6 21:55:02 Pithos kernel: ffffffff8101fd9b 0000000000000000 0000000000000000 0000000000010e10 Jan 6 21:55:02 Pithos kernel: Call Trace: Jan 6 21:55:02 Pithos kernel: [<ffffffff815f12b0>] dump_stack+0x4c/0x6e Jan 6 21:55:02 Pithos kernel: [<ffffffff8104775b>] warn_slowpath_common+0x97/0xb1 Jan 6 21:55:02 Pithos kernel: [<ffffffff8101fd9b>] ? reserve_ds_buffers+0x10e/0x347 Jan 6 21:55:02 Pithos kernel: [<ffffffff810477b6>] warn_slowpath_fmt+0x41/0x43 Jan 6 21:55:02 Pithos kernel: [<ffffffff8101fd9b>] reserve_ds_buffers+0x10e/0x347 Jan 6 21:55:02 Pithos kernel: [<ffffffff8101ab5c>] x86_reserve_hardware+0x141/0x153 Jan 6 21:55:02 Pithos kernel: [<ffffffff8101abb2>] x86_pmu_event_init+0x44/0x240 Jan 6 21:55:02 Pithos kernel: [<ffffffff810a7b77>] perf_try_init_event+0x42/0x74 Jan 6 21:55:02 Pithos kernel: [<ffffffff810ad260>] perf_init_event+0x9d/0xd4 Jan 6 21:55:02 Pithos kernel: [<ffffffff810ad61c>] perf_event_alloc+0x385/0x4f7 Jan 6 21:55:02 Pithos kernel: [<ffffffffa01c1553>] ? stop_counter+0x2f/0x2f [kvm] Jan 6 21:55:02 Pithos kernel: [<ffffffff810ad7bc>] perf_event_create_kernel_counter+0x2e/0x12c Jan 6 21:55:02 Pithos kernel: [<ffffffffa01c166e>] reprogram_counter+0xc0/0x109 [kvm] Jan 6 21:55:02 Pithos kernel: [<ffffffffa01c1739>] reprogram_fixed_counter+0x82/0x8d [kvm] Jan 6 21:55:02 Pithos kernel: [<ffffffffa01c1921>] reprogram_idx+0x4a/0x4f [kvm] Jan 6 21:55:02 Pithos kernel: [<ffffffffa01c2183>] kvm_handle_pmu_event+0x66/0x87 [kvm] Jan 6 21:55:02 Pithos kernel: [<ffffffffa01ee2dc>] ? vmx_invpcid_supported+0x1b/0x1b [kvm_intel] Jan 6 21:55:02 Pithos kernel: [<ffffffffa01ab2dc>] kvm_arch_vcpu_ioctl_run+0x4f9/0xeb0 [kvm] Jan 6 21:55:02 Pithos kernel: [<ffffffffa01f0773>] ? __vmx_load_host_state.part.53+0x125/0x12c [kvm_intel] Jan 6 21:55:02 Pithos kernel: [<ffffffffa01a665b>] ? kvm_arch_vcpu_load+0x139/0x143 [kvm] Jan 6 21:55:02 Pithos kernel: [<ffffffffa019dff1>] kvm_vcpu_ioctl+0x169/0x48f [kvm] Jan 6 21:55:02 Pithos kernel: [<ffffffffa019de88>] ? vcpu_put+0x3b/0x3b [kvm] Jan 6 21:55:02 Pithos kernel: [<ffffffff8110c316>] do_vfs_ioctl+0x367/0x421 Jan 6 21:55:02 Pithos kernel: [<ffffffff81114033>] ? __fget+0x6c/0x78 Jan 6 21:55:02 Pithos kernel: [<ffffffff8110c409>] SyS_ioctl+0x39/0x64 Jan 6 21:55:02 Pithos kernel: [<ffffffff815f69ae>] system_call_fastpath+0x12/0x71 Jan 6 21:55:02 Pithos kernel: ---[ end trace cf2394a39c569833 ]--- Jan 6 21:57:02 Pithos bunker: added 3 files from /mnt/disk2 with mask *. Duration: 00:04:54 Average speed: 46.6 MB/s Jan 6 21:57:27 Pithos bunker: added 8 files from /mnt/disk1 with mask *. Duration: 00:05:20 Average speed: 93.0 MB/s Jan 6 21:57:59 Pithos emhttp: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog Jan 6 21:58:11 Pithos emhttp: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog Jan 6 21:59:41 Pithos bunker: exported 42588 files from /mnt/disk1 with mask *. Duration: 00:00:31 Jan 6 22:00:24 Pithos emhttp: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog Jan 6 22:01:51 Pithos bunker: exported 34208 files from /mnt/disk2 with mask *. Duration: 00:02:41 Quote Link to comment
John_M Posted January 7, 2016 Share Posted January 7, 2016 There's something I may have misunderstood, so I'm running my tests again, regarding my .DS_Store problem. Can I just check something, please? Under the File Integrity Control tab, am I right in thinking that the Check button initiates a check of the files against their exported checksums and not against the checksums stored in the extended attributes and it isn't a manual equivalent of the scheduled daily/weekly check, which does use the extended attributes? If I am right, then what is the result of clicking the Check button without ever clicking the Export button? Could that explain the "bunker: warning: is missing" messages in my syslog? I've stopped adding files to my shares for the time being and I've run Build again, and then Export. My disks now have both Initial Build and Initial Export green checks. I'm running Check at the moment and hope to see no errors this time. Nice plugin, btw Many thanks. EDIT: The manual check has completed with no errors reported. I'll use the server as normal and see if any errors are reported after the scheduled check on Sunday. Quote Link to comment
itimpi Posted January 7, 2016 Share Posted January 7, 2016 Over the last few weeks my server has been getting random kernel warnings. I have been trying to figure out why they are happening all of the sudden and I think they have something to do with this plugin. After updating to the latest plugin I started running the initial build and export on both of my disks. This kernel warning happened right after I started that process. Does any of this log mean anything too you? What format are the disks? Recently a warning was added that problems might be encountered with the plugin if Reiserfs is being used for the disks. Quote Link to comment
Dephcon Posted January 7, 2016 Share Posted January 7, 2016 This utility only reports file corruption but does not have any means to repair it. This action is with the user. So what options are available for us to fix bitrot/whatever? Quote Link to comment
trurl Posted January 7, 2016 Share Posted January 7, 2016 This utility only reports file corruption but does not have any means to repair it. This action is with the user. So what options are available for us to fix bitrot/whatever? Restore from backup. Or Squid has a checksum plugin that also makes par2 files but I haven't tried that functionality yet. Quote Link to comment
mattkhan Posted January 7, 2016 Share Posted January 7, 2016 Over the last few weeks my server has been getting random kernel warnings. I have been trying to figure out why they are happening all of the sudden and I think they have something to do with this plugin. After updating to the latest plugin I started running the initial build and export on both of my disks. This kernel warning happened right after I started that process. Does any of this log mean anything too you? bts buffer is the branch trace store buffer, google indicates there was a patch last year to reduce the frequency of such logs & whether it happens or not is a function of memory fragmentation. I would think that seeing this is just a sign that this plugin is working your CPU hard. Any other problems noted or just that warn? Quote Link to comment
archedraft Posted January 7, 2016 Share Posted January 7, 2016 Over the last few weeks my server has been getting random kernel warnings. I have been trying to figure out why they are happening all of the sudden and I think they have something to do with this plugin. After updating to the latest plugin I started running the initial build and export on both of my disks. This kernel warning happened right after I started that process. Does any of this log mean anything too you? What format are the disks? Recently a warning was added that problems might be encountered with the plugin if Reiserfs is being used for the disks. All disks are XFS Quote Link to comment
archedraft Posted January 7, 2016 Share Posted January 7, 2016 Over the last few weeks my server has been getting random kernel warnings. I have been trying to figure out why they are happening all of the sudden and I think they have something to do with this plugin. After updating to the latest plugin I started running the initial build and export on both of my disks. This kernel warning happened right after I started that process. Does any of this log mean anything too you? bts buffer is the branch trace store buffer, google indicates there was a patch last year to reduce the frequency of such logs & whether it happens or not is a function of memory fragmentation. I would think that seeing this is just a sign that this plugin is working your CPU hard. Any other problems noted or just that warn? That is the only relevant info in the log. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.