SuperW2

Can't change DNS Settings on Network settings page... everything is "greyed" out and no option to change (and array is stopped). The network settings can not be changed as long as Docker or libvirt (VM) is still running. Stopping the array should stop these services as well. Or alternatively Docker and VM services can be stopped from the settings page. Your picture is showing however the array as active and your issue is a missing DNS entry. If you don't have a local DNS server or DNS proxy, you can use 8.8.8.8 and 8.8.4.4. Array is stopped and those DNS entrys are still greyed out (and I've never ever used 8.8.8.8/8.8.4.4 and never had any issues whatsoever, been running UnRAID for years!) Docker and VM ware also both set to "NO" for enable for both in settings.

Can't change DNS Settings on Network settings page... everything is "greyed" out and no option to change (and array is stopped).

And it doesn't seem to matter if I disable bonding, bridging, switch NIC cables, plug into a different network switch or reboot the server...

So I was trying to figure out why my Windows VM was getting an "external" IP instead of one from my internal LAN, looked at VM and Network was set to Virbr0 instead bro (and it was the only option in VM Settings), so I went to go look at my Network settings, bridging was already enabled, so I disabled it, saved, re-enabled it, and then started getting this message from the "Fix Common Problems" App, although BR0 was now an option in my VM. Here is my Network settings, as they have always been set and everything had previously worked fine (except for VM bridge which I don't use often). Now things like the community apps won't connect and getting new weird messaged on Fix Common Problems, like I assume it's not connecting to external internet.. Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/fix.common.problems/include/tests.php on line 1179 Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/fix.common.problems/include/tests.php on line 1207 Diag's attached. media-diagnostics-20160820-0929.zip

I've been having this issue as well. Any time I try and transfer files from a share to a flash drive or another computer, it will lock up Samba, WebGUI, Dockers, pretty much everything. I can SSH as well, but powerdown does nothing. I've also observed this same behavior when Sabnzbd is processing very large files and moving them into the array. Seems it's definitely because of heavy IO load. I'll try the suggested fix of changing the md_num_stripes to 8192 and report back if that works around the issue. I too have had several issues, when transferring files (between drives in array, between drives and cache, between local USB and drives in array) with everything locking up, losing GUI, etc but can still SSH and my Powerdown script also does nothing. Hopefully the LT guys are reading this and notice a pattern of some kind here. I'm transfering over 50GB right now after trying out the workaround of adjusting the md_num_stripes. So far I haven't run into an issue. Usually for me it would lock up almost immediately. I made the md_num_stripes change on mine, but havn't attempted any large file transfers since (and was tired of hard power cycling my server several times a day when doing previous trouble shooting.

I've been having this issue as well. Any time I try and transfer files from a share to a flash drive or another computer, it will lock up Samba, WebGUI, Dockers, pretty much everything. I can SSH as well, but powerdown does nothing. I've also observed this same behavior when Sabnzbd is processing very large files and moving them into the array. Seems it's definitely because of heavy IO load. I'll try the suggested fix of changing the md_num_stripes to 8192 and report back if that works around the issue. I too have had several issues, when transferring files (between drives in array, between drives and cache, between local USB and drives in array) with everything locking up, losing GUI, etc but can still SSH and my Powerdown script also does nothing. Hopefully the LT guys are reading this and notice a pattern of some kind here.

Just for giggles, I tested Total Commander and it locked after attempting to move 800ish MB of a single 1GB file from Disk 18 to Disk 1... Docker Disabled, VM's Disable, NTP Off, I forgot to disable my extra plugins, but almost nothing else running. Latest Diags attached after last hard reboot. I also went into BIOS after boot, and my BIOS clock was off again, forward several hours... odd... media-diagnostics-20160513-1532.zip

Just using native Windows Explorer using either Disk or User shares. I've never tried anything else.

Thanks for the reply... I understand bits of that, an can explain some of it. As far as the RSync's, I use the Indexer App/Excel spreadsheet from http://lime-technology.com/forum/index.php?topic=33689.0 to help move sections of my file shares from one drive to the other (with 18 data drives, I try to keep the files/folders somewhat organized and I have to do that manually. I can get an example of the RSync command that it uses. I'm not changing the RSync commands at all, just a direct copy and paste from spreadsheet into SSH connection to server. I have previously, on occassion had 5 or 6 SSH sessions, each with it's own RSYNC thread moving files from one drive to another and never had an issue. WinVM... I run a Win10 Test VM, occasionally, and would have likely be running during some of these last hang ups. - Easy to disable Java... No idea from where or what - ? Plex is running from the "linuxserver" docker, but I don't think any of my devices would have actively been using it. I only have a few plugin's running now (removed several) : Powerdown Package 2.20 (that doesn't work), Fix Common Problems (that doesn't find any with my system), Communicty Application and then the main Unraid Server OS and Dynamix WebGui ones. For troubleshooting, I can disable my Docker completely (where Crashplan/Plex/SabNZB/Sickbeard, etc are all running) and the "extra" plugins and see if there is any change. Disabled NTP for now No clue on the sbmd or the python processes. I can almost "replicate" a GUI lockup/crash on command by starting just 2 simultaneous SMB file copy operations on my Win10 Box from any one drive to another anywhere in my array. Once that that happens the copy/move fails, GUI unreachable, but I never lose Ping/SSH/Server Console access. ...and possibly unrelated, I still think it's goofy that a Powerdown script from a SSH command line won't actually shut down my Server (maybe a BIOS Setting?) and even when I change the Syslinux settings on the FlashDrive in the console, they never update, I don't see the GUI setting and can only enter it on server when I halt the autostart, hit tab and add the GUI command line.

Totally dead in the water again... Server GUI hung no less than a dozen times today and had to hard power reset each time. Attached latest Diag Zip media-diagnostics-20160511-1904.zip

.. and now GUI Locked up doing nothing... no file move copy operations, just becomes unresponsive, but can can still ping/ssh and access from server console.

Something is still off... it appears the gui becomes unresponsive if I start more than 1 file copy/move operation (either via Windows Share or from SSH), even when dealing with neither Disk 6 or Disk 11 (still trying to move some data around on other disks to get stuff in in order). Woke up this morning after stating a couple file move operations via SSH and have lost all my user shares in Windows, can still ping, but a new SSH connection just asks for username and then password (which I don't have one set for root account and it never normally asks me). I'm unable to do a clean powerdown from the server prompt using the powerdown script (it creates the diag log file and then just hangs on "The System is going down for system half NOW!" but never physically powers off. New Diag Log attached.... 22:28-22:33 was me starting SSH sessions for file data moves after a freshish reboot (because locked/lost gui), 6:57 was me trying to log in again. The USB Losses/at 6:54:38 was me turning off Monitor that has USB Hub where local Server KB/Mouse are plugged in and turned it back on at 7:01:15. Also see another time sync error at 22:31:17. media-diagnostics-20160511-0703.zip

Thanks, I'll check the data and power going to sdg/Disk11 once I get to a reboot point (Still have 1.6TB of data to move off of Disk6)... Most likely one 5:1 reverse breakout cables going to one of my SAS Cards, and the power is fed by 3 4Pin's to the back of the 5-in-3 cage... -W

Thanks for the great Reply RobJ... A couple updates and comments to your comments... I did a XFS_Repair on Disk6 and had to use the "-L" option since I couldn't mount the disk... I don't remember specifically what it said about call traces. I did get a couple of these messages below, but nothing after I did the XFS_Repair. The was the last message I received like this;l May 9 16:09:33 media kernel: ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300) May 9 16:09:33 media kernel: ata6.00: configured for UDMA/133 May 9 16:09:33 media kernel: ata6: EH complete May 9 16:10:31 media kernel: ata6: limiting SATA link speed to 3.0 Gbps May 9 16:10:31 media kernel: ata6.00: exception Emask 0x10 SAct 0x0 SErr 0x280100 action 0x6 frozen May 9 16:10:31 media kernel: ata6.00: irq_stat 0x08000000, interface fatal error May 9 16:10:31 media kernel: ata6: SError: { UnrecovData 10B8B BadCRC } May 9 16:10:31 media kernel: ata6.00: failed command: READ DMA EXT May 9 16:10:31 media kernel: ata6.00: cmd 25/00:40:78:54:62/00:05:0b:00:00/e0 tag 15 dma 688128 in May 9 16:10:31 media kernel: res 50/00:00:77:54:62/00:00:0b:00:00/e0 Emask 0x10 (ATA bus error) May 9 16:10:31 media kernel: ata6.00: status: { DRDY } May 9 16:10:31 media kernel: ata6: hard resetting link Once I did the XFS_repair, I was able to start the array and it immediately started a data rebuild of Disk6 (had yellow/orange triangle). That took the rest of the day/night/morning to complete (16ish hours) but hadn't seen any ATA Errors since it completed and no errors reported in console or Syslog that I can see. I am in the process of moving my Data off of Disk6 "just in case"...but that's going to take a while since it was almost topped off with 4TB of data (and again no errors in log after 6 or 7 hours of data transfer). The Network drops in the logs was likely me, physically putzing... I swapped both physical cables on the server with new ones (just as a precaution trying to rule out anything), while it was running... not the smartest I know but I'd bet that is what was recorded then. As far as the clock, I changed the CMOS Battery yesterday (although the old one was registering good voltage in BIOS), reset BIOS to defaults, etc. Again just trying to eliminate any variable of something that could have been causing issues. So far since changing have not noticed anything odd with the clock settings. I was seeing some oddity in the Syslinux configuration on my Flash disk and No "unRAID OS GUI Mode" appearing on Server Startup, even though it was listed in the Config on the console. I clicked "Default" button and it did appear to change a few lines (which I didn't do a before/after comparison), and haven't rebooted since, so will do that once I finishing moving the data off Disk 6 to see if that addressed that issue. So right now, I have an apparently "fully operational" running array with a rebuilt data on Disk 6, trying to empty it just in case and crossing fingers I can get it all off in case Disk6 is going out. Thanks SW2

Well, I started the Array and it's doing a data rebuild now on disk6 and it has been running about an hour and about 9% done... we'll see what happens, but at least a step in right direction.

Yes, I can get to the GUI now remotely (once I disabled plugins and set Array Start/Dockers to not autostart)... I can't seem to get the full "GUI Mode" to work at all from my Physical Server with a Monitor/KB/Mouse connected... Its not even an option when I boot my Flash This is in my Flash Settings on Syslinux Configuration tab but no option on startup (on monitor connected to server). label unRAID OS GUI Mode menu default kernel /bzimage append initrd=/bzroot,/bzroot-gui *EDIT* I was able to hit tab and append the ",/bzroot-gui" on the main Unraid item and the GUI started... odd that it didn't show up on the list as it should. I also ran a xfs_repair -L on Disk 6 after this log was created this morning. Attached is a current one from Remote GUI, Array started in Maintenance Mode only... assume if I try to mount array regularly it will hang again. The issue now is when I start the array, everything grinds to a halt again, GUI becomes unresponsive... this was the last bit of the last Diag Zip that seems to be happening dozens of times per second... May 9 10:45:24 media kernel: swapper/0: page allocation failure: order:0, mode:0x2080020 May 9 10:45:24 media kernel: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.4.6-unRAID #1 May 9 10:45:24 media kernel: Hardware name: Supermicro X10SAE/X10SAE, BIOS 3.0 05/20/2015 May 9 10:45:24 media kernel: 0000000000000000 ffff88041dc03c28 ffffffff813688da 0000000000000000 May 9 10:45:24 media kernel: 0000000000000000 ffff88041dc03cc0 ffffffff810bc9b0 ffffffff818b0e38 May 9 10:45:24 media kernel: ffff88041dff9b00 ffffffffffffffff ffffffff008b0680 0000000000000000 May 9 10:45:24 media kernel: Call Trace: May 9 10:45:24 media kernel: <IRQ> [<ffffffff813688da>] dump_stack+0x61/0x7e May 9 10:45:24 media kernel: [<ffffffff810bc9b0>] warn_alloc_failed+0x10f/0x127 May 9 10:45:24 media kernel: [<ffffffff810bf9c7>] __alloc_pages_nodemask+0x870/0x8ca May 9 10:45:24 media kernel: [<ffffffff814333a9>] ? device_has_rmrr+0x5a/0x63 May 9 10:45:24 media kernel: [<ffffffff810bfabd>] __alloc_page_frag+0x9c/0x15f May 9 10:45:24 media kernel: [<ffffffff8152e310>] __napi_alloc_skb+0x61/0xc1 May 9 10:45:24 media kernel: [<ffffffffa053e92a>] igb_poll+0x441/0xc06 [igb] May 9 10:45:24 media kernel: [<ffffffff815390ac>] net_rx_action+0xd8/0x226 May 9 10:45:24 media kernel: [<ffffffff8104d4c0>] __do_softirq+0xc3/0x1b6 May 9 10:45:24 media kernel: [<ffffffff8104d73d>] irq_exit+0x3d/0x82 May 9 10:45:24 media kernel: [<ffffffff8100db9a>] do_IRQ+0xaa/0xc2 May 9 10:45:24 media kernel: [<ffffffff8161ab42>] common_interrupt+0x82/0x82 May 9 10:45:24 media kernel: <EOI> [<ffffffff815041b7>] ? cpuidle_enter_state+0xf0/0x148 May 9 10:45:24 media kernel: [<ffffffff81504170>] ? cpuidle_enter_state+0xa9/0x148 May 9 10:45:24 media kernel: [<ffffffff81504231>] cpuidle_enter+0x12/0x14 May 9 10:45:24 media kernel: [<ffffffff81076247>] call_cpuidle+0x4e/0x50 May 9 10:45:24 media kernel: [<ffffffff810763cf>] cpu_startup_entry+0x186/0x1fd May 9 10:45:24 media kernel: [<ffffffff8160fbdd>] rest_init+0x84/0x87 May 9 10:45:24 media kernel: [<ffffffff818eaec0>] start_kernel+0x3f7/0x404 May 9 10:45:24 media kernel: [<ffffffff818ea120>] ? early_idt_handler_array+0x120/0x120 May 9 10:45:24 media kernel: [<ffffffff818ea339>] x86_64_start_reservations+0x2a/0x2c May 9 10:45:24 media kernel: [<ffffffff818ea421>] x86_64_start_kernel+0xe6/0xf3 May 9 10:45:24 media kernel: Mem-Info: May 9 10:45:24 media kernel: active_anon:468687 inactive_anon:4711 isolated_anon:0 May 9 10:45:24 media kernel: active_file:443016 inactive_file:3009187 isolated_file:32 May 9 10:45:24 media kernel: unevictable:0 dirty:64349 writeback:152019 unstable:0 May 9 10:45:24 media kernel: slab_reclaimable:51705 slab_unreclaimable:30682 May 9 10:45:24 media kernel: mapped:51722 shmem:85744 pagetables:5236 bounce:0 May 9 10:45:24 media kernel: free:17874 free_pcp:104 free_cma:0 May 9 10:45:24 media kernel: Node 0 DMA free:15580kB min:12kB low:12kB high:16kB active_anon:304kB inactive_anon:16kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:320kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes May 9 10:45:24 media kernel: lowmem_reserve[]: 0 3512 16022 16022 May 9 10:45:24 media kernel: Node 0 DMA32 free:51276kB min:3524kB low:4404kB high:5284kB active_anon:572120kB inactive_anon:3316kB active_file:391584kB inactive_file:2440188kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:3607096kB managed:3597428kB mlocked:0kB dirty:61208kB writeback:129236kB mapped:48616kB shmem:74916kB slab_reclaimable:44168kB slab_unreclaimable:26384kB kernel_stack:3376kB pagetables:5800kB unstable:0kB bounce:0kB free_pcp:144kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:44 all_unreclaimable? no May 9 10:45:24 media kernel: lowmem_reserve[]: 0 0 12510 12510 May 9 10:45:24 media kernel: Node 0 Normal free:4640kB min:12564kB low:15704kB high:18844kB active_anon:1302324kB inactive_anon:15512kB active_file:1380480kB inactive_file:9596560kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13074432kB managed:12810880kB mlocked:0kB dirty:196188kB writeback:478840kB mapped:158240kB shmem:267740kB slab_reclaimable:162652kB slab_unreclaimable:96344kB kernel_stack:11968kB pagetables:15144kB unstable:0kB bounce:0kB free_pcp:272kB local_pcp:140kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no May 9 10:45:24 media kernel: lowmem_reserve[]: 0 0 0 0 May 9 10:45:24 media kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 3*64kB (UM) 2*128kB (UM) 1*256kB (U) 1*512kB (M) 2*1024kB (UM) 2*2048kB (UM) 2*4096kB (M) = 15580kB May 9 10:45:24 media kernel: Node 0 DMA32: 499*4kB (ME) 306*8kB (UME) 807*16kB (UME) 358*32kB (UME) 93*64kB (UME) 23*128kB (UME) 7*256kB (ME) 1*512kB (E) 7*1024kB (M) 2*2048kB (M) 0*4096kB = 51276kB May 9 10:45:24 media kernel: Node 0 Normal: 324*4kB (M) 140*8kB (UME) 89*16kB (UME) 35*32kB (M) 3*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5152kB May 9 10:45:24 media kernel: 3537970 total pagecache pages May 9 10:45:24 media kernel: 0 pages in swap cache May 9 10:45:24 media kernel: Swap cache stats: add 0, delete 0, find 0/0 May 9 10:45:24 media kernel: Free swap = 0kB May 9 10:45:24 media kernel: Total swap = 0kB May 9 10:45:24 media kernel: 4174378 pages RAM May 9 10:45:24 media kernel: 0 pages HighMem/MovableOnly May 9 10:45:24 media kernel: 68326 pages reserved Well, I started the Array and it's doing a data rebuild now on disk6 and it has been running about an hour and about 9% done... we'll see what happens, but at least a step in right direction.

Yes, I can get to the GUI now remotely (once I disabled plugins and set Array Start/Dockers to not autostart)... I can't seem to get the full "GUI Mode" to work at all from my Physical Server with a Monitor/KB/Mouse connected... Its not even an option when I boot my Flash This is in my Flash Settings on Syslinux Configuration tab but no option on startup (on monitor connected to server). label unRAID OS GUI Mode menu default kernel /bzimage append initrd=/bzroot,/bzroot-gui *EDIT* I was able to hit tab and append the ",/bzroot-gui" on the main Unraid item and the GUI started... odd that it didn't show up on the list as it should. I also ran a xfs_repair -L on Disk 6 after this log was created this morning. Attached is a current one from Remote GUI, Array started in Maintenance Mode only... assume if I try to mount array regularly it will hang again. The issue now is when I start the array, everything grinds to a halt again, GUI becomes unresponsive... this was the last bit of the last Diag Zip that seems to be happening dozens of times per second... May 9 10:45:24 media kernel: swapper/0: page allocation failure: order:0, mode:0x2080020 May 9 10:45:24 media kernel: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.4.6-unRAID #1 May 9 10:45:24 media kernel: Hardware name: Supermicro X10SAE/X10SAE, BIOS 3.0 05/20/2015 May 9 10:45:24 media kernel: 0000000000000000 ffff88041dc03c28 ffffffff813688da 0000000000000000 May 9 10:45:24 media kernel: 0000000000000000 ffff88041dc03cc0 ffffffff810bc9b0 ffffffff818b0e38 May 9 10:45:24 media kernel: ffff88041dff9b00 ffffffffffffffff ffffffff008b0680 0000000000000000 May 9 10:45:24 media kernel: Call Trace: May 9 10:45:24 media kernel: <IRQ> [<ffffffff813688da>] dump_stack+0x61/0x7e May 9 10:45:24 media kernel: [<ffffffff810bc9b0>] warn_alloc_failed+0x10f/0x127 May 9 10:45:24 media kernel: [<ffffffff810bf9c7>] __alloc_pages_nodemask+0x870/0x8ca May 9 10:45:24 media kernel: [<ffffffff814333a9>] ? device_has_rmrr+0x5a/0x63 May 9 10:45:24 media kernel: [<ffffffff810bfabd>] __alloc_page_frag+0x9c/0x15f May 9 10:45:24 media kernel: [<ffffffff8152e310>] __napi_alloc_skb+0x61/0xc1 May 9 10:45:24 media kernel: [<ffffffffa053e92a>] igb_poll+0x441/0xc06 [igb] May 9 10:45:24 media kernel: [<ffffffff815390ac>] net_rx_action+0xd8/0x226 May 9 10:45:24 media kernel: [<ffffffff8104d4c0>] __do_softirq+0xc3/0x1b6 May 9 10:45:24 media kernel: [<ffffffff8104d73d>] irq_exit+0x3d/0x82 May 9 10:45:24 media kernel: [<ffffffff8100db9a>] do_IRQ+0xaa/0xc2 May 9 10:45:24 media kernel: [<ffffffff8161ab42>] common_interrupt+0x82/0x82 May 9 10:45:24 media kernel: <EOI> [<ffffffff815041b7>] ? cpuidle_enter_state+0xf0/0x148 May 9 10:45:24 media kernel: [<ffffffff81504170>] ? cpuidle_enter_state+0xa9/0x148 May 9 10:45:24 media kernel: [<ffffffff81504231>] cpuidle_enter+0x12/0x14 May 9 10:45:24 media kernel: [<ffffffff81076247>] call_cpuidle+0x4e/0x50 May 9 10:45:24 media kernel: [<ffffffff810763cf>] cpu_startup_entry+0x186/0x1fd May 9 10:45:24 media kernel: [<ffffffff8160fbdd>] rest_init+0x84/0x87 May 9 10:45:24 media kernel: [<ffffffff818eaec0>] start_kernel+0x3f7/0x404 May 9 10:45:24 media kernel: [<ffffffff818ea120>] ? early_idt_handler_array+0x120/0x120 May 9 10:45:24 media kernel: [<ffffffff818ea339>] x86_64_start_reservations+0x2a/0x2c May 9 10:45:24 media kernel: [<ffffffff818ea421>] x86_64_start_kernel+0xe6/0xf3 May 9 10:45:24 media kernel: Mem-Info: May 9 10:45:24 media kernel: active_anon:468687 inactive_anon:4711 isolated_anon:0 May 9 10:45:24 media kernel: active_file:443016 inactive_file:3009187 isolated_file:32 May 9 10:45:24 media kernel: unevictable:0 dirty:64349 writeback:152019 unstable:0 May 9 10:45:24 media kernel: slab_reclaimable:51705 slab_unreclaimable:30682 May 9 10:45:24 media kernel: mapped:51722 shmem:85744 pagetables:5236 bounce:0 May 9 10:45:24 media kernel: free:17874 free_pcp:104 free_cma:0 May 9 10:45:24 media kernel: Node 0 DMA free:15580kB min:12kB low:12kB high:16kB active_anon:304kB inactive_anon:16kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:320kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes May 9 10:45:24 media kernel: lowmem_reserve[]: 0 3512 16022 16022 May 9 10:45:24 media kernel: Node 0 DMA32 free:51276kB min:3524kB low:4404kB high:5284kB active_anon:572120kB inactive_anon:3316kB active_file:391584kB inactive_file:2440188kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:3607096kB managed:3597428kB mlocked:0kB dirty:61208kB writeback:129236kB mapped:48616kB shmem:74916kB slab_reclaimable:44168kB slab_unreclaimable:26384kB kernel_stack:3376kB pagetables:5800kB unstable:0kB bounce:0kB free_pcp:144kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:44 all_unreclaimable? no May 9 10:45:24 media kernel: lowmem_reserve[]: 0 0 12510 12510 May 9 10:45:24 media kernel: Node 0 Normal free:4640kB min:12564kB low:15704kB high:18844kB active_anon:1302324kB inactive_anon:15512kB active_file:1380480kB inactive_file:9596560kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13074432kB managed:12810880kB mlocked:0kB dirty:196188kB writeback:478840kB mapped:158240kB shmem:267740kB slab_reclaimable:162652kB slab_unreclaimable:96344kB kernel_stack:11968kB pagetables:15144kB unstable:0kB bounce:0kB free_pcp:272kB local_pcp:140kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no May 9 10:45:24 media kernel: lowmem_reserve[]: 0 0 0 0 May 9 10:45:24 media kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 3*64kB (UM) 2*128kB (UM) 1*256kB (U) 1*512kB (M) 2*1024kB (UM) 2*2048kB (UM) 2*4096kB (M) = 15580kB May 9 10:45:24 media kernel: Node 0 DMA32: 499*4kB (ME) 306*8kB (UME) 807*16kB (UME) 358*32kB (UME) 93*64kB (UME) 23*128kB (UME) 7*256kB (ME) 1*512kB (E) 7*1024kB (M) 2*2048kB (M) 0*4096kB = 51276kB May 9 10:45:24 media kernel: Node 0 Normal: 324*4kB (M) 140*8kB (UME) 89*16kB (UME) 35*32kB (M) 3*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5152kB May 9 10:45:24 media kernel: 3537970 total pagecache pages May 9 10:45:24 media kernel: 0 pages in swap cache May 9 10:45:24 media kernel: Swap cache stats: add 0, delete 0, find 0/0 May 9 10:45:24 media kernel: Free swap = 0kB May 9 10:45:24 media kernel: Total swap = 0kB May 9 10:45:24 media kernel: 4174378 pages RAM May 9 10:45:24 media kernel: 0 pages HighMem/MovableOnly May 9 10:45:24 media kernel: 68326 pages reserved media-diagnostics-20160509-1421.zip

I started a thread in V6 support forum but wonder if it's getting passed over since I'm running B21... http://lime-technology.com/forum/index.php?topic=48979.0 GUI is unresponsive... was working previously (updated to B21 when released). Was finally able to get GUI to boot if I disabled all plugins and set Array and Docker to not start up. GUI locked up as soon as I tried to start array. Did a fs repair on one drive, started array and then it hung again within a few seconds. Hoping to get more eyes to see if there something I'm missing. Attached Diag Zip from this morning. Please help -Sw2 media-diagnostics-20160509-0703.zip

In the Syslog, I'm seeing this over and over and over (several times every second).... GUI still hung, May 9 10:45:24 media kernel: swapper/0: page allocation failure: order:0, mode:0x2080020 May 9 10:45:24 media kernel: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.4.6-unRAID #1 May 9 10:45:24 media kernel: Hardware name: Supermicro X10SAE/X10SAE, BIOS 3.0 05/20/2015 May 9 10:45:24 media kernel: 0000000000000000 ffff88041dc03c28 ffffffff813688da 0000000000000000 May 9 10:45:24 media kernel: 0000000000000000 ffff88041dc03cc0 ffffffff810bc9b0 ffffffff818b0e38 May 9 10:45:24 media kernel: ffff88041dff9b00 ffffffffffffffff ffffffff008b0680 0000000000000000 May 9 10:45:24 media kernel: Call Trace: May 9 10:45:24 media kernel: <IRQ> [<ffffffff813688da>] dump_stack+0x61/0x7e May 9 10:45:24 media kernel: [<ffffffff810bc9b0>] warn_alloc_failed+0x10f/0x127 May 9 10:45:24 media kernel: [<ffffffff810bf9c7>] __alloc_pages_nodemask+0x870/0x8ca May 9 10:45:24 media kernel: [<ffffffff814333a9>] ? device_has_rmrr+0x5a/0x63 May 9 10:45:24 media kernel: [<ffffffff810bfabd>] __alloc_page_frag+0x9c/0x15f May 9 10:45:24 media kernel: [<ffffffff8152e310>] __napi_alloc_skb+0x61/0xc1 May 9 10:45:24 media kernel: [<ffffffffa053e92a>] igb_poll+0x441/0xc06 [igb] May 9 10:45:24 media kernel: [<ffffffff815390ac>] net_rx_action+0xd8/0x226 May 9 10:45:24 media kernel: [<ffffffff8104d4c0>] __do_softirq+0xc3/0x1b6 May 9 10:45:24 media kernel: [<ffffffff8104d73d>] irq_exit+0x3d/0x82 May 9 10:45:24 media kernel: [<ffffffff8100db9a>] do_IRQ+0xaa/0xc2 May 9 10:45:24 media kernel: [<ffffffff8161ab42>] common_interrupt+0x82/0x82 May 9 10:45:24 media kernel: <EOI> [<ffffffff815041b7>] ? cpuidle_enter_state+0xf0/0x148 May 9 10:45:24 media kernel: [<ffffffff81504170>] ? cpuidle_enter_state+0xa9/0x148 May 9 10:45:24 media kernel: [<ffffffff81504231>] cpuidle_enter+0x12/0x14 May 9 10:45:24 media kernel: [<ffffffff81076247>] call_cpuidle+0x4e/0x50 May 9 10:45:24 media kernel: [<ffffffff810763cf>] cpu_startup_entry+0x186/0x1fd May 9 10:45:24 media kernel: [<ffffffff8160fbdd>] rest_init+0x84/0x87 May 9 10:45:24 media kernel: [<ffffffff818eaec0>] start_kernel+0x3f7/0x404 May 9 10:45:24 media kernel: [<ffffffff818ea120>] ? early_idt_handler_array+0x120/0x120 May 9 10:45:24 media kernel: [<ffffffff818ea339>] x86_64_start_reservations+0x2a/0x2c May 9 10:45:24 media kernel: [<ffffffff818ea421>] x86_64_start_kernel+0xe6/0xf3 May 9 10:45:24 media kernel: Mem-Info: May 9 10:45:24 media kernel: active_anon:468687 inactive_anon:4711 isolated_anon:0 May 9 10:45:24 media kernel: active_file:443016 inactive_file:3009187 isolated_file:32 May 9 10:45:24 media kernel: unevictable:0 dirty:64349 writeback:152019 unstable:0 May 9 10:45:24 media kernel: slab_reclaimable:51705 slab_unreclaimable:30682 May 9 10:45:24 media kernel: mapped:51722 shmem:85744 pagetables:5236 bounce:0 May 9 10:45:24 media kernel: free:17874 free_pcp:104 free_cma:0 May 9 10:45:24 media kernel: Node 0 DMA free:15580kB min:12kB low:12kB high:16kB active_anon:304kB inactive_anon:16kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:320kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes May 9 10:45:24 media kernel: lowmem_reserve[]: 0 3512 16022 16022 May 9 10:45:24 media kernel: Node 0 DMA32 free:51276kB min:3524kB low:4404kB high:5284kB active_anon:572120kB inactive_anon:3316kB active_file:391584kB inactive_file:2440188kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:3607096kB managed:3597428kB mlocked:0kB dirty:61208kB writeback:129236kB mapped:48616kB shmem:74916kB slab_reclaimable:44168kB slab_unreclaimable:26384kB kernel_stack:3376kB pagetables:5800kB unstable:0kB bounce:0kB free_pcp:144kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:44 all_unreclaimable? no May 9 10:45:24 media kernel: lowmem_reserve[]: 0 0 12510 12510 May 9 10:45:24 media kernel: Node 0 Normal free:4640kB min:12564kB low:15704kB high:18844kB active_anon:1302324kB inactive_anon:15512kB active_file:1380480kB inactive_file:9596560kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13074432kB managed:12810880kB mlocked:0kB dirty:196188kB writeback:478840kB mapped:158240kB shmem:267740kB slab_reclaimable:162652kB slab_unreclaimable:96344kB kernel_stack:11968kB pagetables:15144kB unstable:0kB bounce:0kB free_pcp:272kB local_pcp:140kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no May 9 10:45:24 media kernel: lowmem_reserve[]: 0 0 0 0 May 9 10:45:24 media kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 3*64kB (UM) 2*128kB (UM) 1*256kB (U) 1*512kB (M) 2*1024kB (UM) 2*2048kB (UM) 2*4096kB (M) = 15580kB May 9 10:45:24 media kernel: Node 0 DMA32: 499*4kB (ME) 306*8kB (UME) 807*16kB (UME) 358*32kB (UME) 93*64kB (UME) 23*128kB (UME) 7*256kB (ME) 1*512kB (E) 7*1024kB (M) 2*2048kB (M) 0*4096kB = 51276kB May 9 10:45:24 media kernel: Node 0 Normal: 324*4kB (M) 140*8kB (UME) 89*16kB (UME) 35*32kB (M) 3*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5152kB May 9 10:45:24 media kernel: 3537970 total pagecache pages May 9 10:45:24 media kernel: 0 pages in swap cache May 9 10:45:24 media kernel: Swap cache stats: add 0, delete 0, find 0/0 May 9 10:45:24 media kernel: Free swap = 0kB May 9 10:45:24 media kernel: Total swap = 0kB May 9 10:45:24 media kernel: 4174378 pages RAM May 9 10:45:24 media kernel: 0 pages HighMem/MovableOnly May 9 10:45:24 media kernel: 68326 pages reserved

Will try a xfs_repair now and see what happens. Thanks! *Edit* Had to do a -L but it ran for just a few mins, and then reran with a -V and afterwards I was able to start the array... it immediately started a parity check and I have the yellow triangle next to Disk 6, so unsure exactly what is next, but parity will take 19-20 hours or so (at at least typeical time). *Edit2* Actually the GUI is completely hung again about 60 seconds into it's parity check/rebuild, whatever. Can't stop it, can't get back to GUI at all... still "dead in the water"

Help? Anybody? I'm stuck here and worried!

Any other ideas or analysis of Diags for what to try next? Confirmed that left running overnight last night (9+ hours) that the system is hung at Mounting the Disks, GUI is then unresponsive, but still have console access/SSH, etc (basically the same issue at the start expect that I was able to get to a GUI when setting Arrray Start/Docker to No and disabling all of the plugins. HELP please! I have 50+TB of stuff that I'm not excited about the possibility of losing. *Edit* attached diag Zip from this AM after it hung overnight trying to mount the disk... unsure if any new info. Had to hard power cycle the server again as the powerdown script, while now attempts to run still won't shut down the server. *Edit2* I noticed there is a 7 hour shift in time (this time 7 hours back) in the log even tough I know it was just physically powered off and back on May 9 00:09:42 media avahi-dnsconfd[2270]: Successfully connected to Avahi daemon. May 9 00:09:42 media emhttp: autostart disabled May 9 00:09:43 media avahi-daemon[2261]: Server startup complete. Host name is media.local. Local service cookie is 4271840896. May 9 00:09:44 media avahi-daemon[2261]: Service "media" (/services/ssh.service) successfully established. May 9 00:09:44 media avahi-daemon[2261]: Service "media" (/services/smb.service) successfully established. May 9 00:09:44 media avahi-daemon[2261]: Service "media" (/services/sftp-ssh.service) successfully established. May 9 07:09:59 media emhttp: shcmd (18): rmmod md-mod |& logger May 9 07:09:59 media kernel: md: unRAID driver removed May 9 07:09:59 media emhttp: shcmd (19): modprobe md-mod super=/boot/config/super.dat |& logger May 9 07:09:59 media kernel: md: unRAID driver 2.6.1 installed May 9 07:09:59 media emhttp: Pro key detected, GUID: 0781-5506-0000-173EA180319E FILE: /boot/config/Pro.key May 9 07:09:59 media emhttp: Device inventory: May 9 07:09:59 media emhttp: shcmd (20): udevadm settle May 9 07:09:59 media emhttp: SanDisk_Cruzer_0000173EA180319E-0:0 (sda) 4013860 -SW2 media-diagnostics-20160509-0703.zip

Thanks, I think something is still wonky... I stuffed Disk6 back into the array, tried to start my Array (and Disk6 was Blue balled), and it just hangs on mounting disks. I am watching the server console and see a final "XFS (md18): Ending clean mount" and then nothing after that and it's just hanging there (GUI is basically unresponsive). Can't do anything else with the console at that point and no disk activity lights on any drive in my array. I'll try to force another shutdown and do another Diag and attach but about ready to give up for the night! At least I installed the powerdown plugin first, so it "seems" like it might actually do a shut down this time, but I'm not holding my breath! *EDIT* Nope... not powering down... <sigh>, but able to map to my \\Media\Flash drive to get the new Diags Zip that the Powerdown thing created before it tried to shut down and didn't work. I've attached here. *EDIT2* Interestingly, I also see a TimeSync Error : ntpd[1800]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized I noticed my BIOS time was off by 7-8 hours (forward), when I was in there checking settings and updating BIOS. Wonder if I need a new CMOS Battery. -W media-diagnostics-20160508-2129.zip

OK, so good news....I manually renamed all the PLG files (Except Dynamix.Plg) to OLD and RM'ed the PLG's. Did the same for Network.Cfg, nothing existed in Extra Folder, I set both Docker and Start Array to "No" by VI editing those files (I'm a Windows guy so this Linux command line stuff is complex!). After all of that stuff, I was able to reboot (hard power off/on since my powerdown command line stuff still isn't working), and am able to get back into the GUI... the IP Address obviously is now back to a DHCP one instead of the static it was at before, but that's easy to fix. Now I need to figure out what's going on with that Disk6 (currently pulled out in my Windows Box running SeaTools to see if I can get a fail code or what). Then once I get that sorted, I guess I can go back to re-adding the plugin's one by one and then the docker stuff to see if I can figure out what was handing this thing. -Sw2

Thanks for the attempt... I'm kind of desperate... I tried to manually kill emhttp and rerun from the command line "/usr/local/sbin/emhttp &" (searching the forum and trying anything that might help.... Getting a "segfault at 530 ip <long string> sp <another long string> error 4 in ehmttp[<one more slightly less long string>] emhttp isn't designed to be restarted anymore Grrr.... Thanks... I was able to boot on a new USB stick (trial version)... obviously not going to mount any of the drives or whatever, but I can see that it's booted, am able to see the webconsole, etc. At least tells me that network is OK, and hopefully my Server HW... Not sure where to go from here. -W ok. Then your next step is to use the original stick and restart it in "Safe" mode If that works, then delete all the .plg files from config/plugins on the flash and start normally Then post what happens before we hit the next stage (I'm going to bed soon) If that's still not working, then I know how to transfer the configs over to a new stick, but want someone more experienced because we also have a disk disabled at the same time, and I don't want to make any mistakes that might result in data loss I already tried Safe Mode on the original stick and did exactly the same thing... I'm happy to try removing the PLG's... at this point I"ll try anything.