March 8, 201412 yr So I woke up this morning (or didn't technically) to no alarm, I use my squeezebox as an alarm, so I promptly went to check on my unraid box where squeezebox server runs. I'd been running preclear on a new disk and that console window was showing errors. The samba shares are gone, the web interface is not responding. So I begin trying to shut down manually via the console, and this is where I start running into trouble, fuser worked initially, disk1 had a lot of smbd processes tying it up, disk 7 cache dirs (find process), etc. So I've tried killing them, problem is I can't kill any of the processes anymore. I tried killall smbd but the processes won't die, that find process on disk7 won't die. At the moment fuser /mnt/disk* just hangs and doesn't return. I'd like to just forceably restart it, but I know if I do that there's a chance I'll corrupt a filesystem (I learned that the hard way a while back). Below is the end of /var/log/syslog Any help would be appreciated. Mar 8 03:45:13 unRAID kernel: kworker/2:1H: page allocation failure: order:1, mode:0x8020 Mar 8 03:45:13 unRAID kernel: Pid: 5832, comm: kworker/2:1H Not tainted 3.9.11p-unRAID #4 Mar 8 03:45:13 unRAID kernel: Call Trace: Mar 8 03:45:13 unRAID kernel: [<c1071952>] warn_alloc_failed+0xcc/0xdb Mar 8 03:45:13 unRAID kernel: [<c1076160>] ? wakeup_kswapd+0x7b/0x81 Mar 8 03:45:13 unRAID kernel: [<c1072311>] __alloc_pages_nodemask+0x50b/0x52f Mar 8 03:45:13 unRAID kernel: [<c1005c52>] dma_generic_alloc_coherent+0x69/0xd3 Mar 8 03:45:13 unRAID kernel: [<c10905bb>] pool_alloc_page+0xd0/0x135 Mar 8 03:45:13 unRAID kernel: [<c1005be9>] ? dma_generic_free_coherent+0x1e/0x1e Mar 8 03:45:13 unRAID kernel: [<c1090660>] dma_pool_alloc+0x40/0x9a Mar 8 03:45:13 unRAID kernel: [<f845f828>] mvs_task_prep+0x1a6/0x37e [mvsas] Mar 8 03:45:13 unRAID kernel: [<c1259a94>] ? cpumask_next_and+0x23/0x2f Mar 8 03:45:13 unRAID kernel: [<c106e6db>] ? mempool_alloc_slab+0xe/0x10 Mar 8 03:45:13 unRAID kernel: [<c124ca6a>] ? __blk_segment_map_sg+0x11f/0x15b Mar 8 03:45:13 unRAID kernel: [<f845fa44>] mvs_task_exec+0x44/0xa0 [mvsas] Mar 8 03:45:13 unRAID kernel: [<f84601dd>] mvs_queue_command+0x3a/0x47 [mvsas] Mar 8 03:45:13 unRAID kernel: [<f8442343>] sas_ata_qc_issue+0x19e/0x1f8 [libsas] Mar 8 03:45:13 unRAID kernel: [<c1308190>] ata_qc_issue+0x27c/0x29a Mar 8 03:45:13 unRAID kernel: [<c130bda1>] ata_scsi_translate+0xc4/0xf2 Mar 8 03:45:13 unRAID kernel: [<c130dd7d>] ? ata_scsiop_mode_sense+0x28f/0x28f Mar 8 03:45:13 unRAID kernel: [<c130e880>] ata_sas_queuecmd+0x1a4/0x1d2 Mar 8 03:45:13 unRAID kernel: [<f84411fa>] sas_queuecommand+0x79/0x1bf [libsas] Mar 8 03:45:13 unRAID kernel: [<c12f74b9>] scsi_dispatch_cmd+0xfa/0x125 Mar 8 03:45:13 unRAID kernel: [<c12fba88>] scsi_request_fn+0x253/0x371 Mar 8 03:45:13 unRAID kernel: [<c124756a>] ? blk_run_queue+0x28/0x2c Mar 8 03:45:13 unRAID kernel: [<c124721b>] __blk_run_queue+0x28/0x31 Mar 8 03:45:13 unRAID kernel: [<c1248325>] blk_delay_work+0x19/0x26 Mar 8 03:45:13 unRAID kernel: [<c1039bdb>] process_one_work+0x1cb/0x2cc Mar 8 03:45:13 unRAID kernel: [<c103a170>] worker_thread+0x1c3/0x2f8 Mar 8 03:45:13 unRAID kernel: [<c103ebf1>] kthread+0x90/0x95 Mar 8 03:45:13 unRAID kernel: [<c1039fad>] ? manage_workers+0x89/0x89 Mar 8 03:45:13 unRAID kernel: [<c14013b7>] ret_from_kernel_thread+0x1b/0x28 Mar 8 03:45:13 unRAID kernel: [<c103eb61>] ? kthread_freezable_should_stop+0x4a/0x4a Mar 8 03:45:13 unRAID kernel: Mem-Info: Mar 8 03:45:13 unRAID kernel: DMA per-cpu: Mar 8 03:45:13 unRAID kernel: CPU 0: hi: 0, btch: 1 usd: 0 Mar 8 03:45:13 unRAID kernel: CPU 1: hi: 0, btch: 1 usd: 0 Mar 8 03:45:13 unRAID kernel: CPU 2: hi: 0, btch: 1 usd: 0 Mar 8 03:45:13 unRAID kernel: CPU 3: hi: 0, btch: 1 usd: 0 Mar 8 03:45:13 unRAID kernel: Normal per-cpu: Mar 8 03:45:13 unRAID kernel: CPU 0: hi: 186, btch: 31 usd: 26 Mar 8 03:45:13 unRAID kernel: CPU 1: hi: 186, btch: 31 usd: 179 Mar 8 03:45:13 unRAID kernel: CPU 2: hi: 186, btch: 31 usd: 46 Mar 8 03:45:13 unRAID kernel: CPU 3: hi: 186, btch: 31 usd: 193 Mar 8 03:45:13 unRAID kernel: HighMem per-cpu: Mar 8 03:45:13 unRAID kernel: CPU 0: hi: 186, btch: 31 usd: 24 Mar 8 03:45:13 unRAID kernel: CPU 1: hi: 186, btch: 31 usd: 56 Mar 8 03:45:13 unRAID kernel: CPU 2: hi: 186, btch: 31 usd: 21 Mar 8 03:45:13 unRAID kernel: CPU 3: hi: 186, btch: 31 usd: 42 Mar 8 03:45:13 unRAID kernel: active_anon:26838 inactive_anon:128 isolated_anon:0 Mar 8 03:45:13 unRAID kernel: active_file:278327 inactive_file:521720 isolated_file:0 Mar 8 03:45:13 unRAID kernel: unevictable:104083 dirty:11190 writeback:9672 unstable:0 Mar 8 03:45:13 unRAID kernel: free:36637 slab_reclaimable:44007 slab_unreclaimable:5997 Mar 8 03:45:13 unRAID kernel: mapped:3902 shmem:173 pagetables:327 bounce:0 Mar 8 03:45:13 unRAID kernel: free_cma:0 Mar 8 03:45:13 unRAID kernel: DMA free:3220kB min:68kB low:84kB high:100kB active_anon:0kB inactive_anon:0kB active_file:8188kB inactive_file:372kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15880kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:4004kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 8 03:45:13 unRAID kernel: lowmem_reserve[]: 0 797 4000 4000 Mar 8 03:45:13 unRAID kernel: Normal free:5220kB min:3576kB low:4468kB high:5364kB active_anon:588kB inactive_anon:0kB active_file:226472kB inactive_file:359076kB unevictable:504kB isolated(anon):0kB isolated(file):128kB present:897016kB managed:816224kB mlocked:0kB dirty:38680kB writeback:35436kB mapped:508kB shmem:0kB slab_reclaimable:172024kB slab_unreclaimable:23924kB kernel_stack:1360kB pagetables:44kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no Mar 8 03:45:13 unRAID kernel: lowmem_reserve[]: 0 0 25628 25628 Mar 8 03:45:13 unRAID kernel: HighMem free:138108kB min:512kB low:4104kB high:7700kB active_anon:106764kB inactive_anon:512kB active_file:878648kB inactive_file:1727304kB unevictable:415828kB isolated(anon):0kB isolated(file):0kB present:3280456kB managed:3280456kB mlocked:0kB dirty:6080kB writeback:3252kB mapped:15100kB shmem:692kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:1264kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 8 03:45:13 unRAID kernel: lowmem_reserve[]: 0 0 0 0 Mar 8 03:45:13 unRAID kernel: DMA: 1*4kB (R) 6*8kB (R) 2*16kB (R) 0*32kB 1*64kB (R) 0*128kB 0*256kB 0*512kB 1*1024kB (R) 1*2048kB (R) 0*4096kB = 3220kB Mar 8 03:45:13 unRAID kernel: Normal: 1052*4kB (UM) 132*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5264kB Mar 8 03:45:13 unRAID kernel: HighMem: 1*4kB (M) 5955*8kB (UM) 5142*16kB (UM) 160*32kB (UM) 10*64kB (UM) 11*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 138108kB Mar 8 03:45:13 unRAID kernel: 904359 total pagecache pages Mar 8 03:45:13 unRAID kernel: 0 pages in swap cache Mar 8 03:45:13 unRAID kernel: Swap cache stats: add 0, delete 0, find 0/0 Mar 8 03:45:13 unRAID kernel: Free swap = 0kB Mar 8 03:45:13 unRAID kernel: Total swap = 0kB Mar 8 03:45:13 unRAID kernel: 1310719 pages RAM Mar 8 03:45:13 unRAID kernel: 1082370 pages HighMem Mar 8 03:45:13 unRAID kernel: 274363 pages reserved Mar 8 03:45:13 unRAID kernel: 1474159 pages shared Mar 8 03:45:13 unRAID kernel: 454053 pages non-shared Mar 8 03:45:13 unRAID kernel: mvsas 0000:01:00.0: mvsas prep failed[0]! Mar 8 03:45:43 unRAID kernel: sd 1:0:3:0: [sdk] command f740f780 timed out Mar 8 03:45:43 unRAID kernel: sas: Enter sas_scsi_recover_host busy: 1 failed: 1 Mar 8 03:45:43 unRAID kernel: sas: trying to find task 0xda74c000 Mar 8 03:45:43 unRAID kernel: sas: sas_scsi_find_task: aborting task 0xda74c000 Mar 8 03:45:43 unRAID kernel: BUG: unable to handle kernel NULL pointer dereference at 00000008 Mar 8 03:45:43 unRAID kernel: IP: [<f845e3b0>] mvs_slot_task_free+0xf/0x139 [mvsas] Mar 8 03:45:43 unRAID kernel: *pdpt = 0000000006419001 *pde = 0000000000000000 Mar 8 03:45:43 unRAID kernel: Oops: 0000 [#1] SMP Mar 8 03:45:43 unRAID kernel: Modules linked in: md_mod coretemp hwmon sg mperf i2c_i801 i2c_core ahci libahci e1000e ptp pps_core mvsas libsas scsi_transport_sas [last unloaded: md_mod] Mar 8 03:45:43 unRAID kernel: Pid: 786, comm: scsi_eh_1 Not tainted 3.9.11p-unRAID #4 Supermicro X7SPA-HF/X7SPA-HF Mar 8 03:45:43 unRAID kernel: EIP: 0060:[<f845e3b0>] EFLAGS: 00010282 CPU: 0 Mar 8 03:45:43 unRAID kernel: EIP is at mvs_slot_task_free+0xf/0x139 [mvsas] Mar 8 03:45:43 unRAID kernel: EAX: f76a0000 EBX: 00000000 ECX: 00000000 EDX: da74c000 Mar 8 03:45:43 unRAID kernel: ESI: da74c000 EDI: f76a0000 EBP: f7635e58 ESP: f7635e38 Mar 8 03:45:43 unRAID kernel: DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 Mar 8 03:45:43 unRAID kernel: CR0: 8005003b CR2: 00000008 CR3: 15db8000 CR4: 000007f0 Mar 8 03:45:43 unRAID kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 Mar 8 03:45:43 unRAID kernel: DR6: ffff0ff0 DR7: 00000400 Mar 8 03:45:43 unRAID kernel: Process scsi_eh_1 (pid: 786, ti=f7634000 task=f7535b00 task.ti=f7634000) Mar 8 03:45:43 unRAID kernel: Stack: Mar 8 03:45:43 unRAID kernel: f7635e74 00000246 00000000 00000000 00000000 da74c000 da74c00c f76a1f64 Mar 8 03:45:43 unRAID kernel: f7635e94 f845fe10 278cb590 00000000 da74c000 f2dc4000 da74c000 f2dcb000 Mar 8 03:45:43 unRAID kernel: f76a0000 00000000 00000007 00000000 da74c000 f2dc4000 da74c000 f7635ec4 Mar 8 03:45:43 unRAID kernel: Call Trace: Mar 8 03:45:43 unRAID kernel: [<f845fe10>] mvs_abort_task+0x170/0x1ab [mvsas] Mar 8 03:45:43 unRAID kernel: [<f8440124>] sas_scsi_find_task+0xdd/0x1fd [libsas] Mar 8 03:45:43 unRAID kernel: [<c13feba8>] ? printk+0x38/0x3a Mar 8 03:45:43 unRAID kernel: [<f8440aea>] sas_eh_handle_sas_errors+0x114/0x4da [libsas] Mar 8 03:45:43 unRAID kernel: [<f8441085>] sas_scsi_recover_host+0xa9/0x161 [libsas] Mar 8 03:45:43 unRAID kernel: [<c12fabc8>] scsi_error_handler+0x6c/0xb0 Mar 8 03:45:43 unRAID kernel: [<c103ebf1>] kthread+0x90/0x95 Mar 8 03:45:43 unRAID kernel: [<c12fab5c>] ? scsi_unjam_host+0x152/0x152 Mar 8 03:45:43 unRAID kernel: [<c14013b7>] ret_from_kernel_thread+0x1b/0x28 Mar 8 03:45:43 unRAID kernel: [<c103eb61>] ? kthread_freezable_should_stop+0x4a/0x4a Mar 8 03:45:43 unRAID kernel: Code: 41 10 b9 00 00 02 00 89 04 24 89 d8 ff 96 c0 00 00 00 31 c0 83 c4 34 5b 5e 5f 5d c3 55 89 e5 57 89 c7 56 89 d6 53 89 cb 83 ec 14 <83> 79 08 00 0f 84 18 01 00 00 f6 42 14 05 75 48 8b 49 0c 85 c9 Mar 8 03:45:43 unRAID kernel: EIP: [<f845e3b0>] mvs_slot_task_free+0xf/0x139 [mvsas] SS:ESP 0068:f7635e38 Mar 8 03:45:43 unRAID kernel: CR2: 0000000000000008 Mar 8 03:45:43 unRAID kernel: ---[ end trace d91e447d8413ebac ]--- Mar 8 04:41:34 unRAID kernel: mdcmd (49): spindown 9 Mar 8 04:42:55 unRAID kernel: mdcmd (50): spindown 3 Mar 8 04:43:35 unRAID kernel: mdcmd (51): spindown 6 Mar 8 04:45:16 unRAID kernel: mdcmd (52): spindown 0 Mar 8 04:45:17 unRAID kernel: mdcmd (53): spindown 2 Mar 8 04:45:27 unRAID emhttp: shcmd (73): /usr/sbin/hdparm -y /dev/sdl &> /dev/null Mar 8 09:39:51 unRAID sshd[5666]: Accepted password for root from 192.168.0.5 port 61644 ssh2 Mar 8 09:39:51 unRAID sshd[5795]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory Mar 8 09:39:51 unRAID sshd[5795]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory Mar 8 09:41:48 unRAID sshd[6926]: Accepted password for root from 192.168.0.5 port 61684 ssh2 Mar 8 09:41:48 unRAID sshd[7016]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory Mar 8 09:41:48 unRAID sshd[7016]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory Mar 8 09:42:15 unRAID cache_dirs: killing cache_dirs process 6097 Mar 8 09:42:29 unRAID logger: /usr/local/sbin/mover: line 44: 15583 Killed find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/user0/ \; -delete Mar 8 09:42:29 unRAID logger: mover finished Mar 8 09:48:10 unRAID sshd[7344]: Accepted password for root from 192.168.0.5 port 61715 ssh2 Mar 8 09:48:10 unRAID sshd[7348]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory Mar 8 09:48:10 unRAID sshd[7348]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory Mar 8 10:01:31 unRAID sshd[7404]: Accepted password for root from 192.168.0.5 port 61788 ssh2 Mar 8 10:01:31 unRAID sshd[7408]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory root@unRAID:~#
March 8, 201412 yr Author Right now, after about 45 minutes, "lsof /mnt/disk*" hasn't returned, nor printed anything.
March 8, 201412 yr Author Same thing, just sits there and hangs, even "ls /mnt" hangs, actually that hangs the whole console and I can't even ctrl+c that. I'm kind of thinking I'm just going to have to restart it and hope for the best. -edit For better or worse, since the system was essentially unresponsive I reset it and ran a pass of Memtest on the box and that passed. I've disabled (renamed) the plugins/extras/packages and reverted back to stock go file. Rebooted and all looks OK, all the disks seemed to mount OK. There was a warning about the USB stick not being cleanly shut down, so I scanned that in my laptop and that seems OK now. So I think we're all none the worse for wear. But time will tell.
Archived
This topic is now archived and is closed to further replies.