October 27, 201312 yr Everything was cruising along fine for a few months and then not so much. I had a drive show a few write errors, things were copying really slow. I pulled the drive and used UFS explorer to get my data off. I pulled that drive out and started my parity synch, new config. Everything went fine last night doing around 50mb/S to 80mb/s until today it dropped down to 2.6mb/s when it got to 88%. Obviously something is really wrong. I am using vSphere 5.1 with passthru on the Supermicro cards, the AOC ones. Any idea what is happening is this some type of interrupt issue? Let me post my log.... Oct 27 08:15:31 Hippocampus kernel: mdcmd (65): spindown 11 Oct 27 08:15:32 Hippocampus kernel: mdcmd (66): spindown 12 Oct 27 08:15:33 Hippocampus kernel: mdcmd (67): spindown 13 Oct 27 10:28:15 Hippocampus kernel: mdcmd (68): spindown 7 Oct 27 10:28:16 Hippocampus kernel: mdcmd (69): spindown 8 Oct 27 10:28:19 Hippocampus kernel: mdcmd (70): spindown 9 Oct 27 10:28:19 Hippocampus kernel: mdcmd (71): spindown 10 Oct 27 10:28:20 Hippocampus kernel: mdcmd (72): spindown 11 Oct 27 10:28:21 Hippocampus kernel: mdcmd (73): spindown 12 Oct 27 10:28:22 Hippocampus kernel: mdcmd (74): spindown 13 Oct 27 10:37:30 Hippocampus kernel: irq 19: nobody cared (try booting with the "irqpoll" option) Oct 27 10:37:30 Hippocampus kernel: Pid: 0, comm: swapper/0 Not tainted 3.4.36-unRAID #1 Oct 27 10:37:30 Hippocampus kernel: Call Trace: Oct 27 10:37:30 Hippocampus kernel: [<c1054ede>] __report_bad_irq+0x29/0xb4 Oct 27 10:37:30 Hippocampus kernel: [<c10550a0>] note_interrupt+0x137/0x1ac Oct 27 10:37:30 Hippocampus kernel: [<c10087a7>] ? default_idle+0x1c/0x2c Oct 27 10:37:30 Hippocampus kernel: [<c10536bc>] handle_irq_event_percpu+0x109/0x11a Oct 27 10:37:30 Hippocampus kernel: [<c1055543>] ? handle_percpu_irq+0x3b/0x3b Oct 27 10:37:30 Hippocampus kernel: [<c10536f2>] handle_irq_event+0x25/0x3c Oct 27 10:37:30 Hippocampus kernel: [<c1055543>] ? handle_percpu_irq+0x3b/0x3b Oct 27 10:37:30 Hippocampus kernel: [<c10555b0>] handle_fasteoi_irq+0x6d/0xab Oct 27 10:37:30 Hippocampus kernel: <IRQ> [<c10036c3>] ? do_IRQ+0x37/0x9b Oct 27 10:37:30 Hippocampus kernel: [<c10284d6>] ? irq_exit+0x56/0x58 Oct 27 10:37:30 Hippocampus kernel: [<c10180e9>] ? smp_apic_timer_interrupt+0x72/0x7f Oct 27 10:37:30 Hippocampus kernel: [<c1343369>] ? common_interrupt+0x29/0x30 Oct 27 10:37:30 Hippocampus kernel: [<c10087a7>] ? default_idle+0x1c/0x2c Oct 27 10:37:30 Hippocampus kernel: [<c1008986>] ? cpu_idle+0x4b/0x65 Oct 27 10:37:30 Hippocampus kernel: [<c1331258>] ? rest_init+0x58/0x5a Oct 27 10:37:30 Hippocampus kernel: [<c147a80b>] ? start_kernel+0x2ab/0x2b1 Oct 27 10:37:30 Hippocampus kernel: [<c147a3db>] ? kernel_init+0xf8/0xf8 Oct 27 10:37:30 Hippocampus kernel: [<c147a0a8>] ? i386_start_kernel+0xa8/0xaf Oct 27 10:37:30 Hippocampus kernel: handlers: Oct 27 10:37:30 Hippocampus kernel: [<f846309d>] mvs_interrupt Oct 27 10:37:30 Hippocampus kernel: [<f84eeb1b>] e1000_intr Oct 27 10:37:30 Hippocampus kernel: Disabling IRQ #19 Oct 27 11:36:49 Hippocampus kernel: mdcmd (75): spindown 4 Oct 27 11:36:50 Hippocampus kernel: mdcmd (76): spindown 5 Oct 27 11:36:50 Hippocampus kernel: mdcmd (77): spindown 6 Oct 27 11:55:32 Hippocampus kernel: irq 18: nobody cared (try booting with the "irqpoll" option) Oct 27 11:55:32 Hippocampus kernel: Pid: 1326, comm: unraidd Not tainted 3.4.36-unRAID #1 Oct 27 11:55:32 Hippocampus kernel: Call Trace: Oct 27 11:55:32 Hippocampus kernel: [<c1054ede>] __report_bad_irq+0x29/0xb4 Oct 27 11:55:32 Hippocampus kernel: [<c10550a0>] note_interrupt+0x137/0x1ac Oct 27 11:55:32 Hippocampus kernel: [<c10856c6>] ? __slab_free+0x6e/0x28c Oct 27 11:55:32 Hippocampus kernel: [<c10536bc>] handle_irq_event_percpu+0x109/0x11a Oct 27 11:55:32 Hippocampus kernel: [<c101897e>] ? io_apic_modify_irq+0x3f/0x4c Oct 27 11:55:32 Hippocampus kernel: [<c1055543>] ? handle_percpu_irq+0x3b/0x3b Oct 27 11:55:32 Hippocampus kernel: [<c10536f2>] handle_irq_event+0x25/0x3c Oct 27 11:55:32 Hippocampus kernel: [<c1055543>] ? handle_percpu_irq+0x3b/0x3b Oct 27 11:55:32 Hippocampus kernel: [<c10555b0>] handle_fasteoi_irq+0x6d/0xab Oct 27 11:55:32 Hippocampus kernel: <IRQ> [<c10036c3>] ? do_IRQ+0x37/0x9b Oct 27 11:55:32 Hippocampus kernel: [<c10284d6>] ? irq_exit+0x56/0x58 Oct 27 11:55:32 Hippocampus kernel: [<c10180e9>] ? smp_apic_timer_interrupt+0x72/0x7f Oct 27 11:55:32 Hippocampus kernel: [<c1343369>] ? common_interrupt+0x29/0x30 Oct 27 11:55:32 Hippocampus kernel: [<c10856c6>] ? __slab_free+0x6e/0x28c Oct 27 11:55:32 Hippocampus kernel: [<f85aff8e>] ? md_wakeup_thread+0x4c/0x51 [md_mod] Oct 27 11:55:32 Hippocampus kernel: [<f85b3c16>] ? end_request+0x64/0x6b [md_mod] Oct 27 11:55:32 Hippocampus kernel: [<c108629f>] ? kmem_cache_free+0x86/0x8e Oct 27 11:55:32 Hippocampus kernel: [<c1062d2f>] ? mempool_free_slab+0xe/0x10 Oct 27 11:55:32 Hippocampus last message repeated 2 times Oct 27 11:55:32 Hippocampus kernel: [<c1062dcd>] ? mempool_free+0x64/0x69 Oct 27 11:55:32 Hippocampus kernel: [<c12457fb>] ? scsi_sg_free+0x3f/0x42 Oct 27 11:55:32 Hippocampus kernel: [<c11bbe93>] ? __sg_free_table+0x47/0x5e Oct 27 11:55:32 Hippocampus kernel: [<c12457bc>] ? scsi_init_sgtable+0x7b/0x7b Oct 27 11:55:32 Hippocampus kernel: [<c1245861>] ? __scsi_release_buffers+0x22/0x96 Oct 27 11:55:32 Hippocampus kernel: [<c1245c49>] ? scsi_end_request+0x5f/0x70 Oct 27 11:55:32 Hippocampus kernel: [<c1245f76>] ? scsi_io_completion+0x1b0/0x3fd Oct 27 11:55:32 Hippocampus kernel: [<c1245d0d>] ? scsi_device_unbusy+0x7c/0x82 Oct 27 11:55:32 Hippocampus kernel: [<c1240e2e>] ? scsi_finish_command+0x9d/0xa3 Oct 27 11:55:32 Hippocampus kernel: [<c1246297>] ? scsi_softirq_done+0xc5/0xcd Oct 27 11:55:32 Hippocampus kernel: [<c11a5762>] ? blk_done_softirq+0x4a/0x57 Oct 27 11:55:32 Hippocampus kernel: [<c1028669>] ? __do_softirq+0x6b/0xfa Oct 27 11:55:32 Hippocampus kernel: [<c10285fe>] ? irq_enter+0x41/0x41 Oct 27 11:55:32 Hippocampus kernel: <IRQ> [<c10284b2>] ? irq_exit+0x32/0x58 Oct 27 11:55:32 Hippocampus kernel: [<c1003713>] ? do_IRQ+0x87/0x9b Oct 27 11:55:32 Hippocampus kernel: [<c1343369>] ? common_interrupt+0x29/0x30 Oct 27 11:55:32 Hippocampus kernel: [<c11a0a1c>] ? drive_stat_acct+0x56/0x10e Oct 27 11:55:32 Hippocampus kernel: [<c11a0fa3>] ? bio_attempt_back_merge+0x57/0x5e Oct 27 11:55:32 Hippocampus kernel: [<c11a231f>] ? blk_queue_bio+0x96/0x245 Oct 27 11:55:32 Hippocampus kernel: [<c11a0e64>] ? generic_make_request+0x7b/0xb0 Oct 27 11:55:32 Hippocampus kernel: [<f85b3a91>] ? handle_stripe+0xc91/0xcf6 [md_mod] Oct 27 11:55:32 Hippocampus kernel: [<c103d16d>] ? __wake_up+0x3b/0x42 Oct 27 11:55:32 Hippocampus kernel: [<f85b3b6e>] ? unraidd+0x78/0xbc [md_mod] Oct 27 11:55:32 Hippocampus kernel: [<f85b0b7e>] ? md_thread+0xcc/0xe3 [md_mod] Oct 27 11:55:32 Hippocampus kernel: [<c1037a8d>] ? wake_up_bit+0x5b/0x5b Oct 27 11:55:32 Hippocampus kernel: [<f85b0ab2>] ? import_device+0x166/0x166 [md_mod] Oct 27 11:55:32 Hippocampus kernel: [<c1037794>] ? kthread+0x67/0x6c Oct 27 11:55:32 Hippocampus kernel: [<c103772d>] ? kthread_freezable_should_stop+0x49/0x49 Oct 27 11:55:32 Hippocampus kernel: [<c1343376>] ? kernel_thread_helper+0x6/0xd Oct 27 11:55:32 Hippocampus kernel: handlers: Oct 27 11:55:32 Hippocampus kernel: [<c1269177>] usb_hcd_irq Oct 27 11:55:32 Hippocampus kernel: [<f846309d>] mvs_interrupt Oct 27 11:55:32 Hippocampus kernel: Disabling IRQ #18 Oct 27 12:57:49 Hippocampus kernel: mdcmd (78): spindown 6 Oct 27 12:57:59 Hippocampus kernel: mdcmd (79): spindown 7 Oct 27 12:58:00 Hippocampus kernel: mdcmd (80): spindown 8 Oct 27 13:26:35 Hippocampus kernel: mdcmd (81): spindown 4 Oct 27 14:21:37 Hippocampus in.telnetd[1928]: connect from 192.168.0.53 (192.168.0.53) Oct 27 14:21:40 Hippocampus login[1929]: ROOT LOGIN on '/dev/pts/0' from '192.168.0.53' Apparently from reading other threads, this is indeed caused from the interrupt shutting down, the card will get slow after. Here is my cat /proc/interrupts. root@Hippocampus:~# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 24 0 0 0 IO-APIC-edge timer 1: 9 0 0 0 IO-APIC-edge i8042 6: 0 2 0 1 IO-APIC-edge floppy 7: 0 0 0 0 IO-APIC-edge parport0 9: 0 0 0 0 IO-APIC-fasteoi acpi 12: 4 0 0 0 IO-APIC-edge i8042 14: 55 0 0 0 IO-APIC-edge ide0 15: 0 0 0 0 IO-APIC-edge ide1 16: 781 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1 18: 150504219 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2, mvs as 19: 175684771 0 17797 2 IO-APIC-fasteoi mvsas, eth0 40: 0 0 0 0 PCI-MSI-edge PCIe PME 41: 0 0 0 0 PCI-MSI-edge PCIe PME 42: 0 0 0 0 PCI-MSI-edge PCIe PME 43: 0 0 0 0 PCI-MSI-edge PCIe PME 44: 0 0 0 0 PCI-MSI-edge PCIe PME 45: 0 0 0 0 PCI-MSI-edge PCIe PME 46: 0 0 0 0 PCI-MSI-edge PCIe PME 47: 0 0 0 0 PCI-MSI-edge PCIe PME 48: 0 0 0 0 PCI-MSI-edge PCIe PME 49: 0 0 0 0 PCI-MSI-edge PCIe PME 50: 0 0 0 0 PCI-MSI-edge PCIe PME 51: 0 0 0 0 PCI-MSI-edge PCIe PME 52: 0 0 0 0 PCI-MSI-edge PCIe PME 53: 0 0 0 0 PCI-MSI-edge PCIe PME 54: 0 0 0 0 PCI-MSI-edge PCIe PME 55: 0 0 0 0 PCI-MSI-edge PCIe PME 56: 0 0 0 0 PCI-MSI-edge PCIe PME 57: 0 0 0 0 PCI-MSI-edge PCIe PME 58: 0 0 0 0 PCI-MSI-edge PCIe PME 59: 0 0 0 0 PCI-MSI-edge PCIe PME 60: 0 0 0 0 PCI-MSI-edge PCIe PME 61: 0 0 0 0 PCI-MSI-edge PCIe PME 62: 0 0 0 0 PCI-MSI-edge PCIe PME 63: 0 0 0 0 PCI-MSI-edge PCIe PME 64: 0 0 0 0 PCI-MSI-edge PCIe PME 65: 0 0 0 0 PCI-MSI-edge PCIe PME 66: 0 0 0 0 PCI-MSI-edge PCIe PME 67: 0 0 0 0 PCI-MSI-edge PCIe PME 68: 0 0 0 0 PCI-MSI-edge PCIe PME 69: 0 0 0 0 PCI-MSI-edge PCIe PME 70: 0 0 0 0 PCI-MSI-edge PCIe PME 71: 0 0 0 0 PCI-MSI-edge PCIe PME NMI: 0 0 0 0 Non-maskable interrupts LOC: 5661755 5661768 5661767 5661767 Local timer interrupts SPU: 0 0 0 0 Spurious interrupts PMI: 0 0 0 0 Performance monitoring interrupts IWI: 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 APIC ICR read retries RES: 64236 7181942 7220749 4159202 Rescheduling interrupts CAL: 18 23 17 21 Function call interrupts TLB: 730 631 606 588 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 Machine check exceptions MCP: 189 189 189 189 Machine check polls
November 6, 201312 yr Author Well I am going in circles fast. I figured I would repost here what I have tried. Thinking it could be a problem with an IRQ conflict with the USB ports, I disabled the USB port in vSphere for that VM right after the flash drive is detected to get licensing info. I also tried the irqpoll option with no help. I also disabled the drives from powering down, discussed in a different post. Tonight I am going to edit the syslinux.cfg and try the noirqdebug option, something I found about by digging around Google. I am at a loss here, this server handles so many other things I cannot just reboot all the VM's at whim. It was working for months until a drive died, I am not doing anything differently. It just drops down to 2mb/s during rebuilds.
November 8, 201312 yr Author 11-07-2013 I finally got a better error today from the syslog, this actually shined light on the problem. 16: 1039 IO-APIC-fasteoi ehci_hcd:usb1 18: 127900096 IO-APIC-fasteoi uhci_hcd:usb2, mvsas 19: 198900900 IO-APIC-fasteoi mvsas, eth0 So the onboard NIC is sharing an interrupt with the SAS card, same with the network card. 11-08-2013 I changed the virtual NIC from e1000 to VMXnet3 and got a little magic. You will notice interrupt 19 is no longer shared with that card. On a side not I also disabled all 4 onboard NIC's before this, added a NIC card and that made zero difference on either the host or guest side. So far changing the NIC driver on the guest was the only thing that stopped the sharing on that card. 16: 1019 IO-APIC-fasteoi ehci_hcd:usb1 18: 1549695 IO-APIC-fasteoi uhci_hcd:usb2, mvsas 19: 2295123 IO-APIC-fasteoi mvsas 11-09-2013 I had added noapic and nolapic to the syslinux.cfg at the same time, it was able to finish the sync but when I got up this morning both IRQ 18 and IRQ 19 were disabled.
November 11, 201312 yr Author Well I did find a solution, buying the IBM Serveraid controller off of Amazon and flashing it with the M1015 IT mode firmware. I have the second card coming later this week, but it appears just swapping one card fixed both.
Archived
This topic is now archived and is closed to further replies.