Unraid crashing. Can't identify cause.

[drawmonster]

So I've had a two full Unraid crashes/hangs in the last 3 days. I'm not getting any errors or warnings from Unraid and the physical server keeps running. Unraid just hangs. I have to restart the server, and it will come back up and work fine. It will also try to start a parity check after the reboot.

 

Checking the monitor I have hooked up to the physical server at time of the hang/crash just shows Unraid scrolling errors, getting to the root login, then scrolling more errors. It's a loop. Luckily I had my system log mirrored to my other server at the time of this latest crash.

 

References to "dma" and "ata" in the log makes me think it might be some sort of SAS controller issue? I am using the onboard Broadcom 2308 SAS2 controller on my mobo that then hits the SAS expander in my Supermicro case.

 

Syslog from the crash below and attached system diagnostics I got after it the server was restarted and running normally again.

Dec 22 11:18:43 Unraid kernel: ------------[ cut here ]------------
Dec 22 11:18:43 Unraid kernel: kernel BUG at include/linux/dma-mapping.h:268!
Dec 22 11:18:43 Unraid kernel: invalid opcode: 0000 [#1] SMP PTI
Dec 22 11:18:43 Unraid kernel: CPU: 18 PID: 2214 Comm: kworker/18:1H Not tainted 4.19.88-Unraid #1
Dec 22 11:18:43 Unraid kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.3 08/23/2018
Dec 22 11:18:43 Unraid kernel: Workqueue: kblockd blk_mq_run_work_fn
Dec 22 11:18:43 Unraid kernel: RIP: 0010:ata_qc_issue+0xfc/0x195
Dec 22 11:18:43 Unraid kernel: Code: 8b b5 a0 00 00 00 48 8b b8 40 35 00 00 48 85 ff 74 0c 48 8b 87 10 02 00 00 48 85 c0 75 07 48 8b 05 64 f3 9b 00 83 f9 02 76 02 <0f> 0b 45 31 c0 48 8b 40 30 e8 b1 5e 5a 00 85 c0 79 02 0f 0b 74 6b
Dec 22 11:18:43 Unraid kernel: RSP: 0018:ffffc90007023cf0 EFLAGS: 00010002
Dec 22 11:18:43 Unraid kernel: RAX: ffffffff81c678c0 RBX: ffff888842460000 RCX: 0000000000000003
Dec 22 11:18:43 Unraid kernel: RDX: 0000000000000001 RSI: ffff8888398a3bd0 RDI: ffff88905b33d0a8
Dec 22 11:18:43 Unraid kernel: RBP: ffff8888424606d0 R08: 0000000000000087 R09: 0000000000000000
Dec 22 11:18:43 Unraid kernel: R10: 0000000000000200 R11: fefefefefefefeff R12: ffff888842462040
Dec 22 11:18:43 Unraid kernel: R13: ffff8888594cd800 R14: ffff888842460000 R15: ffff8888398a3900
Dec 22 11:18:43 Unraid kernel: FS:  0000000000000000(0000) GS:ffff88905f800000(0000) knlGS:0000000000000000
Dec 22 11:18:43 Unraid kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 22 11:18:43 Unraid kernel: CR2: 0000145d01098ff8 CR3: 0000000001e0a002 CR4: 00000000001606e0
Dec 22 11:18:43 Unraid kernel: Call Trace:
Dec 22 11:18:43 Unraid kernel: ? ata_scsi_verify_xlat+0x22d/0x22d
Dec 22 11:18:43 Unraid kernel: ata_scsi_translate+0xdd/0x14d
Dec 22 11:18:43 Unraid kernel: ata_scsi_queuecmd+0x254/0x2a8
Dec 22 11:18:43 Unraid kernel: scsi_dispatch_cmd+0xa2/0xca
Dec 22 11:18:43 Unraid kernel: scsi_queue_rq+0x395/0x447
Dec 22 11:18:43 Unraid kernel: blk_mq_dispatch_rq_list+0x2b9/0x491
Dec 22 11:18:43 Unraid kernel: ? __switch_to_asm+0x35/0x70
Dec 22 11:18:43 Unraid kernel: ? __switch_to_asm+0x35/0x70
Dec 22 11:18:43 Unraid kernel: ? __switch_to_asm+0x41/0x70
Dec 22 11:18:43 Unraid kernel: ? syscall_return_via_sysret+0x14/0x83
Dec 22 11:18:43 Unraid kernel: blk_mq_sched_dispatch_requests+0xe6/0x14b
Dec 22 11:18:43 Unraid kernel: __blk_mq_run_hw_queue+0xaf/0xd6
Dec 22 11:18:43 Unraid kernel: process_one_work+0x16e/0x24f
Dec 22 11:18:43 Unraid kernel: worker_thread+0x1e2/0x2b8
Dec 22 11:18:43 Unraid kernel: ? rescuer_thread+0x29e/0x29e
Dec 22 11:18:43 Unraid kernel: kthread+0x10c/0x114
Dec 22 11:18:43 Unraid kernel: ? kthread_park+0x89/0x89
Dec 22 11:18:43 Unraid kernel: ret_from_fork+0x35/0x40
Dec 22 11:18:43 Unraid kernel: Modules linked in: tun veth xt_nat ipt_MASQUERADE iptable_filter iptable_nat nf_nat_ipv4 nf_nat ip_tables xfs nfsd lockd grace sunrpc md_mod ipmi_devintf mlx4_en mlx4_core igb i2c_algo_bit sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd ipmi_ssif mpt3sas glue_helper i2c_i801 i2c_core intel_cstate intel_uncore ahci intel_rapl_perf libahci raid_class scsi_transport_sas pcc_cpufreq wmi ipmi_si acpi_pad button [last unloaded: mlx4_core]
Dec 22 11:18:43 Unraid kernel: ---[ end trace 0235f654514174a1 ]---
Dec 22 11:18:43 Unraid kernel: RIP: 0010:ata_qc_issue+0xfc/0x195
Dec 22 11:18:43 Unraid kernel: Code: 8b b5 a0 00 00 00 48 8b b8 40 35 00 00 48 85 ff 74 0c 48 8b 87 10 02 00 00 48 85 c0 75 07 48 8b 05 64 f3 9b 00 83 f9 02 76 02 <0f> 0b 45 31 c0 48 8b 40 30 e8 b1 5e 5a 00 85 c0 79 02 0f 0b 74 6b
Dec 22 11:18:43 Unraid kernel: RSP: 0018:ffffc90007023cf0 EFLAGS: 00010002
Dec 22 11:18:43 Unraid kernel: RAX: ffffffff81c678c0 RBX: ffff888842460000 RCX: 0000000000000003
Dec 22 11:18:43 Unraid kernel: RDX: 0000000000000001 RSI: ffff8888398a3bd0 RDI: ffff88905b33d0a8
Dec 22 11:18:43 Unraid kernel: RBP: ffff8888424606d0 R08: 0000000000000087 R09: 0000000000000000
Dec 22 11:18:43 Unraid kernel: R10: 0000000000000200 R11: fefefefefefefeff R12: ffff888842462040
Dec 22 11:18:43 Unraid kernel: R13: ffff8888594cd800 R14: ffff888842460000 R15: ffff8888398a3900
Dec 22 11:18:43 Unraid kernel: FS:  0000000000000000(0000) GS:ffff88905f800000(0000) knlGS:0000000000000000
Dec 22 11:18:43 Unraid kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 22 11:18:43 Unraid kernel: CR2: 0000145d01098ff8 CR3: 0000000001e0a002 CR4: 00000000001606e0
Dec 22 11:18:52 Unraid kernel: ------------[ cut here ]------------
Dec 22 11:18:52 Unraid kernel: NETDEV WATCHDOG: eth0 (igb): transmit queue 2 timed out
Dec 22 11:18:52 Unraid kernel: WARNING: CPU: 5 PID: 0 at net/sched/sch_generic.c:465 dev_watchdog+0x161/0x1bb
Dec 22 11:18:52 Unraid kernel: Modules linked in: tun veth xt_nat ipt_MASQUERADE iptable_filter iptable_nat nf_nat_ipv4 nf_nat ip_tables xfs nfsd lockd grace sunrpc md_mod ipmi_devintf mlx4_en mlx4_core igb i2c_algo_bit sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd ipmi_ssif mpt3sas glue_helper i2c_i801 i2c_core intel_cstate intel_uncore ahci intel_rapl_perf libahci raid_class scsi_transport_sas pcc_cpufreq wmi ipmi_si acpi_pad button [last unloaded: mlx4_core]
Dec 22 11:18:52 Unraid kernel: CPU: 5 PID: 0 Comm: swapper/5 Tainted: G      D           4.19.88-Unraid #1
Dec 22 11:18:52 Unraid kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.3 08/23/2018
Dec 22 11:18:52 Unraid kernel: RIP: 0010:dev_watchdog+0x161/0x1bb
Dec 22 11:18:52 Unraid kernel: Code: 71 94 00 00 75 39 48 89 ef c6 05 38 71 94 00 01 e8 85 a8 fd ff 44 89 e9 48 89 ee 48 c7 c7 49 1f da 81 48 89 c2 e8 8f 1b af ff <0f> 0b eb 11 41 ff c5 48 81 c2 40 01 00 00 41 39 cd 75 95 eb 13 48
Dec 22 11:18:52 Unraid kernel: RSP: 0018:ffff88885f743ea0 EFLAGS: 00010286
Dec 22 11:18:52 Unraid kernel: RAX: 0000000000000000 RBX: ffff88883099c438 RCX: 0000000000000007
Dec 22 11:18:52 Unraid kernel: RDX: 0000000000000c05 RSI: 0000000000000002 RDI: ffff88885f7564f0
Dec 22 11:18:52 Unraid kernel: RBP: ffff88883099c000 R08: 000000000000000f R09: ffff8880000bb300
Dec 22 11:18:52 Unraid kernel: R10: 0000000000000000 R11: 0000000000000058 R12: ffff88883099c41c
Dec 22 11:18:52 Unraid kernel: R13: 0000000000000002 R14: ffff88883a77c940 R15: 0000000000000005
Dec 22 11:18:52 Unraid kernel: FS:  0000000000000000(0000) GS:ffff88885f740000(0000) knlGS:0000000000000000
Dec 22 11:18:52 Unraid kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033

Anyone have any ideas what's going on?

unraid-diagnostics-20191222-1308.zip

Edited December 22, 2019 by drawmonster