Server Crashes with Fatal Exception in Interupt

October 12, 20232 yr

I am having issues with my server crashing with this error after less than a day of sitting idle with the array not started.

How can I start troubleshooting this?

Quote

October 13, 20232 yr

Community Expert

Enable the syslog server and post that after a crash together with the complete diagnostics.

Quote

October 14, 20232 yr

Author

I have the syslog server running and should have a crash to report by the morning.

Quote

October 14, 20232 yr

Author

Below are the results from the syslog server and I included my diagnostics file.

Oct 13 20:59:07 10.100.2.2 rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="6887" x-info="https://www.rsyslog.com"] start

Oct 13 21:00:13 10.100.2.2 root: /mnt/cache: 1.8 TiB (1998086459392 bytes) trimmed on /dev/nvme0n1p1

Oct 13 21:01:22 10.100.2.2 Test UDP syslog message

Oct 13 21:05:08 pfsense syslog-ng[12698]: Log statistics; processed='global(internal_queue_length)=0', queued='global(scratch_buffers_count)=0', queued='global(scratch_buffers_bytes)=0', processed='global(msg_clones)=0', processed='src.internal(_DEFAULT#0)=1', stamp='src.internal(_DEFAULT#0)=1697244908', processed='center(received)=7', processed='center(queued)=7', processed='destination(_DEFAULT)=7', processed='global(payload_reallocs)=0', processed='source(_DEFAULT)=7', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0'

Oct 13 21:13:55 10.100.2.2 emhttpd: spinning down /dev/sdd

Oct 13 21:13:55 10.100.2.2 emhttpd: spinning down /dev/sdj

Oct 13 21:13:55 10.100.2.2 emhttpd: spinning down /dev/sdk

Oct 13 21:15:08 pfsense syslog-ng[12698]: Log statistics; processed='global(internal_queue_length)=0', queued='global(scratch_buffers_count)=0', queued='global(scratch_buffers_bytes)=0', processed='global(msg_clones)=0', processed='src.internal(_DEFAULT#0)=2', stamp='src.internal(_DEFAULT#0)=1697245508', processed='center(received)=11', processed='center(queued)=11', processed='destination(_DEFAULT)=11', processed='global(payload_reallocs)=1', processed='source(_DEFAULT)=11', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0'

Oct 13 21:25:08 pfsense syslog-ng[12698]: Log statistics; processed='global(internal_queue_length)=0', queued='global(scratch_buffers_count)=0', queued='global(scratch_buffers_bytes)=0', processed='global(msg_clones)=0', processed='src.internal(_DEFAULT#0)=3', stamp='src.internal(_DEFAULT#0)=1697246108', processed='center(received)=12', processed='center(queued)=12', processed='destination(_DEFAULT)=12', processed='global(payload_reallocs)=2', processed='source(_DEFAULT)=12', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0'

Oct 13 21:31:54 10.100.2.2 emhttpd: sdspin /dev/nvme0n1 down: 25

Oct 13 21:31:54 10.100.2.2 emhttpd: spinning down /dev/nvme0n1

Oct 13 21:35:08 pfsense syslog-ng[12698]: Log statistics; processed='global(internal_queue_length)=0', queued='global(scratch_buffers_count)=0', queued='global(scratch_buffers_bytes)=0', processed='global(msg_clones)=0', processed='src.internal(_DEFAULT#0)=4', stamp='src.internal(_DEFAULT#0)=1697246708', processed='center(received)=15', processed='center(queued)=15', processed='destination(_DEFAULT)=15', processed='global(payload_reallocs)=3', processed='source(_DEFAULT)=15', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0'

Oct 13 21:45:08 pfsense syslog-ng[12698]: Log statistics; processed='global(internal_queue_length)=0', queued='global(scratch_buffers_count)=0', queued='global(scratch_buffers_bytes)=0', processed='global(msg_clones)=0', processed='src.internal(_DEFAULT#0)=5', stamp='src.internal(_DEFAULT#0)=1697247308', processed='center(received)=16', processed='center(queued)=16', processed='destination(_DEFAULT)=16', processed='global(payload_reallocs)=4', processed='source(_DEFAULT)=16', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0'

Oct 13 21:55:08 pfsense syslog-ng[12698]: Log statistics; processed='global(internal_queue_length)=0', queued='global(scratch_buffers_count)=0', queued='global(scratch_buffers_bytes)=0', processed='global(msg_clones)=0', processed='src.internal(_DEFAULT#0)=6', stamp='src.internal(_DEFAULT#0)=1697247908', processed='center(received)=17', processed='center(queued)=17', processed='destination(_DEFAULT)=17', processed='global(payload_reallocs)=5', processed='source(_DEFAULT)=17', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0'

Oct 13 22:00:12 10.100.2.2 root: /mnt/cache: 89.8 GiB (96466243584 bytes) trimmed on /dev/nvme0n1p1

Oct 13 22:04:20 10.100.2.2 kernel: __schedule+0x1c4/0x612

Oct 13 22:04:20 10.100.2.2 kernel: ------------[ cut here ]------------

Oct 13 22:04:20 10.100.2.2 kernel: ---[ end trace 0000000000000000 ]---

Oct 13 22:04:20 10.100.2.2 kernel: ? __die_body+0x1a/0x5c

Oct 13 22:04:20 10.100.2.2 kernel: ? __warn+0xab/0x122

Oct 13 22:04:20 10.100.2.2 kernel: ? asm_exc_invalid_op+0x16/0x20

Oct 13 22:04:20 10.100.2.2 kernel: ? asm_exc_page_fault+0x22/0x30

Oct 13 22:04:20 10.100.2.2 kernel: ? dbs_work_handler+0x4a/0x5e

Oct 13 22:04:20 10.100.2.2 kernel: ? do_idle+0x1a8/0x1fb

Oct 13 22:04:20 10.100.2.2 kernel: ? do_user_addr_fault+0x12e/0x48d

Oct 13 22:04:20 10.100.2.2 kernel: ? exc_invalid_op+0x13/0x60

Oct 13 22:04:20 10.100.2.2 kernel: ? exc_page_fault+0xfb/0x11d

Oct 13 22:04:20 10.100.2.2 kernel: ? fixup_exception+0x22/0x24b

Oct 13 22:04:20 10.100.2.2 kernel: ? handle_bug+0x41/0x6f

Oct 13 22:04:20 10.100.2.2 kernel: ? kthread_complete_and_exit+0x1b/0x1b

Oct 13 22:04:20 10.100.2.2 kernel: ? load_balance+0xea/0x950

Oct 13 22:04:20 10.100.2.2 kernel: ? load_balance+0xf6/0x950

Oct 13 22:04:20 10.100.2.2 kernel: ? page_fault_oops+0x329/0x376

Oct 13 22:04:20 10.100.2.2 kernel: ? pick_next_task_fair+0x90/0x2a7

Oct 13 22:04:20 10.100.2.2 kernel: ? raw_spin_rq_unlock_irqrestore+0x9/0x1b

Oct 13 22:04:20 10.100.2.2 kernel: ? report_bug+0x109/0x17e

Oct 13 22:04:20 10.100.2.2 kernel: ? rescuer_thread+0x281/0x281

Oct 13 22:04:20 10.100.2.2 kernel: ? update_blocked_averages+0x244/0x27a

Oct 13 22:04:20 10.100.2.2 kernel: #PF: error_code(0x0000) - not-present page

Oct 13 22:04:20 10.100.2.2 kernel: #PF: supervisor read access in kernel mode

Oct 13 22:04:20 10.100.2.2 kernel: </TASK>

Oct 13 22:04:20 10.100.2.2 kernel: <TASK>

Oct 13 22:04:20 10.100.2.2 kernel: BUG: kernel NULL pointer dereference, address: 0000000000000010

Oct 13 22:04:20 10.100.2.2 kernel: BUG: kernel NULL pointer dereference, address: 0000000000000080

Oct 13 22:04:20 10.100.2.2 kernel: Call Trace:

Oct 13 22:04:20 10.100.2.2 kernel: Code: 00 48 89 de 48 89 ef 44 89 f2 e8 96 e8 ff ff 48 89 df 89 c6 e8 e3 cb 5f 00 e8 0f ca ff ff 9c 58 0f 1f 40 00 0f ba e0 09 72 08 <0f> 0b fb 0f 1f 44 00 00 e8 1d fe ff ff e9 89 fe ff ff 65 81 25 f8

Oct 13 22:04:20 10.100.2.2 kernel: Code: 00 48 8d 84 24 c8 00 00 00 48 89 84 24 c8 00 00 00 48 89 84 24 d0 00 00 00 e8 8e 8f ff ff 48 8b 44 24 70 8b 94 24 84 00 00 00 <48> 8b 58 10 48 8b 84 24 a8 00 00 00 48 0f a3 10 72 10 48 8b 44 24

Oct 13 22:04:20 10.100.2.2 kernel: Code: ed 48 89 ef e8 f1 dd ff ff 84 c0 74 12 83 bb 90 00 00 00 00 0f 84 4f 01 00 00 e9 8f 00 00 00 4c 89 ee 48 89 ef e8 4c b7 ff ff <48> 8b a8 80 00 00 00 49 89 c5 48 85 ed 75 ab 48 89 c7 e8 2d 6f ff

Oct 13 22:04:20 10.100.2.2 kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6.

Oct 13 22:04:20 10.100.2.2 kernel: cpu_startup_entry+0x1d/0x1f

Oct 13 22:04:20 10.100.2.2 kernel: CPU: 21 PID: 0 Comm: swapper/21 Tainted: P O 6.1.38-Unraid #2

Oct 13 22:04:20 10.100.2.2 kernel: CPU: 21 PID: 403 Comm: kworker/21:1 Tainted: P W O 6.1.38-Unraid #2

Oct 13 22:04:20 10.100.2.2 kernel: CPU: 21 PID: 403 Comm: kworker/21:1 Tainted: P D W O 6.1.38-Unraid #2

Oct 13 22:04:20 10.100.2.2 kernel: CR2: 0000000000000010

Oct 13 22:04:20 10.100.2.2 kernel: CR2: 0000000000000010 CR3: 000000000220a000 CR4: 0000000000350ee0

Oct 13 22:04:20 10.100.2.2 kernel: CR2: 0000000000000080

Oct 13 22:04:20 10.100.2.2 kernel: CR2: 0000000000000080 CR3: 000000000220a000 CR4: 0000000000350ee0

Oct 13 22:04:20 10.100.2.2 kernel: CR2: 000014b92a997840 CR3: 000000000220a000 CR4: 0000000000350ee0

Oct 13 22:04:20 10.100.2.2 kernel: CR2: ffffffffffffffd6 CR3: 000000000220a000 CR4: 0000000000350ee0

Oct 13 22:04:20 10.100.2.2 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

Oct 13 22:04:20 10.100.2.2 kernel: do_exit+0x923/0x923

Oct 13 22:04:20 10.100.2.2 kernel: do_task_dead+0x3f/0x41

Oct 13 22:04:20 10.100.2.2 kernel: FS: 0000000000000000(0000) GS:ffff888fded40000(0000) knlGS:0000000000000000

Oct 13 22:04:20 10.100.2.2 kernel: Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 5220 09/12/2019

Oct 13 22:04:20 10.100.2.2 kernel: kthread+0xe7/0xef

Oct 13 22:04:20 10.100.2.2 kernel: make_task_dead+0x11c/0x11c

Oct 13 22:04:20 10.100.2.2 kernel: Modules linked in: xfs md_mod tcp_diag inet_diag ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge 8021q garp mrp stp llc bonding tls igb i2c_algo_bit nvidia_drm(PO) nvidia_modeset(PO) zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) edac_mce_amd edac_core intel_rapl_msr intel_rapl_common iosf_mbi zcommon(PO) znvpair(PO) kvm_amd spl(O) nvidia(PO) kvm video crct10dif_pclmul crc32_pclmul crc32c_intel drm_kms_helper ghash_clmulni_intel sha512_ssse3 aesni_intel crypto_simd cryptd wmi_bmof mxm_wmi drm mpt3sas rapl nvme backlight i2c_piix4 syscopyarea k10temp sysfillrect ccp raid_class sysimgblt fb_sys_fops nvme_core scsi_transport_sas i2c_core ahci libahci wmi button acpi_cpufreq unix [last unloaded: i2c_algo_bit]

Oct 13 22:04:20 10.100.2.2 kernel: newidle_balance+0x1a4/0x30a

Oct 13 22:04:20 10.100.2.2 kernel: note: kworker/21:1[403] exited with irqs disabled

Oct 13 22:04:20 10.100.2.2 kernel: note: kworker/21:1[403] exited with preempt_count 1

Oct 13 22:04:20 10.100.2.2 kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI

Oct 13 22:04:20 10.100.2.2 kernel: Oops: 0000 [#2] PREEMPT SMP NOPTI

Oct 13 22:04:20 10.100.2.2 kernel: PGD 0 P4D 0

Oct 13 22:04:20 10.100.2.2 kernel: pick_next_task_fair+0x1e3/0x2a7

Oct 13 22:04:20 10.100.2.2 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000

Oct 13 22:04:20 10.100.2.2 kernel: R10: 0000000000000020 R11: 0000000000000052 R12: 0000000000000015

Oct 13 22:04:20 10.100.2.2 kernel: R10: 00000000002191c0 R11: 0000000000000002 R12: ffff888fded5d360

Oct 13 22:04:20 10.100.2.2 kernel: R10: ffff88810872a100 R11: ffff888107b15c20 R12: ffff888108723f00

Oct 13 22:04:20 10.100.2.2 kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000

Oct 13 22:04:20 10.100.2.2 kernel: R13: 0000000000000000 R14: ffff888fded6d4c0 R15: ffffc90001087e78

Oct 13 22:04:20 10.100.2.2 kernel: R13: 0000047df951194d R14: ffff888fded6d440 R15: ffffc90001087e88

Oct 13 22:04:20 10.100.2.2 kernel: R13: ffff888100a64ec0 R14: 0000000000000001 R15: 0000000000000000

Oct 13 22:04:20 10.100.2.2 kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000

Oct 13 22:04:20 10.100.2.2 kernel: RAX: 0000000000000000 RBX: ffff888fded6d440 RCX: 0000000000000009

Oct 13 22:04:20 10.100.2.2 kernel: RAX: 0000000000000000 RBX: ffff888fded6d440 RCX: 00000000ffffffe0

Oct 13 22:04:20 10.100.2.2 kernel: RAX: 0000000000145402 RBX: ffff88810869c800 RCX: 0000000100000000

Oct 13 22:04:20 10.100.2.2 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000

Oct 13 22:04:20 10.100.2.2 kernel: RBP: ffff8881001f4000 R08: 0000000000200200 R09: ffff888fded5d360

Oct 13 22:04:20 10.100.2.2 kernel: RBP: ffff888fded6d4c0 R08: ffff888108723f80 R09: 00000000000000d0

Oct 13 22:04:20 10.100.2.2 kernel: RBP: ffffffff823235a0 R08: 0000000000000002 R09: 0000000000000002

Oct 13 22:04:20 10.100.2.2 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000

Oct 13 22:04:20 10.100.2.2 kernel: RDX: 0000000000000000 RSI: ffff8881001f4098 RDI: 0000000000000000

Oct 13 22:04:20 10.100.2.2 kernel: RDX: 0000000000000001 RSI: ffffffff82090ebc RDI: ffffffff820913c5

Oct 13 22:04:20 10.100.2.2 kernel: RDX: ffffc90001087e78 RSI: 0000000000000000 RDI: ffff888fded6d4c0

Oct 13 22:04:20 10.100.2.2 kernel: ret_from_fork+0x22/0x30

Oct 13 22:04:20 10.100.2.2 kernel: rewind_stack_and_make_dead+0x17/0x17

Oct 13 22:04:20 10.100.2.2 kernel: RIP: 0000:0x0

Oct 13 22:04:20 10.100.2.2 kernel: RIP: 0010:do_idle+0x1a8/0x1fb

Oct 13 22:04:20 10.100.2.2 kernel: RIP: 0010:load_balance+0xf6/0x950

Oct 13 22:04:20 10.100.2.2 kernel: RIP: 0010:pick_next_task_fair+0x90/0x2a7

Oct 13 22:04:20 10.100.2.2 kernel: RSP: 0000:0000000000000000 EFLAGS: 00000000 ORIG_RAX: 0000000000000000

Oct 13 22:04:20 10.100.2.2 kernel: RSP: 0018:ffffc900001f7ef8 EFLAGS: 00155402

Oct 13 22:04:20 10.100.2.2 kernel: RSP: 0018:ffffc90001087cc8 EFLAGS: 00155406

Oct 13 22:04:20 10.100.2.2 kernel: RSP: 0018:ffffc90001087e30 EFLAGS: 00111046

Oct 13 22:04:20 10.100.2.2 kernel: schedule+0x8e/0xcc

Oct 13 22:04:20 10.100.2.2 kernel: secondary_startup_64_no_verify+0xce/0xdb

Oct 13 22:04:20 10.100.2.2 kernel: start_secondary+0x101/0x101

Oct 13 22:04:20 10.100.2.2 kernel: WARNING: CPU: 21 PID: 0 at kernel/sched/idle.c:249 do_idle+0x1a8/0x1fb

Oct 13 22:04:20 10.100.2.2 kernel: worker_thread+0x1f8/0x244

Oct 13 22:04:20 10.100.2.2 kernel: Workqueue: 0x0 (events)

Oct 13 22:05:08 pfsense syslog-ng[12698]: Log statistics; processed='global(internal_queue_length)=0', queued='global(scratch_buffers_count)=0', queued='global(scratch_buffers_bytes)=0', processed='global(msg_clones)=0', processed='src.internal(_DEFAULT#0)=7', stamp='src.internal(_DEFAULT#0)=1697248508', processed='center(received)=161', processed='center(queued)=161', processed='destination(_DEFAULT)=161', processed='global(payload_reallocs)=6', processed='source(_DEFAULT)=161', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0'

Oct 13 22:15:08 pfsense syslog-ng[12698]: Log statistics; processed='global(internal_queue_length)=0', queued='global(scratch_buffers_count)=0', queued='global(scratch_buffers_bytes)=0', processed='global(msg_clones)=0', processed='src.internal(_DEFAULT#0)=8', stamp='src.internal(_DEFAULT#0)=1697249108', processed='center(received)=162', processed='center(queued)=162', processed='destination(_DEFAULT)=162', processed='global(payload_reallocs)=7', processed='source(_DEFAULT)=162', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0'

unraidnas-diagnostics-20231013-2224.zip

Quote

October 14, 20232 yr

Community Expert

That looks more hardware related to me, start here:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=819173

Quote

October 16, 20232 yr

Author

I've started looking at power states and it seems to have helped a bit. I don't get the kernel panic error anymore. Now I am seeing if turning off DOCP for ram will help stabilize it enough to run a full parity check. The latest is that the server did not crash after changing global power state settings but was slow when signing in through the command line and did not give me the prompt to enter my password. The web gui was also not loading. Hopefully the ram setting change will help.

Quote

October 17, 20232 yr

Author

After disabling DOCP for ram it crashed again with kernel panic error.

Quote

Server Crashes with Fatal Exception in Interupt

Featured Replies

Join the conversation

Account

Navigation

Search

Configure browser push notifications

Chrome (Android)

Chrome (Desktop)

Safari (iOS 16.4+)

Safari (macOS)

Edge (Android)

Edge (Desktop)

Firefox (Android)

Firefox (Desktop)