Hello,
from time to time this happens (random after 16-52 days uptime) during nightly backup of container data. Might not be exklusive during nightly backups but this is where it happens most of the time.
I have to login and reboot to get things running again. Sometimes web ui / SSH isn't reachable and i have to do a hard reset on the chassis.
Any clues in what direction to look to figure this out?
Thanks in advance!
...
Aug 19 03:00:11 homeserver CA Backup/Restore: docker stop -t 60 heimdall
Aug 19 03:00:11 homeserver CA Backup/Restore: Stopping Influxdb
Aug 19 03:00:11 homeserver kernel: veth779b620: renamed from eth0
Aug 19 03:00:11 homeserver CA Backup/Restore: docker stop -t 60 Influxdb
Aug 19 03:00:11 homeserver CA Backup/Restore: Stopping iobroker-live
Aug 19 03:00:13 homeserver kernel: BUG: unable to handle page fault for address: fffff8efd18eadc8
Aug 19 03:00:13 homeserver kernel: #PF: supervisor read access in kernel mode
Aug 19 03:00:13 homeserver kernel: #PF: error_code(0x0000) - not-present page
Aug 19 03:00:13 homeserver kernel: PGD 0 P4D 0
Aug 19 03:00:13 homeserver kernel: Oops: 0000 [#1] SMP PTI
Aug 19 03:00:13 homeserver kernel: CPU: 1 PID: 28346 Comm: io.mqtt.0 Tainted: G W 5.15.46-Unraid #1
Aug 19 03:00:13 homeserver kernel: Hardware name: Supermicro Super Server/X11SCH-LN4F, BIOS 1.5 11/17/2020
Aug 19 03:00:13 homeserver kernel: RIP: 0010:_compound_head+0x0/0x11
Aug 19 03:00:13 homeserver kernel: Code: b8 98 0f 00 00 00 00 f0 7f 48 85 c2 0f 95 c0 0f b6 c0 c3 48 c1 ef 3a 83 ff 1f 0f 94 c0 48 83 ff 1e 0f 94 c2 09 d0 0f b6 c0 c3 <48> 8b 57 08 48 89 f8 f6 c2 01 74 04 48 8d 42 ff c3 e8 ea ff ff ff
Aug 19 03:00:13 homeserver kernel: RSP: 0018:ffffc9000300fcc0 EFLAGS: 00010202
Aug 19 03:00:13 homeserver kernel: RAX: 0000000000000001 RBX: 0000000003ff4000 RCX: 000000000000011e
Aug 19 03:00:13 homeserver kernel: RDX: 7c00000000000000 RSI: ffffea002dca5980 RDI: fffff8efd18eadc0
Aug 19 03:00:13 homeserver kernel: RBP: ffffc9000300fe00 R08: ffffea002dca5980 R09: ffff88903f2704f0
Aug 19 03:00:13 homeserver kernel: R10: ffff88903f269388 R11: 0000000000000297 R12: fffff8efd18eadc0
Aug 19 03:00:13 homeserver kernel: R13: 7c00003bbf463ab7 R14: ffff8881738a90b0 R15: ffff8881342f7000
Aug 19 03:00:13 homeserver kernel: FS: 0000000000000000(0000) GS:ffff88903f240000(0000) knlGS:0000000000000000
Aug 19 03:00:13 homeserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 19 03:00:13 homeserver kernel: CR2: fffff8efd18eadc8 CR3: 000000016cdc8001 CR4: 00000000003726e0
Aug 19 03:00:13 homeserver kernel: Call Trace:
Aug 19 03:00:13 homeserver kernel: <TASK>
Aug 19 03:00:13 homeserver kernel: pfn_swap_entry_to_page+0x26/0x35
Aug 19 03:00:13 homeserver kernel: unmap_page_range+0x4c3/0x711
Aug 19 03:00:13 homeserver kernel: unmap_vmas+0x6f/0x9d
Aug 19 03:00:13 homeserver kernel: exit_mmap+0xd6/0x145
Aug 19 03:00:13 homeserver kernel: __mmput+0x43/0xd7
Aug 19 03:00:13 homeserver kernel: do_exit+0x385/0x915
Aug 19 03:00:13 homeserver kernel: ? signal_wake_up_state+0x1e/0x2f
Aug 19 03:00:13 homeserver kernel: do_group_exit+0x93/0x93
Aug 19 03:00:13 homeserver kernel: __x64_sys_exit_group+0x14/0x14
Aug 19 03:00:13 homeserver kernel: do_syscall_64+0x83/0xa5
Aug 19 03:00:13 homeserver kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
Aug 19 03:00:13 homeserver kernel: RIP: 0033:0x7fccd1325699
Aug 19 03:00:13 homeserver kernel: Code: Unable to access opcode bytes at RIP 0x7fccd132566f.
Aug 19 03:00:13 homeserver kernel: RSP: 002b:00007ffc48240098 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
Aug 19 03:00:13 homeserver kernel: RAX: ffffffffffffffda RBX: 00007fccd141a610 RCX: 00007fccd1325699
Aug 19 03:00:13 homeserver kernel: RDX: 000000000000003c RSI: 00000000000000e7 RDI: 000000000000000b
Aug 19 03:00:13 homeserver kernel: RBP: 000000000000000b R08: ffffffffffffff08 R09: 0000000000000000
Aug 19 03:00:13 homeserver kernel: R10: 00007fccd15cbf21 R11: 0000000000000246 R12: 00007fccd141a610
Aug 19 03:00:13 homeserver kernel: R13: 0000000000000306 R14: 00007fccd141aae8 R15: 0000000000000000
Aug 19 03:00:13 homeserver kernel: </TASK>
Aug 19 03:00:13 homeserver kernel: Modules linked in: xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth macvlan xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod ipmi_devintf wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bonding igb x86_pkg_temp_thermal intel_powerclamp i915 ast drm_vram_helper iosf_mbi drm_ttm_helper ttm drm_kms_helper i2c_i801 coretemp kvm_intel kvm crct10dif_pclmul wmi_bmof crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd ipmi_ssif rapl intel_cstate intel_uncore drm i2c_smbus intel_gtt nvme agpgart i2c_algo_bit syscopyarea sysfillrect sysimgblt nvme_core ahci i2c_core
Aug 19 03:00:13 homeserver kernel: fb_sys_fops libahci input_leds led_class acpi_ipmi intel_pch_thermal wmi thermal fan ipmi_si video backlight acpi_pad button acpi_tad acpi_power_meter [last unloaded: igb]
Aug 19 03:00:13 homeserver kernel: CR2: fffff8efd18eadc8
Aug 19 03:00:13 homeserver kernel: ---[ end trace 5539ff0efe87359a ]---
Aug 19 03:00:13 homeserver kernel: RIP: 0010:_compound_head+0x0/0x11
Aug 19 03:00:13 homeserver kernel: Code: b8 98 0f 00 00 00 00 f0 7f 48 85 c2 0f 95 c0 0f b6 c0 c3 48 c1 ef 3a 83 ff 1f 0f 94 c0 48 83 ff 1e 0f 94 c2 09 d0 0f b6 c0 c3 <48> 8b 57 08 48 89 f8 f6 c2 01 74 04 48 8d 42 ff c3 e8 ea ff ff ff
Aug 19 03:00:13 homeserver kernel: RSP: 0018:ffffc9000300fcc0 EFLAGS: 00010202
Aug 19 03:00:13 homeserver kernel: RAX: 0000000000000001 RBX: 0000000003ff4000 RCX: 000000000000011e
Aug 19 03:00:13 homeserver kernel: RDX: 7c00000000000000 RSI: ffffea002dca5980 RDI: fffff8efd18eadc0
Aug 19 03:00:13 homeserver kernel: RBP: ffffc9000300fe00 R08: ffffea002dca5980 R09: ffff88903f2704f0
Aug 19 03:00:13 homeserver kernel: R10: ffff88903f269388 R11: 0000000000000297 R12: fffff8efd18eadc0
Aug 19 03:00:13 homeserver kernel: R13: 7c00003bbf463ab7 R14: ffff8881738a90b0 R15: ffff8881342f7000
Aug 19 03:00:13 homeserver kernel: FS: 0000000000000000(0000) GS:ffff88903f240000(0000) knlGS:0000000000000000
Aug 19 03:00:13 homeserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 19 03:00:13 homeserver kernel: CR2: fffff8efd18eadc8 CR3: 000000016cdc8001 CR4: 00000000003726e0
Aug 19 03:00:13 homeserver kernel: Fixing recursive fault but reboot is needed!
Aug 19 03:01:25 homeserver CA Backup/Restore: docker stop -t 60 iobroker-live
Aug 19 03:01:25 homeserver CA Backup/Restore: Stopping pihole
Aug 19 03:01:29 homeserver kernel: veth25bc94e: renamed from eth0
Aug 19 03:01:29 homeserver CA Backup/Restore: docker stop -t 60 pihole
Aug 19 03:01:29 homeserver CA Backup/Restore: Stopping plex
Aug 19 03:01:34 homeserver kernel: vethe25af7c: renamed from eth0
Aug 19 03:01:34 homeserver CA Backup/Restore: docker stop -t 60 plex
Aug 19 03:01:34 homeserver CA Backup/Restore: Backing Up appData from /mnt/cache/appdata/ to /mnt/user/backup/appdata/
[email protected]
Aug 19 03:01:34 homeserver CA Backup/Restore: Using command: cd '/mnt/cache/appdata/' && /usr/bin/tar -cvaf '/mnt/user/backup/appdata/
[email protected]/CA_backup.tar.gz' * >> /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log 2>&1 & echo $! > /tmp/ca.backup2/tempFiles/backupInProgress
Aug 19 03:01:40 homeserver emhttpd: read SMART /dev/sdb
Aug 19 03:01:40 homeserver emhttpd: read SMART /dev/sdc
Aug 19 03:06:52 homeserver CA Backup/Restore: Backup Complete
Aug 19 03:06:52 homeserver CA Backup/Restore: Verifying backup
Aug 19 03:06:52 homeserver CA Backup/Restore: Using command: cd '/mnt/cache/appdata/' && /usr/bin/tar --diff -C '/mnt/cache/appdata/' -af '/mnt/user/backup/appdata/
[email protected]/CA_backup.tar.gz' > /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log & echo $! > /tmp/ca.backup2/tempFiles/verifyInProgress
Aug 19 03:07:49 homeserver kernel: docker0: port 1(veth1639e18) entered blocking state
Aug 19 03:07:49 homeserver kernel: docker0: port 1(veth1639e18) entered disabled state
Aug 19 03:07:49 homeserver kernel: device veth1639e18 entered promiscuous mode
Aug 19 03:07:50 homeserver kernel: eth0: renamed from vethb6282da
Aug 19 03:07:50 homeserver kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth1639e18: link becomes ready
Aug 19 03:07:50 homeserver kernel: docker0: port 1(veth1639e18) entered blocking state
Aug 19 03:07:50 homeserver kernel: docker0: port 1(veth1639e18) entered forwarding state
Aug 19 03:07:50 homeserver kernel: docker0: port 2(vethdc57047) entered blocking state
Aug 19 03:07:50 homeserver kernel: docker0: port 2(vethdc57047) entered disabled state
Aug 19 03:07:50 homeserver kernel: device vethdc57047 entered promiscuous mode
Aug 19 03:07:50 homeserver kernel: docker0: port 2(vethdc57047) entered blocking state
Aug 19 03:07:50 homeserver kernel: docker0: port 2(vethdc57047) entered forwarding state
Aug 19 03:07:50 homeserver kernel: eth0: renamed from veth5a8fe96
Aug 19 03:07:50 homeserver kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethdc57047: link becomes ready
Aug 19 03:07:51 homeserver kernel: eth0: renamed from veth1b282b7
Aug 19 03:07:51 homeserver kernel: docker0: port 3(veth7ceaa7c) entered blocking state
Aug 19 03:07:51 homeserver kernel: docker0: port 3(veth7ceaa7c) entered disabled state
Aug 19 03:07:51 homeserver kernel: device veth7ceaa7c entered promiscuous mode
Aug 19 03:07:51 homeserver kernel: docker0: port 3(veth7ceaa7c) entered blocking state
Aug 19 03:07:51 homeserver kernel: docker0: port 3(veth7ceaa7c) entered forwarding state
Aug 19 03:07:52 homeserver kernel: docker0: port 3(veth7ceaa7c) entered disabled state
Aug 19 03:07:52 homeserver avahi-daemon[16088]: Joining mDNS multicast group on interface vethdc57047.IPv6 with address fe80::244f:7aff:fe1a:7aca.
Aug 19 03:07:52 homeserver avahi-daemon[16088]: New relevant interface vethdc57047.IPv6 for mDNS.
Aug 19 03:07:52 homeserver avahi-daemon[16088]: Registering new address record for fe80::244f:7aff:fe1a:7aca on vethdc57047.*.
Aug 19 03:07:52 homeserver kernel: eth0: renamed from veth4985384
Aug 19 03:07:52 homeserver kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth7ceaa7c: link becomes ready
Aug 19 03:07:52 homeserver kernel: docker0: port 3(veth7ceaa7c) entered blocking state
Aug 19 03:07:52 homeserver kernel: docker0: port 3(veth7ceaa7c) entered forwarding state
Aug 19 03:07:52 homeserver avahi-daemon[16088]: Joining mDNS multicast group on interface veth1639e18.IPv6 with address fe80::405c:63ff:fe9f:3ded.
Aug 19 03:07:52 homeserver avahi-daemon[16088]: New relevant interface veth1639e18.IPv6 for mDNS.
Aug 19 03:07:52 homeserver avahi-daemon[16088]: Registering new address record for fe80::405c:63ff:fe9f:3ded on veth1639e18.*.
Aug 19 03:07:53 homeserver kernel: eth0: renamed from veth1cad487
Aug 19 03:07:53 homeserver avahi-daemon[16088]: Joining mDNS multicast group on interface veth7ceaa7c.IPv6 with address fe80::d802:e9ff:fe7e:4437.
Aug 19 03:07:53 homeserver avahi-daemon[16088]: New relevant interface veth7ceaa7c.IPv6 for mDNS.
Aug 19 03:07:53 homeserver avahi-daemon[16088]: Registering new address record for fe80::d802:e9ff:fe7e:4437 on veth7ceaa7c.*.
Aug 19 03:07:53 homeserver kernel: eth0: renamed from vethcbb2b36
Aug 19 03:07:54 homeserver kernel: eth0: renamed from veth301a4e7
Aug 19 03:07:54 homeserver CA Backup/Restore: #######################
Aug 19 03:07:54 homeserver CA Backup/Restore: appData Backup complete
Aug 19 03:07:54 homeserver CA Backup/Restore: #######################
Aug 19 03:07:54 homeserver CA Backup/Restore: Deleting /mnt/user/backup/appdata/
[email protected]
Aug 19 03:07:54 homeserver CA Backup/Restore: Backup / Restore Completed
Aug 19 03:14:35 homeserver emhttpd: read SMART /dev/sde
Aug 19 03:24:57 homeserver emhttpd: spinning down /dev/sdb
Aug 19 03:24:57 homeserver emhttpd: spinning down /dev/sdc
Aug 19 03:30:27 homeserver emhttpd: spinning down /dev/sde
Aug 19 03:40:07 homeserver emhttpd: read SMART /dev/sdb
Aug 19 03:40:07 homeserver emhttpd: read SMART /dev/sdc
Aug 19 03:40:16 homeserver crond[1260]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null
Aug 19 03:57:20 homeserver emhttpd: spinning down /dev/sdb
Aug 19 03:57:20 homeserver emhttpd: spinning down /dev/sdc
Aug 19 04:23:48 homeserver emhttpd: read SMART /dev/sde
Aug 19 04:41:56 homeserver emhttpd: spinning down /dev/sde
Aug 19 07:12:50 homeserver webGUI: Successful login user root from 10.253.0.2
Aug 19 07:14:29 homeserver shutdown[11512]: shutting down for system reboot
Aug 19 07:14:29 homeserver init: Switching to runlevel: 6
Aug 19 07:14:29 homeserver init: Trying to re-exec init
Aug 19 07:14:31 homeserver kernel: mdcmd (37): nocheck cancel
Aug 19 07:14:32 homeserver emhttpd: Spinning up all drives...
Aug 19 07:14:32 homeserver emhttpd: spinning up /dev/sdd
Aug 19 07:14:32 homeserver emhttpd: spinning up /dev/sde
Aug 19 07:14:32 homeserver emhttpd: spinning up /dev/sdb
Aug 19 07:14:32 homeserver emhttpd: spinning up /dev/sdc
Aug 19 07:14:43 homeserver emhttpd: read SMART /dev/sdd
Aug 19 07:14:43 homeserver emhttpd: read SMART /dev/sde
Aug 19 07:14:43 homeserver emhttpd: read SMART /dev/sdb
Aug 19 07:14:43 homeserver emhttpd: read SMART /dev/sdc
Aug 19 07:14:43 homeserver emhttpd: read SMART /dev/nvme0n1
Aug 19 07:14:43 homeserver emhttpd: read SMART /dev/nvme1n1
Aug 19 07:14:43 homeserver emhttpd: read SMART /dev/sda
