- Minor
Hello,
from time to time this happens (random after 16-52 days uptime) during nightly backup of container data. Might not be exklusive during nightly backups but this is where it happens most of the time.
I have to login and reboot to get things running again. Sometimes web ui / SSH isn't reachable and i have to do a hard reset on the chassis.
Any clues in what direction to look to figure this out?
Thanks in advance!
... Aug 19 03:00:11 homeserver CA Backup/Restore: docker stop -t 60 heimdall Aug 19 03:00:11 homeserver CA Backup/Restore: Stopping Influxdb Aug 19 03:00:11 homeserver kernel: veth779b620: renamed from eth0 Aug 19 03:00:11 homeserver CA Backup/Restore: docker stop -t 60 Influxdb Aug 19 03:00:11 homeserver CA Backup/Restore: Stopping iobroker-live Aug 19 03:00:13 homeserver kernel: BUG: unable to handle page fault for address: fffff8efd18eadc8 Aug 19 03:00:13 homeserver kernel: #PF: supervisor read access in kernel mode Aug 19 03:00:13 homeserver kernel: #PF: error_code(0x0000) - not-present page Aug 19 03:00:13 homeserver kernel: PGD 0 P4D 0 Aug 19 03:00:13 homeserver kernel: Oops: 0000 [#1] SMP PTI Aug 19 03:00:13 homeserver kernel: CPU: 1 PID: 28346 Comm: io.mqtt.0 Tainted: G W 5.15.46-Unraid #1 Aug 19 03:00:13 homeserver kernel: Hardware name: Supermicro Super Server/X11SCH-LN4F, BIOS 1.5 11/17/2020 Aug 19 03:00:13 homeserver kernel: RIP: 0010:_compound_head+0x0/0x11 Aug 19 03:00:13 homeserver kernel: Code: b8 98 0f 00 00 00 00 f0 7f 48 85 c2 0f 95 c0 0f b6 c0 c3 48 c1 ef 3a 83 ff 1f 0f 94 c0 48 83 ff 1e 0f 94 c2 09 d0 0f b6 c0 c3 <48> 8b 57 08 48 89 f8 f6 c2 01 74 04 48 8d 42 ff c3 e8 ea ff ff ff Aug 19 03:00:13 homeserver kernel: RSP: 0018:ffffc9000300fcc0 EFLAGS: 00010202 Aug 19 03:00:13 homeserver kernel: RAX: 0000000000000001 RBX: 0000000003ff4000 RCX: 000000000000011e Aug 19 03:00:13 homeserver kernel: RDX: 7c00000000000000 RSI: ffffea002dca5980 RDI: fffff8efd18eadc0 Aug 19 03:00:13 homeserver kernel: RBP: ffffc9000300fe00 R08: ffffea002dca5980 R09: ffff88903f2704f0 Aug 19 03:00:13 homeserver kernel: R10: ffff88903f269388 R11: 0000000000000297 R12: fffff8efd18eadc0 Aug 19 03:00:13 homeserver kernel: R13: 7c00003bbf463ab7 R14: ffff8881738a90b0 R15: ffff8881342f7000 Aug 19 03:00:13 homeserver kernel: FS: 0000000000000000(0000) GS:ffff88903f240000(0000) knlGS:0000000000000000 Aug 19 03:00:13 homeserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 19 03:00:13 homeserver kernel: CR2: fffff8efd18eadc8 CR3: 000000016cdc8001 CR4: 00000000003726e0 Aug 19 03:00:13 homeserver kernel: Call Trace: Aug 19 03:00:13 homeserver kernel: <TASK> Aug 19 03:00:13 homeserver kernel: pfn_swap_entry_to_page+0x26/0x35 Aug 19 03:00:13 homeserver kernel: unmap_page_range+0x4c3/0x711 Aug 19 03:00:13 homeserver kernel: unmap_vmas+0x6f/0x9d Aug 19 03:00:13 homeserver kernel: exit_mmap+0xd6/0x145 Aug 19 03:00:13 homeserver kernel: __mmput+0x43/0xd7 Aug 19 03:00:13 homeserver kernel: do_exit+0x385/0x915 Aug 19 03:00:13 homeserver kernel: ? signal_wake_up_state+0x1e/0x2f Aug 19 03:00:13 homeserver kernel: do_group_exit+0x93/0x93 Aug 19 03:00:13 homeserver kernel: __x64_sys_exit_group+0x14/0x14 Aug 19 03:00:13 homeserver kernel: do_syscall_64+0x83/0xa5 Aug 19 03:00:13 homeserver kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae Aug 19 03:00:13 homeserver kernel: RIP: 0033:0x7fccd1325699 Aug 19 03:00:13 homeserver kernel: Code: Unable to access opcode bytes at RIP 0x7fccd132566f. Aug 19 03:00:13 homeserver kernel: RSP: 002b:00007ffc48240098 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 Aug 19 03:00:13 homeserver kernel: RAX: ffffffffffffffda RBX: 00007fccd141a610 RCX: 00007fccd1325699 Aug 19 03:00:13 homeserver kernel: RDX: 000000000000003c RSI: 00000000000000e7 RDI: 000000000000000b Aug 19 03:00:13 homeserver kernel: RBP: 000000000000000b R08: ffffffffffffff08 R09: 0000000000000000 Aug 19 03:00:13 homeserver kernel: R10: 00007fccd15cbf21 R11: 0000000000000246 R12: 00007fccd141a610 Aug 19 03:00:13 homeserver kernel: R13: 0000000000000306 R14: 00007fccd141aae8 R15: 0000000000000000 Aug 19 03:00:13 homeserver kernel: </TASK> Aug 19 03:00:13 homeserver kernel: Modules linked in: xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth macvlan xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod ipmi_devintf wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bonding igb x86_pkg_temp_thermal intel_powerclamp i915 ast drm_vram_helper iosf_mbi drm_ttm_helper ttm drm_kms_helper i2c_i801 coretemp kvm_intel kvm crct10dif_pclmul wmi_bmof crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd ipmi_ssif rapl intel_cstate intel_uncore drm i2c_smbus intel_gtt nvme agpgart i2c_algo_bit syscopyarea sysfillrect sysimgblt nvme_core ahci i2c_core Aug 19 03:00:13 homeserver kernel: fb_sys_fops libahci input_leds led_class acpi_ipmi intel_pch_thermal wmi thermal fan ipmi_si video backlight acpi_pad button acpi_tad acpi_power_meter [last unloaded: igb] Aug 19 03:00:13 homeserver kernel: CR2: fffff8efd18eadc8 Aug 19 03:00:13 homeserver kernel: ---[ end trace 5539ff0efe87359a ]--- Aug 19 03:00:13 homeserver kernel: RIP: 0010:_compound_head+0x0/0x11 Aug 19 03:00:13 homeserver kernel: Code: b8 98 0f 00 00 00 00 f0 7f 48 85 c2 0f 95 c0 0f b6 c0 c3 48 c1 ef 3a 83 ff 1f 0f 94 c0 48 83 ff 1e 0f 94 c2 09 d0 0f b6 c0 c3 <48> 8b 57 08 48 89 f8 f6 c2 01 74 04 48 8d 42 ff c3 e8 ea ff ff ff Aug 19 03:00:13 homeserver kernel: RSP: 0018:ffffc9000300fcc0 EFLAGS: 00010202 Aug 19 03:00:13 homeserver kernel: RAX: 0000000000000001 RBX: 0000000003ff4000 RCX: 000000000000011e Aug 19 03:00:13 homeserver kernel: RDX: 7c00000000000000 RSI: ffffea002dca5980 RDI: fffff8efd18eadc0 Aug 19 03:00:13 homeserver kernel: RBP: ffffc9000300fe00 R08: ffffea002dca5980 R09: ffff88903f2704f0 Aug 19 03:00:13 homeserver kernel: R10: ffff88903f269388 R11: 0000000000000297 R12: fffff8efd18eadc0 Aug 19 03:00:13 homeserver kernel: R13: 7c00003bbf463ab7 R14: ffff8881738a90b0 R15: ffff8881342f7000 Aug 19 03:00:13 homeserver kernel: FS: 0000000000000000(0000) GS:ffff88903f240000(0000) knlGS:0000000000000000 Aug 19 03:00:13 homeserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 19 03:00:13 homeserver kernel: CR2: fffff8efd18eadc8 CR3: 000000016cdc8001 CR4: 00000000003726e0 Aug 19 03:00:13 homeserver kernel: Fixing recursive fault but reboot is needed! Aug 19 03:01:25 homeserver CA Backup/Restore: docker stop -t 60 iobroker-live Aug 19 03:01:25 homeserver CA Backup/Restore: Stopping pihole Aug 19 03:01:29 homeserver kernel: veth25bc94e: renamed from eth0 Aug 19 03:01:29 homeserver CA Backup/Restore: docker stop -t 60 pihole Aug 19 03:01:29 homeserver CA Backup/Restore: Stopping plex Aug 19 03:01:34 homeserver kernel: vethe25af7c: renamed from eth0 Aug 19 03:01:34 homeserver CA Backup/Restore: docker stop -t 60 plex Aug 19 03:01:34 homeserver CA Backup/Restore: Backing Up appData from /mnt/cache/appdata/ to /mnt/user/backup/appdata/[email protected] Aug 19 03:01:34 homeserver CA Backup/Restore: Using command: cd '/mnt/cache/appdata/' && /usr/bin/tar -cvaf '/mnt/user/backup/appdata/[email protected]/CA_backup.tar.gz' * >> /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log 2>&1 & echo $! > /tmp/ca.backup2/tempFiles/backupInProgress Aug 19 03:01:40 homeserver emhttpd: read SMART /dev/sdb Aug 19 03:01:40 homeserver emhttpd: read SMART /dev/sdc Aug 19 03:06:52 homeserver CA Backup/Restore: Backup Complete Aug 19 03:06:52 homeserver CA Backup/Restore: Verifying backup Aug 19 03:06:52 homeserver CA Backup/Restore: Using command: cd '/mnt/cache/appdata/' && /usr/bin/tar --diff -C '/mnt/cache/appdata/' -af '/mnt/user/backup/appdata/[email protected]/CA_backup.tar.gz' > /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log & echo $! > /tmp/ca.backup2/tempFiles/verifyInProgress Aug 19 03:07:49 homeserver kernel: docker0: port 1(veth1639e18) entered blocking state Aug 19 03:07:49 homeserver kernel: docker0: port 1(veth1639e18) entered disabled state Aug 19 03:07:49 homeserver kernel: device veth1639e18 entered promiscuous mode Aug 19 03:07:50 homeserver kernel: eth0: renamed from vethb6282da Aug 19 03:07:50 homeserver kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth1639e18: link becomes ready Aug 19 03:07:50 homeserver kernel: docker0: port 1(veth1639e18) entered blocking state Aug 19 03:07:50 homeserver kernel: docker0: port 1(veth1639e18) entered forwarding state Aug 19 03:07:50 homeserver kernel: docker0: port 2(vethdc57047) entered blocking state Aug 19 03:07:50 homeserver kernel: docker0: port 2(vethdc57047) entered disabled state Aug 19 03:07:50 homeserver kernel: device vethdc57047 entered promiscuous mode Aug 19 03:07:50 homeserver kernel: docker0: port 2(vethdc57047) entered blocking state Aug 19 03:07:50 homeserver kernel: docker0: port 2(vethdc57047) entered forwarding state Aug 19 03:07:50 homeserver kernel: eth0: renamed from veth5a8fe96 Aug 19 03:07:50 homeserver kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethdc57047: link becomes ready Aug 19 03:07:51 homeserver kernel: eth0: renamed from veth1b282b7 Aug 19 03:07:51 homeserver kernel: docker0: port 3(veth7ceaa7c) entered blocking state Aug 19 03:07:51 homeserver kernel: docker0: port 3(veth7ceaa7c) entered disabled state Aug 19 03:07:51 homeserver kernel: device veth7ceaa7c entered promiscuous mode Aug 19 03:07:51 homeserver kernel: docker0: port 3(veth7ceaa7c) entered blocking state Aug 19 03:07:51 homeserver kernel: docker0: port 3(veth7ceaa7c) entered forwarding state Aug 19 03:07:52 homeserver kernel: docker0: port 3(veth7ceaa7c) entered disabled state Aug 19 03:07:52 homeserver avahi-daemon[16088]: Joining mDNS multicast group on interface vethdc57047.IPv6 with address fe80::244f:7aff:fe1a:7aca. Aug 19 03:07:52 homeserver avahi-daemon[16088]: New relevant interface vethdc57047.IPv6 for mDNS. Aug 19 03:07:52 homeserver avahi-daemon[16088]: Registering new address record for fe80::244f:7aff:fe1a:7aca on vethdc57047.*. Aug 19 03:07:52 homeserver kernel: eth0: renamed from veth4985384 Aug 19 03:07:52 homeserver kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth7ceaa7c: link becomes ready Aug 19 03:07:52 homeserver kernel: docker0: port 3(veth7ceaa7c) entered blocking state Aug 19 03:07:52 homeserver kernel: docker0: port 3(veth7ceaa7c) entered forwarding state Aug 19 03:07:52 homeserver avahi-daemon[16088]: Joining mDNS multicast group on interface veth1639e18.IPv6 with address fe80::405c:63ff:fe9f:3ded. Aug 19 03:07:52 homeserver avahi-daemon[16088]: New relevant interface veth1639e18.IPv6 for mDNS. Aug 19 03:07:52 homeserver avahi-daemon[16088]: Registering new address record for fe80::405c:63ff:fe9f:3ded on veth1639e18.*. Aug 19 03:07:53 homeserver kernel: eth0: renamed from veth1cad487 Aug 19 03:07:53 homeserver avahi-daemon[16088]: Joining mDNS multicast group on interface veth7ceaa7c.IPv6 with address fe80::d802:e9ff:fe7e:4437. Aug 19 03:07:53 homeserver avahi-daemon[16088]: New relevant interface veth7ceaa7c.IPv6 for mDNS. Aug 19 03:07:53 homeserver avahi-daemon[16088]: Registering new address record for fe80::d802:e9ff:fe7e:4437 on veth7ceaa7c.*. Aug 19 03:07:53 homeserver kernel: eth0: renamed from vethcbb2b36 Aug 19 03:07:54 homeserver kernel: eth0: renamed from veth301a4e7 Aug 19 03:07:54 homeserver CA Backup/Restore: ####################### Aug 19 03:07:54 homeserver CA Backup/Restore: appData Backup complete Aug 19 03:07:54 homeserver CA Backup/Restore: ####################### Aug 19 03:07:54 homeserver CA Backup/Restore: Deleting /mnt/user/backup/appdata/[email protected] Aug 19 03:07:54 homeserver CA Backup/Restore: Backup / Restore Completed Aug 19 03:14:35 homeserver emhttpd: read SMART /dev/sde Aug 19 03:24:57 homeserver emhttpd: spinning down /dev/sdb Aug 19 03:24:57 homeserver emhttpd: spinning down /dev/sdc Aug 19 03:30:27 homeserver emhttpd: spinning down /dev/sde Aug 19 03:40:07 homeserver emhttpd: read SMART /dev/sdb Aug 19 03:40:07 homeserver emhttpd: read SMART /dev/sdc Aug 19 03:40:16 homeserver crond[1260]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null Aug 19 03:57:20 homeserver emhttpd: spinning down /dev/sdb Aug 19 03:57:20 homeserver emhttpd: spinning down /dev/sdc Aug 19 04:23:48 homeserver emhttpd: read SMART /dev/sde Aug 19 04:41:56 homeserver emhttpd: spinning down /dev/sde Aug 19 07:12:50 homeserver webGUI: Successful login user root from 10.253.0.2 Aug 19 07:14:29 homeserver shutdown[11512]: shutting down for system reboot Aug 19 07:14:29 homeserver init: Switching to runlevel: 6 Aug 19 07:14:29 homeserver init: Trying to re-exec init Aug 19 07:14:31 homeserver kernel: mdcmd (37): nocheck cancel Aug 19 07:14:32 homeserver emhttpd: Spinning up all drives... Aug 19 07:14:32 homeserver emhttpd: spinning up /dev/sdd Aug 19 07:14:32 homeserver emhttpd: spinning up /dev/sde Aug 19 07:14:32 homeserver emhttpd: spinning up /dev/sdb Aug 19 07:14:32 homeserver emhttpd: spinning up /dev/sdc Aug 19 07:14:43 homeserver emhttpd: read SMART /dev/sdd Aug 19 07:14:43 homeserver emhttpd: read SMART /dev/sde Aug 19 07:14:43 homeserver emhttpd: read SMART /dev/sdb Aug 19 07:14:43 homeserver emhttpd: read SMART /dev/sdc Aug 19 07:14:43 homeserver emhttpd: read SMART /dev/nvme0n1 Aug 19 07:14:43 homeserver emhttpd: read SMART /dev/nvme1n1 Aug 19 07:14:43 homeserver emhttpd: read SMART /dev/sda