Hey there,

 

i hope this is the right place to ask ... not sure if its unraid itself or the app-backup-plugin... but often when the Server has some load (like when Backup is running), i cant access anything (not unraid, none of the dockers, no vm). So actually im not very sure whats going on now, but in the past it was while App-Backup was running (sometimes maybe parity check too).

buuut also sometimes when App-backup is running, i can access everything?!

 

So not sure how i should go forward to find out the exact problem and find the solution to fix it.

 

All i can see for sure is that i have about 250W power (USV Display is telling) compared to less than 100 in normal state (no load). So it looks like the server is quite busy

You might want to set appdata backup to not use multicore compression.

 

Or just schedule it in the middle of the night when it doesn't matter.

Edited November 17, 20241 yr by Kilrah

2 hours ago, Kilrah said:

You might want to set appdata backup to not use multicore compression.

 

Or just schedule it in the middle of the night when it doesn't matter.

Ill check that and try

 

it already starts at night but i think in meanwhile it takes 12+h because of immich plus zip

Just checked, i already turned off compression and verify. Meanwhile i forced reboot and log is empty, so not sure if it was due to appdata backup process or something else...

Ok final edit:

UrBackup and Immich were gone. So i assume some issue with "Update after Backup" setting or what else could lead to this situation? Also the parity check did start after i restarted server (Power off and on). So maybe additionally that was running but still server should stay accessible.

UrBackup could be restored, but immich was difficult because i could only restore settings. The container itself also contains the Pictures (about 2TB). At first, i did see no way but then i just downloaded and installed immich with some random settings and afterwards i restored with the xml file selected. It put all settings into the container when hitting "EDIT" but in the overview it did not use these settings. Only a "force update" did lead to immich working again

Edited November 17, 20241 yr by Saarlaender

After some more crashes/freezes, i think its related to the parity check. I did not find any setting regaring if it should use multi-core or something else that could lead to a freeze.

Any ideas?

If it crashes/freezes when running a parity check then the most likely culprit will be power as that is one time all drives are active,   Do you use power splitters as they can cause problems.

I have special hdd cases that supply power (server came with them). Also same config is running since a while (no new added hdds). Are there some logs somewhere to maybe check what happend?

51 minutes ago, Saarlaender said:

Are there some logs somewhere to maybe check what happend?

If you want logs that survive a server crash/reboot then you need to have the syslog server enabled.

 

thx, i did enable it and as soon as i can test again ill start the parity check and hope that ill get the logs. I did not enable copy to flash but i assume that everything will be placed in the selected folder (as its not during boot)

when will Syslog Server put first log files? I still dont see anything in the selected folder.

Just now, Saarlaender said:

when will Syslog Server put first log files? I still dont see anything in the selected folder.

Are you sure you have it configured correctly?    It might be worth posting a screenshot of the settings you have.

 

These are the settings. I just tried a new share (public instead of "daniel beeing private) but still empty

You need to set the Remote Syslog Server files to the local server IP address.

Ok did that and now there is a file 😄

Edited November 22, 20241 yr by Saarlaender

Ok guys, i think ive got the relevant part of the log here. This is the last stuff before Unraid Server becomes unresponsive and drains aloooot of power.

 

 

 

Nov 30 15:04:32 UnraidServer kernel: BUG: unable to handle page fault for address: 000004b4e785d680
Nov 30 15:04:33 UnraidServer kernel: #PF: supervisor read access in kernel mode
Nov 30 15:04:33 UnraidServer kernel: #PF: error_code(0x0001) - permissions violation
Nov 30 15:04:33 UnraidServer kernel: PGD 1405ba067 P4D 1405ba067 PUD 133e87067 PMD 105f16067 PTE 80000008a34fa067
Nov 30 15:04:33 UnraidServer kernel: Oops: 0001 [#1] PREEMPT SMP NOPTI
Nov 30 15:04:33 UnraidServer kernel: CPU: 11 PID: 6279 Comm: gunicorn Tainted: P           O       6.1.106-Unraid #1
Nov 30 15:04:33 UnraidServer kernel: Hardware name: Gigabyte Technology Co., Ltd. Z790 GAMING X AX/Z790 GAMING X AX, BIOS F9 12/14/2023
Nov 30 15:04:33 UnraidServer kernel: RIP: 0010:preempt_latency_start+0x1e/0x46
Nov 30 15:04:33 UnraidServer kernel: Code: cc cc cc f0 80 0f 02 c3 cc cc cc cc 65 8b 05 f8 32 f8 7e 25 ff ff ff 7f 39 c7 75 31 53 48 8b 5c 24 08 48 89 df e8 84 34 02 00 <85> c0 74 09 31 ff 31 db e8 77 34 02 00 65 48 8b 04 25 80 cb 01 00
Nov 30 15:04:33 UnraidServer kernel: RSP: 0018:ffffc90040f97f10 EFLAGS: 00010283
Nov 30 15:04:33 UnraidServer kernel: RAX: 0000000000000000 RBX: ffffffff81883901 RCX: 000004b4e785d680
Nov 30 15:04:33 UnraidServer kernel: RDX: 0000000000000001 RSI: ffffc90040f97f10 RDI: ffffffff81883901
Nov 30 15:04:33 UnraidServer kernel: RBP: ffff8882c0176000 R08: 0000000000000001 R09: 000014d633a55ba0
Nov 30 15:04:33 UnraidServer kernel: R10: 00000000000007d0 R11: 0000000000000000 R12: 0000000000000000
Nov 30 15:04:33 UnraidServer kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Nov 30 15:04:33 UnraidServer kernel: FS:  000014d633a566c0(0000) GS:ffff88907f6c0000(0000) knlGS:0000000000000000
Nov 30 15:04:33 UnraidServer kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 30 15:04:33 UnraidServer kernel: CR2: 000004b4e785d680 CR3: 00000003d0e50001 CR4: 0000000000772ee0
Nov 30 15:04:33 UnraidServer kernel: PKRU: 55555554
Nov 30 15:04:33 UnraidServer kernel: Call Trace:
Nov 30 15:04:33 UnraidServer kernel: <TASK>
Nov 30 15:04:33 UnraidServer kernel: ? __die_body+0x1a/0x5c
Nov 30 15:04:33 UnraidServer kernel: ? page_fault_oops+0x329/0x376
Nov 30 15:04:33 UnraidServer kernel: ? do_user_addr_fault+0x12e/0x465
Nov 30 15:04:33 UnraidServer kernel: ? _raw_spin_lock_irqsave+0x2c/0x37
Nov 30 15:04:33 UnraidServer kernel: ? exc_page_fault+0xfb/0x11d
Nov 30 15:04:33 UnraidServer kernel: ? asm_exc_page_fault+0x22/0x30
Nov 30 15:04:33 UnraidServer kernel: ? schedule+0x87/0xcc
Nov 30 15:04:33 UnraidServer kernel: ? schedule+0x87/0xcc
Nov 30 15:04:33 UnraidServer kernel: ? preempt_latency_start+0x1e/0x46
Nov 30 15:04:33 UnraidServer kernel: schedule+0x87/0xcc
Nov 30 15:04:33 UnraidServer kernel: __do_sys_sched_yield+0xa/0x11
Nov 30 15:04:33 UnraidServer kernel: do_syscall_64+0x65/0x7b
Nov 30 15:04:33 UnraidServer kernel: entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Nov 30 15:04:33 UnraidServer kernel: RIP: 0033:0x14d6d730e7db
Nov 30 15:04:33 UnraidServer kernel: Code: 73 01 c3 48 8b 0d 3d 46 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 18 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 0d 46 0f 00 f7 d8 64 89 01 48
Nov 30 15:04:33 UnraidServer kernel: RSP: 002b:000014d633a55e38 EFLAGS: 00000246 ORIG_RAX: 0000000000000018
Nov 30 15:04:33 UnraidServer kernel: RAX: ffffffffffffffda RBX: 000004b48e248840 RCX: 000014d6d730e7db
Nov 30 15:04:33 UnraidServer kernel: RDX: 00000000000007d0 RSI: 0000000000000001 RDI: 000014d633a55ba0
Nov 30 15:04:33 UnraidServer kernel: RBP: 000004b48e24885a R08: 000004b4e785d680 R09: 0000000000000001
Nov 30 15:04:33 UnraidServer kernel: R10: 000004b4e6f5d680 R11: 0000000000000246 R12: 000004b48e248870
Nov 30 15:04:33 UnraidServer kernel: R13: 000004b48e2488a0 R14: 0000000000000204 R15: 0000000000000001
Nov 30 15:04:33 UnraidServer kernel: </TASK>
Nov 30 15:04:33 UnraidServer kernel: Modules linked in: nvidia_uvm(PO) xt_nat xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap ipvlan xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter dm_crypt dm_mod md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp nvidia_drm(PO) nvidia_modeset(PO) kvm_intel i915 btusb nvidia(PO) btrtl kvm btbcm btintel bluetooth iosf_mbi drm_buddy i2c_algo_bit ttm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel drm_display_helper sha512_ssse3 sha256_ssse3 sha1_ssse3 ecdh_generic ecc drm_kms_helper aesni_intel crypto_simd cryptd mei_hdcp mei_pxp
Nov 30 15:04:33 UnraidServer kernel: gigabyte_wmi wmi_bmof rapl drm intel_cstate intel_uncore nvme intel_gtt i2c_i801 mei_me agpgart i2c_smbus r8169 nvme_core ahci mei i2c_core libahci realtek syscopyarea sysfillrect sysimgblt fb_sys_fops thermal fan tpm_crb tpm_tis video tpm_tis_core tpm wmi intel_pmc_core backlight acpi_tad acpi_pad button unix
Nov 30 15:04:33 UnraidServer kernel: CR2: 000004b4e785d680
Nov 30 15:04:33 UnraidServer kernel: ---[ end trace 0000000000000000 ]---
Nov 30 15:04:33 UnraidServer kernel: RIP: 0010:preempt_latency_start+0x1e/0x46
Nov 30 15:04:33 UnraidServer kernel: Code: cc cc cc f0 80 0f 02 c3 cc cc cc cc 65 8b 05 f8 32 f8 7e 25 ff ff ff 7f 39 c7 75 31 53 48 8b 5c 24 08 48 89 df e8 84 34 02 00 <85> c0 74 09 31 ff 31 db e8 77 34 02 00 65 48 8b 04 25 80 cb 01 00
Nov 30 15:04:33 UnraidServer kernel: RSP: 0018:ffffc90040f97f10 EFLAGS: 00010283
Nov 30 15:04:33 UnraidServer kernel: RAX: 0000000000000000 RBX: ffffffff81883901 RCX: 000004b4e785d680
Nov 30 15:04:33 UnraidServer kernel: RDX: 0000000000000001 RSI: ffffc90040f97f10 RDI: ffffffff81883901
Nov 30 15:04:33 UnraidServer kernel: RBP: ffff8882c0176000 R08: 0000000000000001 R09: 000014d633a55ba0
Nov 30 15:04:33 UnraidServer kernel: R10: 00000000000007d0 R11: 0000000000000000 R12: 0000000000000000
Nov 30 15:04:33 UnraidServer kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Nov 30 15:04:33 UnraidServer kernel: FS:  000014d633a566c0(0000) GS:ffff88907f6c0000(0000) knlGS:0000000000000000
Nov 30 15:04:33 UnraidServer kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 30 15:04:33 UnraidServer kernel: CR2: 000004b4e785d680 CR3: 00000003d0e50001 CR4: 0000000000772ee0
Nov 30 15:04:33 UnraidServer kernel: PKRU: 55555554
Nov 30 15:04:33 UnraidServer kernel: note: gunicorn[6279] exited with irqs disabled
Nov 30 15:04:33 UnraidServer kernel: note: gunicorn[6279] exited with preempt_count 1

 

Edited November 30, 20241 yr by Saarlaender

anyone can help? or should i move this crash report somewhere else?

Can'tr say based on that if it's a hardware or software issue, does it still only happen when running the appdata backup plugin?

it appears to happen whenever it wants

no connection to appdata backup, no connection to parity check (at least some crashes happend during idle)

I was hoping that this log entry helps

kernel: BUG: unable to handle page fault for address: 000004b4e785d680
kernel: #PF: supervisor read access in kernel mode
ernel: #PF: error_code(0x0001) - permissions violation

This can also be a hardware issue, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the individual docker containers. 

thx, i also plan to let memtest run to check the memory 100%. ive found a plugin doing that, but only half of the ram is then beeing checked

Since memtest is only definitive if it finds errors, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.

update: Ive finally got time to run memtest and everything ok.

In meanwhile i am quite sure it is related to Immich as it did occur a few times during massive image uploads, while appdata backup, client backups (urbackup) and also parity check went well

Ive found some post somewhere about crashes related to immich and AI. Should i go with a new topic to possibly get a confirmation about that?!
 

 

PS: log-server did not help as it apparenty also freezes. I did connect a monitor now and i hope it will show something when crashing again. Is there anything else i can do to track the crash?