nraygun

Hmmm, similar sort of thing happening for two users with bittorrent and crashing server with partially inoperative GUI. Do you know if anyone else is having these seemingly related problems? Also, I'm starting to think it's a kernel bug of somesort. Google shows a handful of reports of this null pointer dereference. How can we get Limetech folks involved in this? Or do we need to jump on another thread?

I spoke too soon. The server crashed again. Had to force a reboot at command line. I was doing some things in Plex at the time and things just stopped. Unfortunately, I had syslog turned off. So now it's back on. Continuing to monitor.

@trwolff04 Seems like I am having more problems since going to unRaid 6.11.x. No problems before. In addition to this crash condition, my DelugeVPN goes down overnight sometimes(every day or two). A simple restart brings it back up. Trying to look into that as well in a separate posting.

Oh well, so much for that idea. It went into the error state again. I did a restart and it's back up and seeding a file. Anyone have any ideas why this keeps happening?

I was doing some things in Plex (trying to add a cartoon series) and then all of a sudden the server crashed. I could not log into the GUI, but some of the shares still worked. And I could ssh into it. I managed to grab the dmesg before I did a reboot. I could not do a "powerdown" but I could do a "shutdown -h now". It's doing a parity check as I type this. Any ideas what happened here? [3547852.627297] BUG: kernel NULL pointer dereference, address: 0000000000000000 [3547852.627882] #PF: supervisor write access in kernel mode [3547852.628434] #PF: error_code(0x0002) - not-present page [3547852.628975] PGD 0 P4D 0 [3547852.629511] Oops: 0002 [#1] PREEMPT SMP PTI [3547852.630051] CPU: 8 PID: 19396 Comm: shfs Tainted: G W I 5.19.17-Unraid #2 [3547852.630621] Hardware name: Dell Inc. PowerEdge R710/XXXXXX, BIOS 6.6.0 05/22/2018 [3547852.631155] RIP: 0010:__rb_erase_color+0xe7/0x1ca [3547852.631732] Code: 8b 6b 10 f6 45 00 01 75 2f 4c 8b 75 08 48 89 d8 48 89 ee 31 c9 48 83 c8 01 4c 89 ea 48 89 df 4c 89 73 10 48 89 5d 08 4c 89 f5 <49> 89 06 e8 d8 fe ff ff 2e e8 9a be 79 00 48 8b 45 10 48 85 c0 74 [3547852.632927] RSP: 0018:ffffc9000b057d18 EFLAGS: 00010286 [3547852.633522] RAX: ffff888cc8e3a621 RBX: ffff888cc8e3a620 RCX: 0000000000000000 [3547852.634153] RDX: ffff888370fb0cc8 RSI: ffff888cc8e3a1a0 RDI: ffff888cc8e3a620 [3547852.634759] RBP: 0000000000000000 R08: ffff888acf65f520 R09: ffffc9000b057cd0 [3547852.635361] R10: 00001466f75f7000 R11: 00001466f75f8000 R12: ffffffff811d547a [3547852.636008] R13: ffff888370fb0cc8 R14: 0000000000000000 R15: ffff888cc8e3a600 [3547852.636646] FS: 000014670d18e6c0(0000) GS:ffff88902fb00000(0000) knlGS:0000000000000000 [3547852.637263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [3547852.637876] CR2: 0000000000000000 CR3: 0000000855544004 CR4: 00000000000226e0 [3547852.638513] Call Trace: [3547852.639169] <TASK> [3547852.639824] __do_munmap+0x1c0/0x2e2 [3547852.640466] mmap_region+0x10d/0x45d [3547852.641070] do_mmap+0x3c1/0x42d [3547852.641664] vm_mmap_pgoff+0xbb/0x112 [3547852.642255] ksys_mmap_pgoff+0x138/0x166 [3547852.642835] do_syscall_64+0x6b/0x81 [3547852.643407] entry_SYSCALL_64_after_hwframe+0x63/0xcd [3547852.643971] RIP: 0033:0x14670d7a47f3 [3547852.644526] Code: ef e8 61 b8 ff ff eb e7 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 89 ca 41 f7 c1 ff 0f 00 00 75 14 b8 09 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 25 c3 0f 1f 40 00 48 8b 05 d9 95 0d 00 64 c7 [3547852.645726] RSP: 002b:000014670d18db88 EFLAGS: 00000246 ORIG_RAX: 0000000000000009 [3547852.646339] RAX: ffffffffffffffda RBX: 00005641df847b50 RCX: 000014670d7a47f3 [3547852.646954] RDX: 0000000000000003 RSI: 0000000000001000 RDI: 0000000000000000 [3547852.647569] RBP: 00005641df847a20 R08: 00000000ffffffff R09: 0000000000000000 [3547852.648180] R10: 0000000000000022 R11: 0000000000000246 R12: 00000000000000b8 [3547852.648787] R13: 00005641df847a88 R14: 0000000000000000 R15: 000014670d18dc80 [3547852.649398] </TASK> [3547852.650000] Modules linked in: vhost_net vhost tap kvm_intel kvm macvlan md_mod tls xt_mark xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle tun vhost_iotlb veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc ipmi_devintf iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bnx2 mgag200 drm_shmem_helper i2c_algo_bit drm_kms_helper drm sr_mod cdrom ipmi_ssif backlight intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd intel_cstate i2c_core input_leds syscopyarea intel_uncore mpt3sas led_class joydev ahci sysfillrect ata_piix sysimgblt fb_sys_fops [3547852.650111] libahci raid_class scsi_transport_sas wmi ipmi_si acpi_power_meter button acpi_cpufreq unix [last unloaded: md_mod] [3547852.656724] CR2: 0000000000000000 [3547852.658803] ---[ end trace 0000000000000000 ]--- [3547852.674564] RIP: 0010:__rb_erase_color+0xe7/0x1ca [3547852.675323] Code: 8b 6b 10 f6 45 00 01 75 2f 4c 8b 75 08 48 89 d8 48 89 ee 31 c9 48 83 c8 01 4c 89 ea 48 89 df 4c 89 73 10 48 89 5d 08 4c 89 f5 <49> 89 06 e8 d8 fe ff ff 2e e8 9a be 79 00 48 8b 45 10 48 85 c0 74 [3547852.676900] RSP: 0018:ffffc9000b057d18 EFLAGS: 00010286 [3547852.677696] RAX: ffff888cc8e3a621 RBX: ffff888cc8e3a620 RCX: 0000000000000000 [3547852.678495] RDX: ffff888370fb0cc8 RSI: ffff888cc8e3a1a0 RDI: ffff888cc8e3a620 [3547852.679286] RBP: 0000000000000000 R08: ffff888acf65f520 R09: ffffc9000b057cd0 [3547852.680055] R10: 00001466f75f7000 R11: 00001466f75f8000 R12: ffffffff811d547a [3547852.680811] R13: ffff888370fb0cc8 R14: 0000000000000000 R15: ffff888cc8e3a600 [3547852.681565] FS: 000014670d18e6c0(0000) GS:ffff88902fb00000(0000) knlGS:0000000000000000 [3547852.682360] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [3547852.683168] CR2: 0000000000000000 CR3: 0000000855544004 CR4: 00000000000226e0 [3548148.759343] br0: port 2(vnet1) entered disabled state [3548148.762004] device vnet1 left promiscuous mode [3548148.762668] br0: port 2(vnet1) entered disabled state [3548940.712053] elogind-daemon[1579]: New session c20 of user root. [3549039.152378] elogind-daemon[1579]: Removed session c20. flores-diagnostics-20230408-2107.zip

So far, so good. The recreated container seems to be staying up. It even withstood an appdata backup. Continuing to monitor.

I just recently updated the Nextcloud, Maria, and Swag containers on my 6.11.5 unRaid server. Everything seemed to go OK. I noticed today that when I tried to sync my iPhone, I got a "503 server temporarily unavailable". After looking around at logs and such, it seemed to correct itself and now files are syncing. Should I be concerned? Any ideas why I got this 503 error? Maybe a drive was spun down or something?

Hasn't happened again so closing this thread. Then again, I don't think the container has been had any updates available.

Closing this thread. Hasn't happened since. Shutting off syslog.

Mine stays up for a while, then overnight it goes into the state you are seeing. I think I have a good ovpn file since it does work upon a restart. We'll see what happens tomorrow morning since I recreated the container.

With the AppData backup app disabled, my DelugeVPN still fails overnight. I'll try rebuilding the container unless anyone else has recommendations. (deleted DelugeVPN from Docker tab, then Apps, Previous Apps, reinstall)

This keeps happening. Is there something that needs to be done to the container after a shutdown to make it come back up with a start up? For now, I'll disable the auto backup plugin and see if it stays up for the rest of the week.

No GPU passthrough. It serves its purpose without one for now.

I was able to get it to work with OCC 2.55.0.0. Looking at the page that hosts OCC, I don't see a reference to what version of OC it supports. I've seen in other posts something about 0.7.7, but the latest OpenCore at the https://github.com/thenickdude/KVM-Opencore/releases site says it's version v19. How can I find what version of OC I have? And how can I find what version of OCC supports what version of OC?

I couldn't get Monterey to work so I'm back to Big Sur. Using the new Macinabox container, do I have to use a particular version of Opencore Configurator to complete the installation or should I just go grab the latest one? When I run the latest 2.67.1.0, I get a popup that says "You seem to have an old version of OpenCore bootloader. This app creates a configuration file for 0.9.0 and higher versions. Use at your own risk."

It croaked again. This time it was downloading. Anyone have any suggestions?

It happened again. This time it was seeding. Restarting the container fixes it. Any ideas what's going on here?

Update on my connection refused issue. Without a seeding file, the container seems to shutdown and restart fine. It might be only when there's a seed in motion that it doesn't restart after a backup.

Still monitoring for an update to Boinc. Stay tuned.

Connection refused after container restart I have the AppData Backup plugin setup to run every night. This includes shutting down the containers, then restarting them. It appears that on the restart of DelugeVPN, the container is not accessible. I get a repeating pattern of errors below. unRaid shows the container is running but I can't access the webUI. To get it to work again, I have to restart the container via the unRaid GUI. Any ideas why it goes into this state? (there was an IP in the log so I hid it as iphidden) 2023-02-28 08:25:12,889 DEBG 'start-script' stdout output: [info] Starting OpenVPN (non daemonised)... 2023-02-28 08:25:12,904 DEBG 'start-script' stdout output: 2023-02-28 08:25:12 DEPRECATED OPTION: --cipher set to 'AES-256-CBC' but missing in --data-ciphers (AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305). OpenVPN ignores --cipher for cipher negotiations. 2023-02-28 08:25:12,904 DEBG 'start-script' stdout output: 2023-02-28 08:25:12 WARNING: file 'credentials.conf' is group or others accessible 2023-02-28 08:25:12,904 DEBG 'start-script' stdout output: 2023-02-28 08:25:12 OpenVPN 2.6.0 [git:makepkg/b999466418dddb89+] x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jan 25 2023 2023-02-28 08:25:12 library versions: OpenSSL 3.0.8 7 Feb 2023, LZO 2.10 2023-02-28 08:25:12,905 DEBG 'start-script' stdout output: 2023-02-28 08:25:12 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts 2023-02-28 08:25:12,909 DEBG 'start-script' stdout output: 2023-02-28 08:25:12 TCP/UDP: Preserving recently used remote address: [AF_INET]iphidden:1194 2023-02-28 08:25:12 UDPv4 link local: (not bound) 2023-02-28 08:25:12 UDPv4 link remote: [AF_INET]iphidden:1194 2023-02-28 08:25:13,018 DEBG 'start-script' stdout output: 2023-02-28 08:25:13 read UDPv4 [ECONNREFUSED]: Connection refused (fd=3,code=111) 2023-02-28 08:25:14,277 DEBG 'start-script' stdout output: 2023-02-28 08:25:14 read UDPv4 [ECONNREFUSED]: Connection refused (fd=3,code=111) 2023-02-28 08:25:18,990 DEBG 'start-script' stdout output: 2023-02-28 08:25:18 read UDPv4 [ECONNREFUSED]: Connection refused (fd=3,code=111) 2023-02-28 08:25:27,227 DEBG 'start-script' stdout output: 2023-02-28 08:25:27 read UDPv4 [ECONNREFUSED]: Connection refused (fd=3,code=111) 2023-02-28 08:25:42,547 DEBG 'start-script' stdout output: 2023-02-28 08:25:42 [UNDEF] Inactivity timeout (--ping-exit), exiting 2023-02-28 08:25:42,547 DEBG 'start-script' stdout output: 2023-02-28 08:25:42 SIGTERM[soft,ping-exit] received, process exiting

Roger that. Swag went down overnight. Just restarted the container and it and Nextcloud are back. Parity check still running from the shutdown due to the crash. Just getting a general instability feeling lately. I setup the syslog server so now we wait.

Had a lockup of my server today and trying to pin down if there's some bigger problem in play. I was out and about and had checked for a document in Nextcloud while I was out so I know it was working 3-4 hours previously. When I got home, I found the server was down. It was not accessible from the web GUI, I could not ping it, and the containers were down(couldnt get to Plex or Nextcloud). When I tried to interact with it over the iDRAC virtual console, that was locked up too. So I forced a reboot and it came up. Whew! Googling around I found a few threads, but they dont seem similar to my situation. I've not changed hardware. My Deluge is up to date. However, on the Deluge container, I did notice there was a problem with it when the server came back up. I restarted it and it's working now. https://forums.unraid.net/topic/130543-complete-server-lockup-611x/#comment-1187761 https://forums.unraid.net/topic/135518-server-locking-up/#comment-1232774 https://forums.unraid.net/bug-reports/stable-releases/crashes-since-updating-to-v611x-for-qbittorrent-and-deluge-users-r2153/?page=8&tab=comments The only other thing out of the ordinary is I had an XFS corruption issue on the cache drive a week or so ago. Following the repair process documented on the website seemed to have fixed that issue. I'll setup syslog if I can figure out how. I doubt the diags will show anything, but they're posted. Any ideas? flores-diagnostics-20230226-1756.zip

Following this thread. Mine locked up this afternoon. I couldn't ping it nor bring up the web gui. When I did a remote session into it with iDRAC, it seemed unresponsive/locked up. I went down to the basement to check it out and it appeared on.

Thanks @JorgeB! Looks like drive in question was the cache drive. I ran the repair and it completed. See below. It mentions something about lost+found but I don't see that in my Cache drive. Am I good to go? Also, I've never seen this before. Is this a common thing? Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done

Crapola! I saw this morning that the appdata backup plugin didn't complete. Looks like it couldnt stop the docker containers. I tried to stop them but they wouldn't. I then tried to kill off any zombie processes. I could not kill them nor their parents. I then decided to reboot the server. I can't tell at what stage the errors started, but it's stuck at a "Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0f42f168 dinode". It then says to run xfs_repair and shows the first 128 bytes of corrupted metadata buffer. Looks like it's sdb1 that has the problem. The situation looks similar to what's going on in the link below. Just want to confirm the right steps to recover. 1. Power cycle the server 2. If it comes up, then do the test and repair here: https://wiki.unraid.net/Check_Disk_Filesystems#Checking_and_fixing_drives_in_the_webGui (not sure what to do if it doesn't come back up) Is this correct or should I do something else? Am I have a major server problem here or are XFS corruptions fairly common? PS - I can't provide logs with the server in this state. I