kcossabo

correct, that is the symptom of unRAID not supporting this. This is a simple AB Test, tried 5 DAC Cables, one Fiber cable 4 switches, changed the OS and it works.

dockofthebay-diagnostics-20260628-2022.zip This is definitely a unRAID issue I have two identical servers, one with OPNsense one with unRAID, unRAID will not see the 10GE SFP ports, the other older version, was an unRAID server, never could get the SPF+ to work with unRAID, since I retired it from unRAID put Ubuntu on it, the SFP+ work great.

I have two Supermicro 5019D Server, 4C/8T Xeon D-2123T (Built on the X11SDV-8C-TP8F motherboard), one is setup as a OPNsense router and the other unRAID server. The OPNSense server SFP+ ports work with DAC cable to ubiquity no issues, unRAID build will not recognize them. root@DockOfTheBay:~# for i in /sys/class/net/eth*; do n=${i##*/}; \ printf "%-6s mac=%s " "$n" "$(cat $i/address)"; \ ethtool $n 2>/dev/null | grep -E "Speed:|Port:|Link detected:" | tr '\n' ' '; echo; \ done eth0 mac=3c:ec:ef:dd:48:56 Speed: 10000Mb/s Port: Twisted Pair Link detected: yes eth0.1 mac=3c:ec:ef:dd:48:56 Speed: 10000Mb/s Port: Twisted Pair Link detected: yes eth0.253 mac=3c:ec:ef:dd:48:56 Speed: 10000Mb/s Port: Twisted Pair Link detected: yes eth0.30 mac=3c:ec:ef:dd:48:56 Speed: 10000Mb/s Port: Twisted Pair Link detected: yes eth0.42 mac=3c:ec:ef:dd:48:56 Speed: 10000Mb/s Port: Twisted Pair Link detected: yes eth0.55 mac=3c:ec:ef:dd:48:56 Speed: 10000Mb/s Port: Twisted Pair Link detected: yes eth1 mac=3c:ec:ef:dd:3d:ba Speed: Unknown! Port: Twisted Pair Link detected: no eth2 mac=3c:ec:ef:dd:48:57 Speed: Unknown! Port: Other Link detected: no eth3 mac=3c:ec:ef:dd:3d:bb Speed: Unknown! Port: Twisted Pair Link detected: no eth4 mac=3c:ec:ef:dd:48:58 Speed: Unknown! Port: Other Link detected: no eth5 mac=3c:ec:ef:dd:48:59 Speed: 10000Mb/s Port: Direct Attach Copper Link detected: no eth6 mac=3c:ec:ef:dd:3d:bc Speed: Unknown! Port: Twisted Pair Link detected: no eth7 mac=3c:ec:ef:dd:3d:bd Speed: Unknown! Port: Twisted Pair Link detected: no root@DockOfTheBay:~# eth0 is the 10GbT connection and eth5 is the DAC. Thoughts how to turn it up? I saw some post on a melonox driver but have no reason to think this is mellonox. Switch side sees Up / connected, 10G full-duplex

FYI Version - 0.9.15.11 required me to change the PGID: from 100 to 1000 container was failing with addgroup: gid '100' in use adduser: unknown group cabernet su-exec: getpwnam(cabernet): Invalid argument addgroup: gid '100' in use adduser: unknown group cabernet su-exec: getpwnam(cabernet): Invalid argument Starting with UID : 99 Starting with UID : 99

Assumed Root Cause After 2 weeks of stability the assumed root cause is one of the following In compatibility with nVidia drivers and 7.1.4 - low probability as I assume there are a lot of installed and not having the issue In compatibility with nVidia drivers and 7.1.4 and other drivers and combinations on my server - higher probability as this get to more of a snowflake of some combination is causing the issue BAD nVidia card - high probability, as the removal did result in a stability. Though not crashing on earlier versions of unRAID. Each of these could be refined, but, not my jam to figure this out, just provide the family with stability.

When building the key, did not ‘think’ and used the standard / latest version (Server down for the upgrade of the USB DOM, working through the backup of the old key, and missed selecting the 6.x), and did an unintended upgrade. This was a 6.x to 7, and after the new DOM Key install, did not think the effort to go back right away (back to the server rack, unmount the server, open up the case, and remove the DOM from the mother board, back to the workstation, and build again) was worth it, it is the new stuff. Then the crashes started. If it this was immediate, and constant, I probably would have gone back. As you know the networking changed, and had a history of issues, moving down to a lower 7.x looked like a network issue risk, and after research the move back to 6.x looked risky after all the changes over the weeks of being at 7.x Was not an easy option, see above. Moving back in the 7.x introduces more known bugs, at the time of writing, there were no stated issues with nVIDIA and like most posted, assumed a memory issue, and worked it. Thank You - the stability of a DOM, also introduces the need for me to dive into the rack server, and pull the motherboard based USB drive. The upgrade was done by building a new key via the Mac Tool, for a new DoM after too many USB Stick failure. (Skip rant on the need to use a USB to boot a server). With 20+ Containers and this server being a key daily/hourly used services, going through a process like this was not an option. It would be if it crashed, after an hour, but it is days between them. So have a server (that is used hourly) down for a week, to feel comfortable it is ‘stable’ and layer the next fix on was not an option. I am extremely, extremely disappointed that there is so little diagnostic tools with this product that the guess work is the procedure to fix issues. I have a Syslog Server (SPLUNK), I have console Crash captures, and the best troubleshooting we have is, try this, or some ‘canned’ response of it is your memory. Working with two AI models, diagnosing 4 crashed console and SysLogs, everything pointed to the NVIDIA issue. Two weeks stability with the GPU and drivers gone. Moved the nVIDIA workloads to another server on a lower release, and have stability. (btw, why not move everything? The server with instability is Intel, with integrated GPU, that I use, the other servers are XEON and AMD and can not support the workloads). I do like the unRAID platform with three servers. The value it brings for both hosting and experimenting is great. Support on USB Failure is fast, I could not express how incredible getting support with in an hour on the weekend due to a USB Key failure is. Unfortunately I have learned. how fast the support is too many times. Not enterprise priced, not enterprise quality. PS. this post is over a month after the failures / crashes / ‘upgrade’ started, thus the frustration. Not sure who has the need for unRAID, and can support 5+ weeks of outages.. The good thing is not having a USB Drive, (holding back on the rant), and after too many server failures because of a $10 USB Stick, I built a DOM Based server that seems to be the community hive’s recommendation for stability. I was on a VERY VERY old version for the great advice you just mentioned. This Product is made great by great people like you willing to help. Thank You

Update. As suspected the server has become extremely stable with the Nvidia drivers and video card removed. Update. As suspected the server has become extremely stable with the Nvidia drivers and video card removed. With the removal of the Nvidia GPU and drivers the server has not failed with the removal of the With the removal of the Nvidia graphics card and associated drivers there hasn’t been a crash.

As stated, did the memory test for 10 hours clean Retest, is just wait until some unknown time to feel like the issue was the GPU, and then try it again........... The crashes were not "I did this, than that occurred" it is the server is down, oh crap, let me fix. In an effort for it to not crash I removed the nVidia drivers (as indicated), in a hope that I have stability until I return home this weekend. My assumptions are nvidia driver issue removed the drivers, so all the apps that use nvidia are not? I assume they will try CPU transcoding? or OLLAMA will just fail wait 4 days in this state if no crashes, then put them back in? Down grading the OS is my GUT move as it is the new OS that started this, but that does seem to have risk.

Server log from Syslog server (SPLUNK) and Diag, though I did remove the plugins before this was run. dockyard-diagnostics-20250908-0530.zip there are a lot of these Sep 7 20:51:46 10.10.9.1 Sep 7 20:51:46 DockYard kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0 Sep 7 20:51:46 10.10.9.1 Sep 7 20:51:46 DockYard kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x22:0x51:884) also Sep 7 20:54:18 10.10.9.1 Sep 7 20:54:18 DockYard kernel: #PF: supervisor instruction fetch in kernel mode Sep 7 20:54:18 10.10.9.1 Sep 7 20:54:18 DockYard kernel: BUG: unable to handle page fault for address: ffffffff824fc5c0 1757334860_88.csv.zip

Crash 9-7.pdfcrash 9-8.pdf Issue started when I upgraded the USB Key to a DoM and when building upgraded the server from 7.1.2 to 7.1.4 Issuetwo crashes in last 24 hours, three in the since upgrade 2 weeks ago Steps so farmemory test for 10 hours clean Put both screen grabs of the crash into ChatGPTThanks for sharing the second screenshot — this gives more detail. Here’s what’s happening: What the Trace ShowsThe panic here is in the RCU subsystem (Read-Copy-Update), which is a core Linux synchronization mechanism. Specifically: rcu: rcu_preempt kthread starved for 664912 jiffies! g=7744983 fqs=6 This means the RCU preempt kernel thread was stuck/starved for too long — the kernel watchdog considers this a deadlock. Backtrace shows: native_queued_spin_lock_slowpath raw_spin_lock resched_curr rcu_gp_fqs_loop rcu_gp_kthread → This indicates the kernel got stuck trying to grab a spinlock during RCU grace period cleanup. Why This HappensKernel/Driver Bug Most common cause. Since you’re on Unraid 7.1.4 (kernel 6.12.24), this could be a regression in RCU + ZFS or RCU + NVIDIA drivers. Both ZFS and GPU drivers are heavy users of locking. Resource Starvation If one CPU core is pegged at 100% (tight loop), the RCU threads can get starved and trigger this warning. Check if you had a VM, Plex, Tdarr, or GPU job maxing out CPU/GPU before the crash. Hardware Edge Case Bad RAM or unstable overclocking (if enabled) can cause locks to never resolve. How to Safely RestartSince your system is already throwing RCU stalls, it may not shut down cleanly. Do this: Try a graceful stop first (recommended): /usr/local/sbin/powerdown * This will attempt to stop the array, unmount disks, and kill processes. If that hangs for >2–3 minutes: sync reboot -f * `sync` flushes writes to disk. * `reboot -f` forces a reboot without waiting for hung threads. Next Steps After RestartCheck syslog (Tools → System Log) for warnings before the panic. Update: Make sure you’re on the latest BIOS for the ASRock Z690 Extreme. See if Unraid 7.1.5 (or newer kernel) is available — 6.12.24 has had reports of RCU-related hangs. Isolate: Temporarily disable ZFS or NVIDIA drivers to test stability. Run memtest from the Unraid boot menu. ⚠️ Bottom line: This wasn’t you doing anything wrong — this is most likely a kernel/module bug aggravated by ZFS or NVIDIA drivers under 6.12.24. Do you want me to build you a step-by-step recovery & isolation plan (BIOS update → memtest → driver isolation → kernel update) so you can nail down whether this is software or hardware? Fix common issuesShare appdata set to cache-only, but files / folders exist on the array [no idea how to fix] Share plex_cache set to cache-only, but files / folders exist on the array [no idea how to fix] The plugin folder.view.plg is not known to Community Applications and is possibly incompatible with your server [action plan to remove, which sucks it is GOLD] other actionsgoing to disable nVIDIA as well

I can not see the actual containers to add to the folder. All of the existing folders are in the same state, none of the existing containers are there, and can not add new ones.

problem when people do this is that the links change and there is no permanent answer.

Went to the config of the container and edited the IP to force a docker view docker run -d --name='ActualServer' --net='eth2.40' --ip='10.10.20.155' --pids-limit 2048 -e TZ="America/New_York" -e HOST_OS="Unraid" -e HOST_HOSTNAME="Zeus" -e HOST_CONTAINERNAME="ActualServer" -e 'TCP_PORT_5006'='5006' -l net.unraid.docker.managed=dockerman -l net.unraid.docker.webui='http://[IP]:[PORT:5006]' -l net.unraid.docker.icon='https://github.com/actualbudget/actual/raw/master/packages/desktop-electron/icons/icon.png' -v '/mnt/remotes/10.10.10.2_ActualBudget/':'/data':'rw' 'actualbudget/actual-server' 7d40c0fb3e922314ed6d887ccd12db4925af07d37ed21597888fd3aaf66fc9de docker: Error response from daemon: error while creating mount source path '/mnt/remotes/10.10.10.2_ActualBudget': mkdir /mnt/remotes/10.10.10.2_ActualBudget: file exists. The mount point is mounted in the MAIN Open console root@Zeus:/mnt/remotes# ls 10.10.10.2_ActualBudget /bin/ls: cannot access '10.10.10.2_ActualBudget': Stale file handle go to main page and unmount the share Feb 28 08:27:35 Zeus unassigned.devices: NFS mount failed: 'mount.nfs: access denied by server while mounting 10.10.10.2:/volume1/ActualBudget'. Feb 28 08:27:35 Zeus unassigned.devices: Remote Share '10.10.10.2:/volume1/ActualBudget' failed to mount. root@Zeus:/mnt/remotes# ls 10.10.10.2_ActualBudget /bin/ls: cannot access '10.10.10.2_ActualBudget': Stale file handle check the published drives root@Zeus:/mnt/remotes# showmount -e 10.10.10.2 Export list for 10.10.10.2: /volume1/ROM 10.10.0.0/16 /volume1/hoarder 10.10.0.0/16 /volume1/DVR 10.10.0.0/16 /volume1/AudioBooks_Indexed 10.10.0.0/16 /volume1/Paperless-ngx 10.10.0.0/16 /volume1/CD_Backup 10.10.0.0/16 /volume1/homes 10.10.0.0/16 /volume1/LightRoom Backup 10.10.0.0/16 /volume1/Plex-Music-m4a 10.10.0.0/16 /volume1/consolidation 10.10.0.0/16 /volume1/Zeus 10.10.0.0/16 /volume1/photoshare 10.10.0.0/16 /volume1/audiobooks-inbound 10.10.0.0/16 /volume1/Local-copyof-Brandon-Videos 10.10.0.0/16 /volume1/Docker_Configs 10.10.0.0/16 /volume1/Plex-AudioBooks 10.10.0.0/16 /volume1/Downloads 10.10.0.0/16 /volume1/Audible Book Storage 10.10.0.0/16 /volume1/Plex Videos 10.10.0.0/16 root@Zeus:/mnt/remotes# Go to the NAS - An encrypted share - Not mounted - needed to enter encryption key - share mounts Go to UnRAID - Mount remote share - Start Docker # working.

zeus-diagnostics-20250228-0813.zip Can not launch Container "Actual Server" no logs on system, no logs on app (that I can find) Click start, get error Hit ok, app does not start.

Problem fixed on Synology!!! Overview: NAS mounts failed after the plugin upgrade. Root determination was that one of the three Synology NAS would not respond to a > showmount -e {host} And the plugin was now requiring the response as a health check or something. Other NASs on network were responding. Resolution: On Synology Control Panel -> Networking -> General -> Advance Setting -> (uncheck) Reply to ARP request if the target IP address is Identical to a local address configured on the incoming interface. Turning this option off enabled unRAID to receive a response to the showmount command. Packet traces of the command showed Problem Troubleshooting 1. Portmap (rpcbind) Lookup The command starts by performing a portmap lookup to determine the port associated with the NFS mount daemon on 10.10.10.2. Packets 104–114: A connection is established on port 111 (typically used by the rpcbind service), and the client sends a V4 GETADDR call. The GETADDR response is received, and then both the client and server initiate a graceful close of the connection, which completes without errors. 2. Connection Attempt to NFS (Port 2049) After resolving the mount daemon’s port (likely 2049), the client tries to establish a connection directly to port 2049. Packets 284–296: The client and server successfully complete the TCP three-way handshake on port 2049, but soon after, the client sends only a single byte of data (Len=1). This could be a part of the showmount -e command query. Following the initial data, the client immediately sends a FIN to terminate the connection. The server responds with an ACK, but also sends a duplicate ACK (Packet 291), which may indicate either minor packet reordering or an unexpected early termination. Shortly afterward, the server sends a FIN and then a RST (Packets 292–293), suggesting the server is forcefully terminating the session. 3. Repeated Portmap Lookups Packets 297–306: The client repeats the connection to port 111 to perform another V4 GETADDR call, indicating that it’s attempting to re-establish the lookup for NFS service ports. This cycle of portmap lookups and brief connections to port 2049 continues (as seen with additional V4 GETADDR calls in Packets 322–326). The Synology NAS that was having the issue has multiple interfaces. Two interfaces are configured for the same subnet (only one active). From Synology's documentation https://kb.synology.com/en-ca/DSM/help/DSM/AdminCenter/connection_network_route?version=7 " if you have more than two network interfaces in the same LAN, enabling this option causes network flow on each interface to be transported separately. " The inactive interface is a built-in 1GE port, with the active interface an expansion card for 10GE This seems like a Synology BUG, as the second interface is disabled, but the packet reordering was causing the issue.