July 28, 20241 yr My Unraid server is freezing completely, and I believe the issue is related to the Nvidia GPU and the Frigate containers, which use the GPU. System Information: Server Model: Lenovo P910 (recently exchanged due to previous issues) GPU: Nvidia Quadro M2000 Unraid Version: 6.12.6 Timeline of Events: Hardware Exchange: Recently exchanged Lenovo P910 server due to previous issues. Everything worked fine initially after the exchange. Cache Drive Crash: Cache drive crashed, causing all Docker containers to be disorganized (The mover wasn't properly configured to move Appdata to the array) Reconfiguration: Set up all Docker containers again. Everything worked fine initially. Nvidia driver was auto-updated, which caused the Quadro M2000 not to be recognized. Downgraded the driver, and everything seemed to work fine again. Server Freezing: After a few days, the server started freezing completely (no more WebUI access). The server ran for about a week without issues when the Frigate container wasn't started. Now in the test starting the Frigate container, the server immediately froze. (Usually the server runs some days, before not being accessible) Sidenote: I need to start the container twice because the USB enumerator changes (this wasn't necessary before the cache drive failure). Issues: Complete server freeze after starting the Frigate container. Unable to access WebUI when the server freezes. Difficulty obtaining a proper diagnostics file as the device completely freezes, and after a hard shutdown, only logs from the current session are available in the diagnostics file. Request: Can you help me troubleshoot this issue? Any advice or guidance on how to resolve the freezing problem or obtain more useful diagnostic information would be greatly appreciated. Additional Information: Attached is a screenshot showing the need to start the Frigate container twice due to USB enumerator changes. Attached 2 diagnostics - one before the container was started - one after the system hang again and was hard shut down tower-diagnostics-20240728-2047.zip tower-diagnostics-20240728-2017.zip
August 1, 20241 yr Author just happened again - cannot even SSH into the server - no idea how to get logs in such a state?
August 25, 20241 yr Author so sad there is absolutely no replies here ... just replaced my USB Stick as it was also being disconnected/lost from time to time ... was hoping this fixes it - today I lost the Unraid USB stick again ... that's so frustrating ... The full Lenovo ThinkStation P910 which Unraid is running on has recently also been changed .. can't think it is the machine causing those problems?!
August 25, 20241 yr Community Expert Is anything else using your GPU? Have you tried asking in the container support thread/forum? Have you mirrored your syslog to flash so it's saved before it locks? Edited August 25, 20241 yr by Michael_P
August 26, 20241 yr Author On 8/25/2024 at 12:48 PM, Michael_P said: Is anything else using your GPU? Have you tried asking in the container support thread/forum? Have you mirrored your syslog to flash so it's saved before it locks? No, actually Frigate is the only thing using my GPU - I've also changed it (to the same model which I got here multiple times) .. Is there any guidance to mirror the syslog to flash? This could eventually be really helpful ..
August 26, 20241 yr Community Expert https://docs.unraid.net/unraid-os/troubleshooting/diagnostics-information/#persistent-logs-syslog-server
August 29, 20241 yr Author it happened again - USB device is lost (see attached screenshot) /boot is empty as there is no USB device attached any longer after rebooting Unraid I've found this information in syslog - no clue what to do with it alot of mover entries like Quote Aug 28 03:43:22 Tower root: skip: /mnt/cache/appdata/influxdb2/engine/wal/25399bf07c4ffab4/autogen/32/_00003.wal Aug 28 03:43:22 Tower root: skip: /mnt/cache/appdata/influxdb2/engine/wal/25399bf07c4ffab4/autogen/33/_00001.wal Aug 28 03:43:22 Tower root: skip: /mnt/cache/appdata/unifi-network-application/logs/access.log Aug 28 03:43:22 Tower root: file: /mnt/cache/appdata/unifi-network-application/logs/tasks.log.1 Aug 28 03:43:23 Tower root: skip: /mnt/cache/appdata/unifi-network-application/logs/tasks.log Aug 28 03:43:23 Tower root: skip: /mnt/cache/appdata/frigate/frigate.db-wal Aug 28 03:43:23 Tower root: skip: /mnt/cache/appdata/frigate/frigate.db-shm Aug 28 03:43:23 Tower root: mover: finished some HDD entries about Smart and energy saving Quote Aug 28 11:55:03 Tower emhttpd: spinning down /dev/sdd Aug 28 11:55:13 Tower emhttpd: spinning down /dev/sdb Aug 28 11:56:10 Tower emhttpd: read SMART /dev/sdd Aug 28 11:56:20 Tower emhttpd: read SMART /dev/sdb Aug 28 12:26:03 Tower emhttpd: spinning down /dev/sdd Aug 28 12:26:13 Tower emhttpd: spinning down /dev/sdb Aug 28 12:27:11 Tower emhttpd: read SMART /dev/sdd Aug 28 12:27:20 Tower emhttpd: read SMART /dev/sdb nothing about any error - probably loglevel is set wrong or the entry is out of scope due to all those mover messages logged?
August 29, 20241 yr Community Expert The snippets posted should not be relevant, post the complete persistent syslog if you have it.
August 29, 20241 yr Author I found that unraid seems to have automatically created 3 diagnostics files yesterday at /boot/logs ? Maybe that's a trace Trying to dig into them by myself ... not sure what to look for tower-diagnostics-20240728-2017.zip tower-diagnostics-20240728-2021.zip tower-diagnostics-20240728-2047.zip syslog syslog-previous
August 29, 20241 yr Community Expert The one that matters is the syslog-previous, assuming that one was after a crash, but unfortunately there's nothing relevant logged, this can be a hardware issue, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.
August 29, 20241 yr Author 3 hours ago, JorgeB said: The one that matters is the syslog-previous, assuming that one was after a crash, but unfortunately there's nothing relevant logged, this can be a hardware issue, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one. I can be unlucky of course .. I was using a Lenovo P910 before - had issues (no longer booting - hard freezing) - exchanged the whole thing ... How are the odds the next Lenovo P910 has hardware issues again? I had frigate stopped for a while - did not encounter issues then ... as I got this USB device identitiy (the google coral adapter) switching all the time - could it be, that this also affects the usb stick Unraid is running on? Eventually some USB setting in the UEFI. Did you look into the diagnostics? I've searched with visual studio code through all files at once - there are alot of "usb" entries - nothing that shouts ERROR to my face - but as I said before - not sure what I'd be searching for
August 29, 20241 yr Community Expert 3 hours ago, nOerkH said: Did you look into the diagnostics? I did now, and I do see macvlan call traces there, curiously, there weren't any in the syslog-previous, suggesting it's not complete, in any case, macvlan call traces are known to crash servers, update to 6.12.13 since that issue is resolved since 6.12.11, and then retest.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.