onno204

Hi, I would still like to keep this thread open. I am experiencing the exact same issue as you are describing. I also found the posts you mentioned. When the server is crashing/lagging I see lot's of "access" to /boot/bzfirmware. (using `inotifywait --timefmt %c --format '%T %_e %w %f' -mr /boot`) (When the server is running fine (no gui opened) all files in boot/config/shares/ gets read every second, I have no idea if this is normal behaviour.) TL;DR: I think this is being caused by either SabNZBD and/or the -arr suite. Whenever a file is being downloaded the `Mon May 13 15:42:37 2024 ACCESS /boot/ bzfirmware` logs popup like crazy. I have also installed Synchting, just like you. I didn't have time to properly debug this, but maybe it's related as well. Can you relate to this -arr suite conclusion? Is there possibly anything happing with your files when your crash occures? @deienache Or are you moving/uploading/reading any big files on your array/cache? Long story: Symptoms When crashing my flash drive is getting a lot of reads(just as you describe). The GUI is getting slower and slower each second. I even noticed a 464 MB/s reads speed. I have tried some troubleshooting but it's really hard to do since the whole server slowsdown (All docker-containers not responding, GUI not loading and ssh really really slow), VM's however keep running fine without any problems and the internet(using the same ethernet port as unraid) keeps working for the VM's. This indicates some kind of issue with the unraid OS. (@deienache I assume your Nextcloud VM also kept running without any problems?) History The first time I noticed this kind of slowdown was on April 5th 2024. (Unraid version 6.12.6. I upgraded to 6.12.10 on 2024 April 23) I didn't get the change to properly debug the problem until a few days ago. Since April the 5th Unraid slowed down/crashed about +-4 times, a quick unraid reboot fixed the issue for a few days/weeks. (These crashes are noticed quickly since unraid also runs my AdGuard docker container, which causes DNS issues when unraid is crashing) Investigation To investigate this issue I wanted to install some monitoring (Grafana, prometheus, cadvisor, prometheus-node-exporter), I started doing this 2 days ago. After I installed the `prometheus_node_exporter.plg` plugin 2 days ago (saterday evening, 11-may-2024) I noticed the slowdown starting. I removed the prometheus_node_exporter.plg file and rebooted shortly after and the server ran fine the next day(sunday). I this morning(monday) reinstalled the prometheus_node_exporter.plg plugin and in about one hour later the slowdown started again. I rebooted and now it is running fine for a few hours. (sadly unable to create a diagnostics export of this crash) The prometheus_node_exporter.plg could be a coincidence since both times I used the unraid-GUI heavily(mostly updating containers and browsing apps) while I didn't use the unraid-GUI this sunday. This however does not explain the previous slowdowns last month, those were completely random, I wasn't using the unraid-GUI at all. This however got me thinking, maybe it is plugin or docker related? When rebooting or installing a docker-container I see a lot of reads to the /boot/bzfirmware file. After the crash of 2 days ago I ran `# docker ps`. This showed 3 containers unhealthy (idk why, they are healthy normally). Maybe some containers get into a docker boot loop causing the /boot/bzfirmware to be read, which cascades to more reboots and reads? Network data Everytime the crash occurs the network data/second is peaking. This seems to be happing everytime. I am unsure if it maybe is the omada-controller container not logging the correct data because the server is lagging. Or maybe the docker network being overloaded causing the unraid OS to hang. (I still think it is weird the VM's network keep working without any problem and are accessable but non of the unraid-os/container resources are accessable) (This is network data from my omada-switch, which get's logged by the omada-controller which is running on unraid(docker). The data-gap is a reboot of unraid) This got met thinking again. Why is there a high spike in network right before it crashes. At first I thought is was either the docker network overloading, or a container going haywire. (I have an mDNS relay container to relay 'pings' to other vlan's) Possiblities Now I finally found something, most of the crash-dates can be matched with SabNZBD downloading a file. (This is also the case for the crashes of today and saterday evening) After Sab downloads a file, it is moved by Radarr, which is then import by Jellyfin and encoded by Tdarr. My current theorie is that these crashes are related to (large) files being moved or modified. In your diagnostics I can see you also have Tdarr installed. Is it possible this is causing the issue? Is anything happing with your files when your Unraid server crashes? If so I can investigatie furthing into Tdarr. @deienache Ending I don't have an anonymized diagnostics of the crash. I do however have 2 "personal" diagnostics (If needed a can share certain files, but I am not too keen on sharing the full non-anonymized diagnostics) I currently have no idea how to fully trigger the crash. I tried (re)installing some more plugins and docker apps to see if that causes any problems. But the server is still running fine for now. The closed I have come to imitating the crash is triggering a file download for SabNZBD. This causes a low of `ACCESS /boot/ bzfirmware` logs which is similar behaviour when crashing. I do have anonymized logs after this is happing. I hope this will help to find the issue we are both facing. Your answers to my questions are appreciated! (ps. I don't think downgrading should be the final solution, this will causes problems in the long term if it issue isn't fixed in a future update)