shadowlord

So the problem turned out to be within the Immich container. Somehow, the number of background thumbnail generation processes was set to a super-high number (30) and this was causing my system to slow down. It was hard to pin down that Immich and specifically the background-thumbnail-generator as the problem. I think there needed to be very specifically conditions to take place (like a user uploading a large batch of photos that needed thumbnail generation) in order for the problem to be triggered. Ultimately, logging in as tImmich's Administrator user and resetting the number of background jobs to defaults values has addressed this high CPU problem for me.

Unfortunately, the Unraid server crashed (likely due to high load) before I could create the output file. Here is the output about 40 minutes after a forced reset-button style reboot. I realize it's not the same, however, part of me also suspects this may not be caused by the "du" program after all, but maybe by one of the docker containers that's running away. I've been battling this problem with a few months now, but in the past, I could go about a month without having to reboot. ps.output.txt server03-diagnostics-20250519-2328.zip

I am running Unraid 7.0.1 and my server has suddenly because sluggish with very high IOwaits. I didn't realize my docker /appdata and /docker folders were not on my SSD-cache, so I moved them over. Now it seems like there are dozens of "du -sb" processes running, and I see the system load get higher and higher, to the point where I have to reboot the system daily just to get to the Web-UI. Here are my diagnostics a few minutes after my last reboot (while the system was still responsive). I am reasonable confident is it not docker, because I shutdown my dockers containers one-by-one before the last reboot and the system did not become more responsive. Loads are slowly creeping upwards . 16:22:19 up 41 min, 1 user, load average: 52.45, 47.13, 34.34 server03-diagnostics-20250519-1606.zip

Deleted 2 very old keys in my flash drive's /config directory, Trial.key and Plus.key. Then I got the expected "Registration key / USB Flash GUID mismatch" page. Clicked on the Replace Key button and this took me to the Unraid website. I had to link my old key to my Unraid.net account. Returned to the Unraid webGUI and went to the "Registration key / USB Flash GUID mismatch" page. Clicked on replace key again, and I was presented with "Replace Key" page on the unraid.net website with a big red "GUID to be blacklisted" containing my old USB key GUID... also with the a New GUID for my new key. After acknowledging, I clicked Confirm. I then got windows that said "success" at the top, but indicated that while my "Pro Key Replaced Successfully" (sic), there was a "Post Install LIcense Key Error" saying that I was "Missing key file" (sic) an error message. I clicked the "Fix Error" button. Then it proceeded to show me a "Thank you for choosing Unraid OS!" screen. I am now able to start the array and my containers are starting up. (Sorry for writing down all the details. I'm using this post to document what I did as a record for my future-self.)

Okay. It seems that the USB flash backup .zip was missing key files needed to boot unraid. For reference, I used the backup zip from Unraid Connect and the Unraid USB Creator tool. When I copied all the files manually from my previous-but-failing USB flash drive onto the new USB flash drive, now I am able to boot into my config. However, the registration page now says: I guess I have to figure out which files are the licence key files in the /config folder...

Okay, this is not good. I followed the steps for restoring my backup .zip file TO the new USB key, it behaves like a completely new installation. My hostname is reset to "Tower" instead of server03, and I can't access the webGUI at the IPv4 address shown on the console. Help!

Server crashed after about 2 days of stability. But here is the syslog. I believe the problems start around timestamp=Jul 21 01:58:59 (My new USB stick is still on the way, so I haven't changed the flash drive yet) syslog-10.10.10.20 - copy.log

Cool! I did not know this was an option. I will start looking into this (and also seeing what internal USB headers are still open on my server's motherboard).

Thanks. Repalcement USB flash drive is on the way (ordered when I first made this post). Normally, I'd just run out to the local store and grab any old stick, but Limeware only allows you to change sticks once per year (without contacting them) and I want a low-profile one so that it doesn't get physically broken when I'm moving my server around.

Would a failing flash drive explain the symptoms that I've been seeing? Symptoms: - All Docker containers and VMs no longer responsive - UnRAID webGUI inaccessible - Wireguard VPN is running and will accept new connections (weird...) - Network card responds to pings - Unable to SSH into the Unraid server from within local LAN - Console attached directly to Unraid server shows the login screen - Attempts to login with a valid username/password via the console show an error message like "bash-completion: input/output error"

OKay. Thanks, I've enabled the syslog. It hasn't crashed yet; but I've attached an updated diagnostics file. I'm specifically interested in this line, which leads me to suspect a USB drive failure. sda is the USB boot drive, and I've been seeing these errors in the log recently (maybe in the past 2-3 weeks) Jul 19 02:10:11 server03 kernel: critical medium error, dev sda, sector 16978124 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 2 server03-diagnostics-20240719-0529.zip

Unraid version 6.12.10 My server has been running well for a couple of years. In the last week, it has crashed daily, and can only be saved with a power-off/power-on. I think it may be the USB drive, but I wanted to check. Diagnositics are attached. Symptoms: - All Docker containers and VMs no longer responsive - UnRAID webGUI inaccessible - Wireguard VPN is running and will accept new connections (weird...) - Network card responds to pings - Unable to SSH into the Unraid server from within local LAN - Console attached directly to Unraid server shows the login screen - Attempts to login with a valid username/password via the console show an error message like "bash-completion: input/output error" - Once or twice: power cycling the computer, the BIOS says "no valid boot device found", but then after shutting down for a minute or so, and booting up, I can get into Unraid, and it seems normal for a while. Notes: - I have backed up my USB drive via Unraid Connect and downloaded a copy of that to my local PC server03-diagnostics-20240718-1736.zip

One of the smaller disks in my array started making cricket noises overnight and has now died. Unraid was showing it as having "read errors", but the data is still accessible on the array thanks to the emulation provided by the parity drive. I rebooted the unraid server and now the drive is not detected at all. Unraid reports that slot (disk4) as "not installed", but otherwise runs normally. I shutdown the unraid server, pulled the drive out and was about to plug it into a completely different PC, using a USB-to-SATA kit (so completely different cables). Before I plugged the USB-to-SATA adapter to the other PC, I powered up the drive on its own and could hear the clicking noise again, so I'm sure the drive is gone. I don't have a spare disk to put into that slot right now, so I'm going to have to go without it. However, I have plenty of free space on the other disks to store the data. I believe the next steps to do are as follows: 1) Move the data off the emulated [dead] disk4 using the Unbalance plug-in 2) Stop the array 3) Recreate a new configuration with disk4 removed from the array (ie one less disk) 4) Start the array 5) Rebuild parity Is this the best approach?

I have a number of disks, (2x14TB, 2x12TB and 2x10TB), and 4 SSDs (1x120GB, 1x240GB, 2x1TB). Right now, I have the 120GB and 240GB SSDs operating as separate, single disk caches, and the 2x1TB SSDs as a RAID1 cache. I notice that 3 of my 6 HDDs don't spin down ever. but the 3 that are mainly contain Plex media do. How can I configure my shares so that nearly all the disks spin down, to reduce power consumption? This is mostly a Plex machine, but it has a Linux VM running on. I have appdata, docker, domains folder set to Cache-Prefer; and my downloads, backups and system set to cache-Yes.