Running Unraid 7.01

This is an odd one. This system (Zaphod) is generally bulletproof when running containers 24/7 - Plex, Immich, etc. These containers read and write from remote SMB shares on another Unraid system (Ford) regularly.

 

Remote SMB shares are all mounted by Unassigned Devices.

 

Over the past couple of days I've been doing some long downloads using the JDownloader container, averaging about 150MBytes per second. The download path is to a different SMB share on the same (Ford) Unraid server mentioned above.

 

 

Yesterday afternoon I noticed that downloads had slowed way down, the JD webUI very sluggish and I could no longer open the Unraid WebUI on this Docker/App system. The login prompt on video out was completely unresponsive to keyboard input. I can't recall where/how (maybe reloading the JD webUI) I saw an Unraid Docker 403 error - which is documented to mean something's wrong with the docker image, including having taken up all the disk space.

Clicking the power button on the machine successfully started the shutdown process and the machine shut down normally. Restarting it put everything back to normal - docker containers all worked normally, webUI working and download speeds and JD's web UI back to normal and responsive. I didn't rebuild the docker image as there didn't appear to be any problem loading any of the containers.

 

Today I noticed again no Unraid webUI nor JD's webUI can be opened. Login prompt again unresponsive. Power button this time will not initiate shutdown. The system seems completely locked up, and I had to force it to shutdown with long-pressing the power button. Everything loads and responds normally after restart.

 

Can anyone help track down what might be going on here? Diags attached below.

 

The only other thing of note, is that JDownloader shows a "Disk Full" error on all remaining after the system is restarted. The destination path can't be accessed from within the running container at this time. The share itself is working fine from Unraid and can be browsed. Restarting the docker allows it to be accessed normally again.

 

zaphod-diagnostics-20250401-1536.zip

Edited April 1, 20251 yr by Espressomatic

Enable the syslog server and post that after it happens again.

This system is running a syslog server to capture the logs of my Edge Router system.   Now I've set up another system to capture this one's logs too.

 

I'll set up another heavy download and see what happens.

 

 

I'm at the state right now where the WebUI of the system won't load, but container WebUI are still functional. For example, JDownloader and QBittorrent. This seems to be the first situation I previously described.

 

QB is downloading and uploading right now, though at a much reduced speed compared to expected throughput.

 

Here's the syslog as it stands right now:

 

 

syslog-Zaphod.log

Edited April 3, 20251 yr by Espressomatic

I pressed the power button to shut down but it seemed to be stalling. The docker containers were still running, so I quit QB from its webUI and that seems to have allowed the system to finish shutting down.

 

Here's the syslog after shutdown:

 

 

syslog-Zaphod(1).log

Unfortunately, there's nothing relevant logged, could be an issue with a container, still, a couple of things you can try, go to Settings - Global share settings and set the Number of fuse File Descriptors to the max, and enable this:

 

https://docs.unraid.net/unraid-os/release-notes/7.0.0/#excessive-flash-drive-activity-slows-the-system-down

Besides trying to mitigate the problem, is there anything else I can do/install/enable to help track down the root cause?

 

This system runs for months on end without a hiccup when I'm not doing this kind of download/data moving. Incidentally, I'm running exclusively with ZFS pools on all my systems. This one in particular has a single 500GB NVME at the moment. The destination remote share is 6x 14TB spinners RAIDZ2

 

Two interesting data points:

 

When the WebUI hangs as described, or the whole system locks up, I can hear the system fans easily from 5 to 6 feet away. When the machine is running normally, I can't hear the fans at all from even 2 feet away.

 

Second, there's a small issue of remote shares not being mounted before dockers spin up. If I start or restart or boot the system, some of the containers (JD and QB in particular) can access any of the remote shares. The containers themselves need to be restarted again. This doesn't seem to affect Plex. Shares are specified with slave option in the containers. I've remedied this by putting a 10 second delay on the first container to start, giving all the shares plenty of time to mount before any other containers start up. It's just not something I've ever needed to do in the past.

 

 

 

Edited April 3, 20251 yr by Espressomatic

I just started another long multi-TB download session with an Array-based share on a different server as the destination (opposed to ZFS volume) - will update as it progresses.

 

27TB volume and 10TB download for testing purposes

 

 

It's been over 24 hours and the system is still up, system running normally and downloading steadily, albeit much more slowly overall.

 

The big difference between downloading to RAIDZ2 and Array-based XFS volumes is sustained throughput on the network (high saturation vs. low saturation with sporadic bursts)

 

Same group of files, anywhere from 2-10GB each - 4 concurrent downloads with up to 3 connections per DL

 

Downloading to RAIDZ2 share over SMB (Unraid to Unraid server)

 

 

 

JD finishes a file and starts another very quickly and can sustain a high rate on any/all files, so network traffic between the systems is always high.

 

Downloading to Array-based share over SMB (Unraid to Unraid server)

 

 

  

 

JD sits on a file long after its finished, before starting a new file - pretty sure this is the time it's taking to close out the file. Download rate per file is much more erratic than in other example, so sustained network traffic is low.

 

 

It's expected that writing to an Array share is slower, as it's only writing to a single disk at a time. Now I just need to figure out why the sustained high throughput of the first example is eventually causing the system running JDownloader to hang.

 

 

Edited April 7, 20251 yr by Espressomatic

  

On 4/3/2025 at 5:50 AM, JorgeB said:

 a couple of things you can try, go to Settings - Global share settings and set the Number of fuse File Descriptors to the max, and enable this:

 

https://docs.unraid.net/unraid-os/release-notes/7.0.0/#excessive-flash-drive-activity-slows-the-system-down

 

I did the above a few days ago and testing since has been with those changes in place.

 

Issue finally replicated sometime this afternoon. I can hear the fans on the Zaphod system are running faster, webUI won't load, JDownloader docker UI very slow (mostly unusable), download speed cut down to 10MB/s. Destination server (Ford) is 100% up and running fine, despite errors showing up in Zaphod's syslog saying it's timing out.

 

I can SSH into the system as well as access it with a keyboard on HDMI output (had to replug the keyboard cable a couple of times to get it to respond and it was VERY slow to log in)

 

From the JDownloader UI I managed to "exit" the running docker app and very quickly the fans on the system died down and I'm able to now open the Unraid WebUI. Sounds like the container may be eating up a lot of CPU - but is it the root cause of the issue or might this be an issue with dockerman triggered by this particular container?.

 

Diagnostics captured while system was running in the degraded state.

 

 

zaphod-diagnostics-20250408-1758.zip

 

syslog-Zaphod(2).log

 

 

Edited April 8, 20251 yr by Espressomatic

I'm afraid that I don't see anything relevant logged.

6 hours ago, JorgeB said:

I'm afraid that I don't see anything relevant logged.

 

Completely down this morning, can't connect to the system at all, including any containers.

 

I'll be trying to replicate the issue on one of the other systems doing local downloads, but I'm lacking some storage to flip the rolls of the machines around and use this crashing system as network storage for the other. I'll also try using something other than JD to do download and SMB simultaneous testing and then try to figure out how to do network-only testing on SMB without WAN traffic.

 

syslog-Zaphod(3).log

Edited April 9, 20251 yr by Espressomatic