I've been having some stability issues with my Unraid setup lately. I actually upgraded the server recently which I assumed would probably resolve it, but its not, so I'm keen to get to the bottom of it, as I replaced pretty much everything in the process.

 

I use the server purely to run containers - all my storage is on an external Synology, accessed via NFS shared using Unassigned Devices. I run a pair of SSDs (replaced during the migration) in a RAID1 cache pool using UD, and I have a USB drive running an array, purely to allow things like Docker to actually run.

 

The obvious symptoms I get are:

 

• The Dashboard stops show certain data - notably Processor and Memory info is blank:

 

 

• The Main screen is blank:

 

 

• I'm unable to launch the command line terminal - when I press the button a window appears but remains blank.
• Other things accessed via the UI don't necesararily work. For instance whilst I can generate a Diagnostic file, it won't download within the UI.

 

Despite this, containers continue to run and I'm able to SSH in. Generally I reboot it via SSH and all is good for a while.

 

I'm getting this intermittently - maybe once a week? So tricky to pin down. I'm wondering if this could relate to the USB drive running the array, as that and the boot USB are the only items that carried over from the previous server. 

 

I've attahced diags so hoping that may provide some clues. I won't reboot the server in case there's anything I can try, as once I reboot it'll probably be fine for a while.

 

server-diagnostics-20230224-1626.zip

I'm continuing to get this issue intermittently. Yesterday I rebooted due to the server going into this partial failure state, and this morning I'm getting the same issue. Although the server is stable other than this, I'm clearly anxious that there's an underlying fault which will only get worse.

 

Given that I was having similar issues last year, my suspicion is that maybe the USB drive running as the array is failing - the two cache drives have both been replaced recently along with the server, so the USB drives used to boot Unraid and ru the array are the only elements that remain.

 

Any ideas on what I can try?

server-diagnostics-20230502-1621.zip

You are getting a lot of nginx errors, could be a plugin/app, you could try running in safe mode to see if it helps.

Thanks. I run nginx proxy manager as a docker container - do the errors relate to that, or something else running natively on the host? I have a bunch of plug-ins running though nothing that isn't pretty popular.

 

What are the implications of running in Safe Mode? Happy to give it a go, though as its quite an intermittent problem I'd likely need to leave it for a week or so to have a sense for whether it helped. So whether that's viable mainly depends on what's disabled in Safe Mode.

The GUI also uses nginx, so nginx errors can affect the GUI

@JorgeB

No significant progress on this issue - I did cut back on plug-ins, but today I've had back to back system instability, even after a reboot. Checking the logs at the time these errors stood out as potentially problematic, and not something I'd seen before (hence the post):

 

May 15 01:02:44 Server kernel: traps: lsof[6947] general protection fault ip:1486821704ee sp:a468894e97de6910 error:0 in libc-2.36.so[148682158000+16b000]
May 15 11:39:30 Server kernel: nvme0: Admin Cmd(0x2), I/O Error (sct 0x0 / sc 0x13) 
May 15 11:39:30 Server kernel: nvme0: Admin Cmd(0x2), I/O Error (sct 0x0 / sc 0x13) 

 

After the reboot I see these errors:

 

May 15 12:12:17 Server kernel: serial 0000:00:16.3: Couldn't register serial port 30a0, irq 17, type 0, error -28
May 15 12:13:20 Server kernel: I/O error, dev sda, sector 4251598 op 0x0:(READ) flags 0x84700 phys_seg 2 prio class 0
May 15 12:15:32 Server root: error: /webGui/include/InitCharts.php: wrong csrf_token

 

I've since enabled a remote syslog server which will hopefully help. Both cache drives are new, but is it possible they're causing the problem?

server-diagnostics-20230515-1249.zip

2 minutes ago, Nuuki said:

May 15 12:13:20 Server kernel: I/O error, dev sda, sector 4251598 op 0x0:(READ) flags 0x84700 phys_seg 2 prio class 0

This one would be of concern, since it's an error with the flash drive.

Ah right - so should I just pickup a new drive? Easy enough fix if that's the issue, and clearly want to address it before it gets any worse.

Not sure that is the problem, but it is a problem, so worth trying.

I've swapped it out so I'll see if that makes any difference. As you say even if its not the root cause its clearly a problem.