High iowait, UI crash, can't get diagnostic logs, lsof never returns HELP!

[drwatson]

I have an UnRaid server running for a few weeks now. Every once in a while the UI becomes unavailable (500 Internal Server Error). 

I can ssh into the machine though and samba still seems to be working.

I tried to gather some logs using the diagnostics command and it never returns.

I ran top and then pressed 1 to get the CPUs split out. 

Wait time (wa) on CPU1 is sitting at ~95. All other wa numbers for other CPUs is 0.

This is telling me that it has to do with IO somehow. I ran lsof and it never returns either.

I'd like to install iostat but I don't know how to do that in this flavor of Linux (tried yum, apt-get, etc). Any posts I check tell me to install using nerd tools. But I don't have access to a UI to be able to do that since the UI is crashed.

I can reboot the system, but the crash will occur again so while it's in this state, I'd like to collect as much information as I can.

I think this may have to do with an external USB drive that I have connected to the system. It seems to crash most often when I perform a backup to the external drive. I got an email from CloudBerry Backup that the last backup "failed". No other explanation. I looked on the system (the UI was still working at that point) and noticed that the external USB drive was not longer mounted. Could this be a clue?

There seems to be a lot of posts about iowait times, but none of them seem to fit with the circumstances I am seeing here.

Can anyone suggest what else I should look for? I'm stuck.

 

syslog