Hello all,

 

This morning my son was watching a movie when it started buffering.  When I looked at my unraid dashboard all the cores were pegged at 100%.  I tried rebooting and even turned off some VMs and dockers, to no avail.

 

The reboot came up unclean so it has started a parity check, which normally only causes some slowness with disk transfer but the containers and VMs usually work fine.  This time the VMs are not completing boot, they delay at the bios splash screen for around a minute or so, and then eventually time out waiting for the vdisk.

 

From the console of one VM:

 

[ TIME ] Timed out waiting for device dev-disk-by\x2duuid-3657111a\x2d1f89\x2d4906\x2dabf7\x2daba104ab1ca1.device

[DEPEND] Dependency failed for /boot

[DEPEND] Dependency failed for Unattended Upgrades Shutdown

[DEPEND] Dependency failed for /boot/efi

[DEPEND] Dependency failed for Local File Systems

[DEPEND] Dependency failed for File System Check on /dev/disk/by-uuid/3657111a-1f89-4906-abf7-aba104ab1ca1.

You are in emergency mode.  After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or "exit" to boot into default mode.

Give root password for maintenance

 

The other VM gets to the login prompt, but trying to reach the web app I built on it times out.  I tried logging in but it hangs without asking for the password.

 

Both VMs are Ubuntu 20.04

 

As for my docker containers, I figured the problem would be with flood, but I have stopped it and the system is still hanging all over.  Flood never gets to loading the UI, Sonarr and Radarr UI's also wont come up, with what looks like timeout errors in the log.

 

I haven't added anything new to unraid in weeks, the only change was replacing a SAS breakout cable and re-adding a 4TB drive that was having issues (because of the old cable).

 

Oddly enough, after the reboot, Plex (hosted on a different computer, but reading the movie files from the array over NFS) seems to have no problem, movies are streaming ok despite the system load.

 

If anyone can suggest where to look I'd really appreciate it!

 

 

 

 

unraid-diagnostics-20220123-1641.zip

for no reason I can tell, things look like they are coming back to normal.  Restarting docker containers is only taking seconds instead of minutes again, the vm's are booting and getting to a login prompt in reasonable time, etc.  Everything still seems slower than it used to be though and I am still clueless about what the cause could be.

I've attached a new diagnostics.  I ended up migrating my web app vm to my pve cluster, which I probably should have done anyways.  The other VM seems to be ok, it is only a quorum device for the pve cluster, and it is reporting as a vote once again.  Almost everything seems to be back to normal, system load is back to being 30-50% instead of 80-100%.  It's still in the middle of a full parity-check but it seems to be going at normal speeds.  My containers all seem to be working again, although I do see flood reporting that every 10 minutes or so transfers just stop for like 5 seconds, then it takes 3-5 seconds to get back to full speed.

 

unraid-diagnostics-20220124-0145.zip