VM becomes unresponsive while preclear running

[SimonHampel]

Long story - sorry, lots of context required. I've been using Unraid for many years now with a 20 drive 100TB array working as my main file server for backups. The CPU in that machine has no virtualisation support and no capacity for cache drives, so I have not used VMs (or Dockers) on that server.

 

I've recently built a second Unraid server (glad I bought that dual license back in the day!!) to replace my ageing media server that was running Sage TV / Plex / Sonarr / SABnzbd

 

I've successfully migrated Plex, Sonarr and SABnzbd to run in dockers - that's working well.

 

However, due to a lack of driver support for my old Hauppauge TV tuners, I haven't gone the docker path for SageTV but instead have converted my old server to a VM. This also allowed me to continue running my old SageTV v7 setup without any changes. I have the tuners passed through to the VM so they can use the Windows drivers.

 

So my SageTV vdisk is contained on a dedicated SSD (not part of the cache pool) mounted via Unassigned Devices and uses the VirtIO bus.

 

I have two HDDs for recorded TV which aren't in the array - both of which are passed through to the VM and are formatted using NTFS. They both use the VirtIO bus.

 

I'm not sure if this is the best mechanism for passing dedicated drives through - but from what I've read, unless you have a dedicated controller to pass through, then VirtIO is the best choice for dedicated drives?

 

Anyway - the server has been working well, I've migrated all of my media across and I have three SSDs in a cache pool.

 

However, in the last few days I've been having difficulty accessing SageTV via any of my clients (HD200 extender / Placeshifter / etc). 

 

I can start the SageTV UI and navigate the menus. I can even watch live TV without issues. But as soon as I try to watch a recorded show, it just sits there waiting (spinning circle in SageTV).

 

When accessing the VM via Teamviewer, I can see there is nothing really using any CPU and the machine is generally responsive, but as soon as I try to access either of the recording drives via Windows Explorer (just to get a directory listing!), the machine grinds to a halt with Resource Monitor showing 100% disk utilisation on the recording drive and a growing disk queue length.

 

The vdisk boot drive is fine, so the OS is still running - it's the recording drives which cause the issue.

 

Now after a bit of head scratching (because it has been working fine up until the last couple of days), I've remembered that I'm also running a preclear on a new 8TB drive which is housed in a portable enclosure and connected via USB-C.

 

The preclear is running at around 180MB/s or more (wow - so fast compared to my other Unraid box!!) but is still going to take days to complete.

 

I'm wondering whether there is some kind of bus saturation thing happening here while the preclear is running?

 

I'm disinclined to stop the preclear right now to check whether that solves the issue - so I'll have to wait a few days to confirm. But either way, this is a problem if running a preclear stops my VM from recording any shows!

 

I'm wondering whether it is the VirtIO bus that I'm using for the pass through recording drives which is the issue? Is there a better setup I could use?

 

Let me know what info you need to help me diagnose the problem here - thanks!

Edited January 31, 2021 by SimonHampel

[SimonHampel]

I was testing a few more things and can now make the following observations:

 

1. trying to access D:\ in windows explorer (recording drive #1) results in 100% disk usage on that drive with a growing queue length showing in resource monitor

2. trying to access E:\ in windows explorer (recording drive #2) results in 100% disk usage for a short period before erroring out with the message: "E:\ is not accessible - The request could not be performed because of an I/O device error"

 

I'm not sure why the result would be different between these drives.

 

Actual read/write speed seems to be pretty much zero, despite 100% activity

[SimonHampel]

Preclear has finished and I can confirm that the SageTV VM is now performing as expected.

 

I was able to record 4 HD shows at once while watching another via my extender - no issues at all.

 

So why is the preclear causing the drives used by the VM to become unresponsive?

Edited February 1, 2021 by SimonHampel

[ChatNoir]

Maybe a bottleneck in your system ?

[SimonHampel]

Yes, I think it definitely is a bottleneck - I think perhaps the preclear is saturating the bus or something - but I'm not sure what to do about it.

 

Is it the VirtIO driver?

 

Is it the controller?

 

Is it the USB-C hard drive?

[SimonHampel]

Okay - the plot thickens.

 

I'm running an rsync from this server to my file server backing up some media files - running at around 40MB/s or so (turbo write turned on for the destination)

 

While this is running - I'm getting exactly the same behaviour - the drives in the SageTV VM are timing out when I try to access them.

 

D drive just sits there with ever-growing disk queue length, while E drive errors out "The request could not be performed because of an I/O device error"

 

So this is not specifically related to prelcear like I first thought 

 

Any suggestions?

[jonp]

As of right now, using unassigned devices or pre-clear aren't things that we at Unraid developed ourselves, so you'd have to look to those plugin authors for any support on issues related to their usage.  We do have plans with 6.9's upcoming release to support multiple pools natively, which means you wouldn't need to use the Unassigned Devices plugin anymore for those download disks.  In fact, you could just put both into a btrfs raid 0 if you wanted and then create a vdisk on them that utilizes the combined capacity of the two devices.

 

If you could replicate this behavior where the OS is configured using standard tools, we could have something to investigate.  All of that being said, this is likely something unique to your hardware / configuration as I know there are plenty of users out there leveraging preclear and Unassigned Devices without issue.  Another thing that would help is if you attached your system diagnostics to this forum thread so we all could see the OS configuration you have.