Hi all.

 

I have been doing a LOT of reading about unRAID inside of ESXi.  I have a Lenovo Thinkstation D20 (2x Xeon E5530, 20GB RAM, VT-d supported) that I am going to make my ESXi box...transplant the innards to my RPC-4220.

 

Before doing any of this, I put a couple of 160gb 2.5" drives on the SATA/SAS RAID controller for unRAID drives via passthrough, and, 1 160gb on the non-raid controller to be used for ESXi/datastores so I could get my feet wet.  I gave the unRAID VM 2gb of mem and 1 CPU x 4 cores (unRAID b5RC11).

 

Long story short...I was getting PSODs pretty regularly.  I then realized that the onboard SATA/SAS controller has the same exact chip as the MV8s so I applied the MV8 hack.  I'll watch things to see if the PSODs return, but I have run into an issue...

 

I am streaming Avatar BD to my openelec box and it plays flawlessly.  However, if I initaite a parity check, the movie begins to buffer.  Has anyone esle experienced this?  I tested the same on my baremetal unRAID (core i3, 8gb ram) and do not have the issue.

 

Maybe the issue is that I have only one data drive (which obviously houses Avatar) and the parity check is pounding the crap out of it?

 

TIA for the help!!

 

John

OK...I'm pretty certain I answered my own question...

 

Looking at the "Stats" --> "System Stats" page in SimpleFeatures:  when streaming just the Avatar BD, the Storage chart shows consistent I/O between 2.5MB/s and 5.1MB/s (as expected).  As soon as I initiate a parity check, Disk I/O jumps to ~145MB/s.

 

So.

 

1.  Is this expected?

2.  If so, only because I have a single data disk?

 

I guess I am confused why I would have never seen this on my baremetal box.  Doesn't parity check move right down the line of disks?  If so, I think at least once I would have seen this before.

 

John

Another possiblity that I've heard is that you are better off in ESXi using a single core with a VM then multiple.  The switching required for multi core VMs can actually slow them down from using a single core.  So if that was true in your case you might switch to a single core for your unRAID VM.  Then install any plugins that were on unRAID to a separate linux VM from unRAID if you need them to transcode video for instance.

Another possiblity that I've heard is that you are better off in ESXi using a single core with a VM then multiple.  The switching required for multi core VMs can actually slow them down from using a single core.  So if that was true in your case you might switch to a single core for your unRAID VM.  Then install any plugins that were on unRAID to a separate linux VM from unRAID if you need them to transcode video for instance.

 

Reconfigured to be 1 CPU with 1 core...no help.  I'm pretty certain that has to be the fact that there is only one data drive and it is getting hit hard.  In fact, I am running a parity check right now and watching the data drive temp rise (10 degrees in the last 5 minutes).

While a parity check is not very CPU intensive, it is VERY disk I/O intensive, as all disks are active for the entire duration of the check.    ANY activity besides the parity check will severely "thrash" the disk(s) involved in the other activity -- generally only one disk;  but even that will slow the entire parity check down to the effective speed of the thrashing disk.

 

A stand-alone UnRAID box can sustain this thrashing without notable impact to the streaming (it will have a much more notable impact on parity check time) thanks to the speed of modern disks and the memory it has available for buffering.    I suspect that a virtualized implementation simply has enough hypervisor overhead that it can't keep up with this -- even with pass-through disks (which still require interaction through the hypervisor.

 

The number of data drives is not relevant => a parity check runs the parity disk and all data drives (whether 1 or 20) as fast as it can until the check is done.    In fact, the number of drives makes little difference in the parity check time -- what's FAR more important is whether or not all the drives are the same.  [A mixed set of sizes will notably slow down the check.]

 

 

While a parity check is not very CPU intensive, it is VERY disk I/O intensive, as all disks are active for the entire duration of the check.    ANY activity besides the parity check will severely "thrash" the disk(s) involved in the other activity -- generally only one disk;  but even that will slow the entire parity check down to the effective speed of the thrashing disk.

 

A stand-alone UnRAID box can sustain this thrashing without notable impact to the streaming (it will have a much more notable impact on parity check time) thanks to the speed of modern disks and the memory it has available for buffering.    I suspect that a virtualized implementation simply has enough hypervisor overhead that it can't keep up with this -- even with pass-through disks (which still require interaction through the hypervisor.

 

The number of data drives is not relevant => a parity check runs the parity disk and all data drives (whether 1 or 20) as fast as it can until the check is done.    In fact, the number of drives makes little difference in the parity check time -- what's FAR more important is whether or not all the drives are the same.  [A mixed set of sizes will notably slow down the check.]

 

Thank you for that explanation Gary...it really answers a lot of the questions that I had.

 

Honestly, I'm surprised that more people have not brought this up as an issue.  If disk i/o is that much worse under ESXi than baremetal, I imagine that other applications (other than movie streaming) would also be severley impacted.

 

I know you say that I will not make a difference but I am going to add 2 or 3 more drives to the array and see if that helps at all.  I'm still hoping that disk i/o on the drive with the movie will be reduced enough to help.

 

John

So, I added a second disk (60gb 5400 RMP 2.5") to my ESXi unRAID array and the overall disk i/o dropped to about 127MB/s while running a parity check.  I was able to stream Avatar wihtout issue. 

 

Out of curiosity I checked the same graph on my baremetal unRAID.  That one has 6 drives...all SATA II (3GB/s).  Running a parity check  produced an overall disk i/o of ~750MB/s...well below the potential of 1.8GB/s (6 * 300MB/s).  Obviously, this leaves plenty of overhead for doing other things...like movie streaming...and explains why I never saw the issue before.

 

John

I've never had a problem with my esxi build and a parity check / streaming (within reason of course!)

 

I suspect you were just a victim of your own success with the small  number of disks - those figures you were seeing must have been close to 100% of the disk performance - so any other access would be impacted.

 

The more disks the harder your system is working (from hardware up to unraid) which likely gives you a bit more leeway. I would doubt anyone has simply added data disks and seen a parity check speed up (note added..not replaced for faster disks!) - best case it would stay the same speed but in theory it should slow down.

 

The exception to this would be if you had, say, a system with 1x 3TB parity, 1x 3TB data and 5x 2TB data. All your disks would be working until you hit 2TB worth of 'parity checks' at which point access would drop off to only the parity and single 3TB data drive (no more data on the 2TB's to read so they would become idle). You might see similar issues that you have now at that point if what you're streaming happens to come from that single 3TB disk.

 

It's not something I've encountered but I couldn't guarantee that any streaming I've done during a parity check would implicitly tickle that scenario. And I suspect if your system is generally 'in use' for other things constantly (cache_dirs, other plugins etc etc) the overhead of the rest of the system would likely slow things down enough to avoid any serious issue.