VM Claiming unrealistic VDISK performance as 10,803/4812 MB/s

[jaybee]

Hey guys,

 

I want to setup an nvme drive as pass though in my VM to get close to bare metal performance for gaming. Before I do this, I took some benchmarks of the existing vdisk performance before I did it for comparison. I noticed that the numbers for usual benchmarking tools are claiming very unrealistic performance stating numbers that are simply too fast and impossible. I gather that this is due to the way vdisk works with caching? Is there a way to turn this off and test more easily to get a baseline? I tried i/o zone but could not understand how to use it easily as it is cmd based and does not give easy to read results. I tried i/o meter and the control panel for the software looks ancient and again not sure how to use. 

 

The results I got on a vdisk running on a sata cache disk which would normally have read speed of approx 500MB/s and write of around 350 MB/s are below: 

 

Crystal disk mark:

Sequential - Read = 10,803MB/s, Write = 4,812MB/s

4KiB Q8T8 - Read = 192MB/s, Write = 112MB/s

4KiB Q32T1 - Read = 396MB/s, Write = 94MB/s

4KiB Q1T1 - Read = 46MB/s, Write = 36MB/s

 

ASSSD:

Sequential - Read = 6,582MB/s, Write = 2,142MB/s

4K - Read = 47MB/s, Write = 33MB/s

4K 64 Thrd - Read = 292MB/s, Write = 72MB/s

Acc. Time - Read = 0.193ms, Write = 0.106ms 

Edited March 28, 2021 by jaybee

[MarKol4]

The same here. I have an array purely on mediocre HDD drives. I'm doing some testing now and for that purpose there is also no SSD Cache Pool in my unRAID.

 

There is VM with a virtual disk (raw, VirtIO) attached. I have prepared virtual disk by running SDelete -c in VM in order to get virtual disk fully initialized (just to make sure that entire space of the disk was written with some data).

 

After that (and restart of the unRAID server) I've tested a virtual disk performance by running CrystalDiskMark in the VM. These are my example results:

 

------------------------------------------------------------------------------
CrystalDiskMark 8.0.1 x64
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
  SEQ    1MiB (Q=  8, T= 1):   791.518 MB/s [    754.9 IOPS] < 10448.04 us>
  SEQ    1MiB (Q=  1, T= 1):   885.947 MB/s [    844.9 IOPS] <  1180.72 us>
  RND    4KiB (Q= 32, T= 1):    18.143 MB/s [   4429.4 IOPS] <  7121.53 us>
  RND    4KiB (Q=  1, T= 1):    19.230 MB/s [   4694.8 IOPS] <   212.33 us>

[Write]
  SEQ    1MiB (Q=  8, T= 1):   221.328 MB/s [    211.1 IOPS] < 37301.93 us>
  SEQ    1MiB (Q=  1, T= 1):   195.541 MB/s [    186.5 IOPS] <  5350.16 us>
  RND    4KiB (Q= 32, T= 1):   168.338 MB/s [  41098.1 IOPS] <   763.88 us>
  RND    4KiB (Q=  1, T= 1):    14.531 MB/s [   3547.6 IOPS] <   280.21 us>

Profile: Default
   Test: 16 GiB (x5) [D: 0% (0/80GiB)]
   Mode: [Admin]
   Time: Measure 10 sec / Interval 0 sec
   Date: 2021/06/05 20:14:42
     OS: Windows 10 Professional [10.0 Build 19041] (x64)

 

Write results are way beyond that HDD hardware can do. Especially random 4K writes.

 

I can see no other explanation than that an unRAID is doing a lot of write-back caching when VM is writing to a virtual disk.

Is it safe for our data when unRAID is doing that kind caching? Any machine crash or a power outage can corrupt a lot of data if write-back caching is in use. For that purpose we have a BBU on hardware raid controllers which makes write-back caching safe.

 

Lets imagine that we have a database running in VM and it stores data files on a virtual disk. Should it not be the case that if a database engine commits a transaction and user code is informed about successful commit all the data is already persisted and safe? Is it even possible when unRAID seems to be doing write-back caching?

 

Can you explain what is going on here and how much safe are our data already written to virtual disk?

[jaybee]

On 6/5/2021 at 8:05 PM, MarKol4 said:

The same here. I have an array purely on mediocre HDD drives. I'm doing some testing now and for that purpose there is also no SSD Cache Pool in my unRAID.

 

There is VM with a virtual disk (raw, VirtIO) attached. I have prepared virtual disk by running SDelete -c in VM in order to get virtual disk fully initialized (just to make sure that entire space of the disk was written with some data).

 

After that (and restart of the unRAID server) I've tested a virtual disk performance by running CrystalDiskMark in the VM. These are my example results:

 

------------------------------------------------------------------------------
CrystalDiskMark 8.0.1 x64
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
  SEQ    1MiB (Q=  8, T= 1):   791.518 MB/s [    754.9 IOPS] < 10448.04 us>
  SEQ    1MiB (Q=  1, T= 1):   885.947 MB/s [    844.9 IOPS] <  1180.72 us>
  RND    4KiB (Q= 32, T= 1):    18.143 MB/s [   4429.4 IOPS] <  7121.53 us>
  RND    4KiB (Q=  1, T= 1):    19.230 MB/s [   4694.8 IOPS] <   212.33 us>

[Write]
  SEQ    1MiB (Q=  8, T= 1):   221.328 MB/s [    211.1 IOPS] < 37301.93 us>
  SEQ    1MiB (Q=  1, T= 1):   195.541 MB/s [    186.5 IOPS] <  5350.16 us>
  RND    4KiB (Q= 32, T= 1):   168.338 MB/s [  41098.1 IOPS] <   763.88 us>
  RND    4KiB (Q=  1, T= 1):    14.531 MB/s [   3547.6 IOPS] <   280.21 us>

Profile: Default
   Test: 16 GiB (x5) [D: 0% (0/80GiB)]
   Mode: [Admin]
   Time: Measure 10 sec / Interval 0 sec
   Date: 2021/06/05 20:14:42
     OS: Windows 10 Professional [10.0 Build 19041] (x64)

 

Write results are way beyond that HDD hardware can do. Especially random 4K writes.

 

I can see no other explanation than that an unRAID is doing a lot of write-back caching when VM is writing to a virtual disk.

Is it safe for our data when unRAID is doing that kind caching? Any machine crash or a power outage can corrupt a lot of data if write-back caching is in use. For that purpose we have a BBU on hardware raid controllers which makes write-back caching safe.

 

Lets imagine that we have a database running in VM and it stores data files on a virtual disk. Should it not be the case that if a database engine commits a transaction and user code is informed about successful commit all the data is already persisted and safe? Is it even possible when unRAID seems to be doing write-back caching?

 

Can you explain what is going on here and how much safe are our data already written to virtual disk?

 

 

 

Did you ever get the bottom of this? Sometimes testing with larger file sisze in crystal disk mark like 4gb gives results showing sequential write of around only 1000MB/s which may be more realistic given the bare metal performance should be around 3000MB/s. It definitely seems like a caching issue which makes it hard to truly compare passed through nvme performance vs vdisk. Vdisk has the nice portability aspect of it and ease of backup, but nvme has the advantage of supposedly supporting proper trim and wear levelling stuff when passed through to windows properly I have read.