Hi all, I'm new to unraid and i've just finished my first setup and things seem to be looking good so far, apart from one thing, and that is random IO read and write performance in my Windows VM.

 

Here is my setup in short;

AMD Ryzen 7950X3D

32GB DDR6000

Nvidia RTX4060

4 x 12TB Toshiba HDD as unraid array (xfs)

1 x 1TB Samsung 970 pro as cache pool (btrfs)

3 x 1TB Samsung 970 evo plus windows pool (btrfs raid 0)

 

Cores 0:3 un-isolated.

Cores 4:5 + 2GB ram assigned to HomeAssistant VM.

Cores 6:15 + 20GB ram + GPU passthrough assigned to Windows VM.

 

All in all, compute performance is upwards of 80% of what windows would perform on baremetal, and it's a huge win (3 to 4x) in performance given my old setup. Most notably, the dockers and home assistant VM are way more responsive than when they were running on raspberry pi's and a synology NAS, while the system is consuming less energy!

I have chosen to partition only 2TB of the total 3TB to leave room for OP and wear leveling. However the random IO performance of the NVME pool is lackluster, whereas the sequential performance is somewhat in line with what I would expect, see attached screenshot:


I don't have exact numbers on the drives performance on baremetal, but from guru3d, for a single disk;

 

As you can see, the sequential read and write improved by (resp.) ~200% and 150%, what you would expect from a raid setup, but the random read and write tanked! It doesn't seem like there is a problem compute or memory wise, since even during these benchmarks the loads are <10%.

Does anyone have a clue or a hint on how I can improve this performance? Since it noticably impacts my work (code compile/linting etc.).

 

Also don't worry about the safety of Windows VM being in raid 0, the VM is being backed up regularly to the array, on the synology NAS and off-site.

Edited January 7, 20242 yr by un1ty

Ps. I'm aware I should format my 1 disk NVME cache to xfs as well, but that shouldn't impact the random IO performance of the "windows" share. 
What I'm really asking is should I use a different file system for the windows share, or should I directly parse the disks to the OS as a raid 0 to gain the expected benefits?

Edited January 6, 20242 yr by un1ty

On 1/6/2024 at 7:03 PM, un1ty said:

It doesn't seem like there is a problem compute or memory wise, since even during these benchmarks the loads are <10%.

I have to retract this statement, the load wasn't visible due to the setting of the timescale of the graph. I do see a 100% loads on the number of threads specified in the benchmarks, so it could very well be the bottleneck. I have performed tests in several different CPU configurations; first or second CCD only, core isolation on/off, more cores towards VM or unraid etc. And the performance change of this was within 10% (so similar changes were visible from run to run). So this doesn't seem to be the solution.

I have changed from btrfs to zfs and did see a bump up in sequential speeds, but practically none in the random speeds. Moreover, even when I parsed through the drives as a raid directly to the VM, I saw a negligible raise in performance.

I've also tried to compare between 1440fx and Q35.. 

Haven't gone through extensive testing yet, since sadly 3/4 of my brand new harddrives failed, with luckily all of the VM's and dockers stored on SSD's and private photos/videos backup to a NAS and cloud.

 

But also this seems to provide a minimum performance improvement of around 10%.

Sound like you are not pass through NVMe to VM, for better performance, you should try pass through NVMe, dedicated for single VM.

 

29 minutes ago, un1ty said:

3/4 of my brand new harddrives failed

Toshiba, which model ?

Edited January 10, 20242 yr by Vr2Io

6 minutes ago, Vr2Io said:

Toshiba, which model ?

Toshiba MG07 series, one of the only non-SMR models available at 12TB (product page) for data centre/nas usage, and typically known for it's reliability. I think the post order service had been rough handling the package, even though the drives are known for being more audible, to me it felt they were too audible to make sense even though they made 'normal' sounds.

 

10 minutes ago, Vr2Io said:

Sound like you are not pass through NVMe to VM, for better performance, you should try pass through NVMe. Dedicated for single VM purpose.

This is somewhat true now, however I did test with direct passthrough and it didn't give any performance benefits, I wanted a single 'storage space' to be passed through to the VM for higher performance across all 3 drives and  underlying 2 partitions that I was going to use. However I did a test with a passed through single drive and got similar results; sequential was somewhat equal, but random was aberrant. 

However, I think I have found the solution! and this should be probably be a template change for the Windows 11 VM (with newer CPU's?).

Before I used the default CPU 'passthrough' mode, but when I ran the same test again with CPU 'emulated (QEMU64)' mode this was the result:
 

 

This change is massive, and noticable even from just opening windows explorer and browsing through directories even though the random IO is still sub-par.

15 minutes ago, un1ty said:

Toshiba MG07 series, one of the only non-SMR models available at 12TB (product page) for data centre/nas usage, and typically known for it's reliability. I think the post order service had been rough handling the package, even though the drives are known for being more audible, to me it felt they were too audible to make sense even though they made 'normal' sounds.

You mean they are not actually failure, but generate abnormal sound ?

 

I use lot of Toshiba 6TB disk, but next 8TB - 18TB were WD, because they are cheapest.

Edited January 10, 20242 yr by Vr2Io

16 minutes ago, Vr2Io said:

You mean they are not actually failure, but generate abnormal sound ?

 

I use lot of Toshiba 6TB disk, but next 8TB - 18TB were WD, because they are cheapest.

The ones I was using were 12TB HDD's (9 internal disks) from toshiba, they were making a much louder normal read/write noise (clicking/clacking) from the head assembly, compared to my old 'shucked' WD red plus which I intended to replace, and those were also producing chirping sounds as if they were spinning down when they shouldn't while not performing any better. Which resulted into complete failure, at first it was only the first drive, but eventually the second drive failed as well.

 

TL:DR;

They were louder than need be, they (completely) failed without warning, they wouldn't even show up in bios eventually.

Here is the unformatted report of the failure on drive 1:
 

Quote

text error warn system array login Jan 9 00:15:03 un1ty kernel: sd 8:0:0:0: [sdc] 23437770752 512-byte logical blocks: (12.0 TB/10.9 TiB) Jan 9 00:15:03 un1ty kernel: sd 8:0:0:0: [sdc] 4096-byte physical blocks Jan 9 00:15:03 un1ty kernel: sd 8:0:0:0: [sdc] Write Protect is off Jan 9 00:15:03 un1ty kernel: sd 8:0:0:0: [sdc] Mode Sense: 00 3a 00 00 Jan 9 00:15:03 un1ty kernel: sd 8:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jan 9 00:15:03 un1ty kernel: sd 8:0:0:0: [sdc] Preferred minimum I/O size 4096 bytes Jan 9 00:15:03 un1ty kernel: sdc: sdc1 Jan 9 00:15:03 un1ty kernel: sd 8:0:0:0: [sdc] Attached SCSI disk Jan 9 00:15:29 un1ty emhttpd: TOSHIBA_MG07ACA12TE_91M0A1U6F95G (sdc) 512 23437770752 Jan 9 00:15:29 un1ty kernel: mdcmd (2): import 1 sdc 64 11718885324 0 TOSHIBA_MG07ACA12TE_91M0A1U6F95G Jan 9 00:15:29 un1ty kernel: md: import disk1: (sdc) TOSHIBA_MG07ACA12TE_91M0A1U6F95G size: 11718885324 Jan 9 00:15:29 un1ty emhttpd: read SMART /dev/sdc Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=12s Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#0 Sense Key : 0x5 [current] Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#0 ASC=0x21 ASCQ=0x4 Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#0 CDB: opcode=0x88 88 00 00 00 00 00 85 29 96 b0 00 00 01 00 00 00 Jan 9 22:16:17 un1ty kernel: I/O error, dev sdc, sector 2234095280 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 2 Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=12s Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#5 Sense Key : 0x5 [current] Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#5 ASC=0x21 ASCQ=0x4 Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#5 CDB: opcode=0x88 88 00 00 00 00 04 9b 4f 67 28 00 00 01 00 00 00 Jan 9 22:16:17 un1ty kernel: I/O error, dev sdc, sector 19785541416 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 2 Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#6 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=12s Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#6 Sense Key : 0x5 [current] Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#6 ASC=0x21 ASCQ=0x4 Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#6 CDB: opcode=0x88 88 00 00 00 00 04 9b 4f 68 28 00 00 01 00 00 00 Jan 9 22:16:17 un1ty kernel: I/O error, dev sdc, sector 19785541672 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 2 Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#7 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=12s Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#7 Sense Key : 0x5 [current] Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#7 ASC=0x21 ASCQ=0x4 Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#7 CDB: opcode=0x88 88 00 00 00 00 04 9b 4f 69 28 00 00 01 00 00 00 Jan 9 22:16:17 un1ty kernel: I/O error, dev sdc, sector 19785541928 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 2 Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#17 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=12s Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#17 Sense Key : 0x5 [current] Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#17 ASC=0x21 ASCQ=0x4 Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#17 CDB: opcode=0x88 88 00 00 00 00 04 95 2f aa 48 00 00 01 00 00 00 Jan 9 22:16:17 un1ty kernel: I/O error, dev sdc, sector 19682798152 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 2 Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#28 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=12s Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#28 Sense Key : 0x5 [current] Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#28 ASC=0x21 ASCQ=0x4 Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#28 CDB: opcode=0x88 88 00 00 00 00 00 85 29 95 b0 00 00 00 c0 00 00 Jan 9 22:16:17 un1ty kernel: I/O error, dev sdc, sector 2234095024 op 0x0:(READ) flags 0x0 phys_seg 24 prio class 2 Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#29 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=12s Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#29 Sense Key : 0x5 [current] Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#29 ASC=0x21 ASCQ=0x4 Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#29 CDB: opcode=0x88 88 00 00 00 00 00 85 29 96 90 00 00 00 20 00 00 Jan 9 22:16:17 un1ty kernel: I/O error, dev sdc, sector 2234095248 op 0x0:(READ) flags 0x0 phys_seg 4 prio class 2 Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#30 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=12s Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#30 Sense Key : 0x5 [current] Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#30 ASC=0x21 ASCQ=0x4 Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] tag#30 CDB: opcode=0x88 88 00 00 00 00 00 85 29 cc d0 00 00 00 20 00 00 Jan 9 22:16:17 un1ty kernel: I/O error, dev sdc, sector 2234109136 op 0x0:(READ) flags 0x0 phys_seg 4 prio class 2 Jan 9 22:16:17 un1ty kernel: I/O error, dev sdc, sector 15108480952 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 2 Jan 9 22:16:17 un1ty kernel: I/O error, dev sdc, sector 3158244384 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] Synchronizing SCSI cache Jan 9 22:16:17 un1ty kernel: sd 8:0:0:0: [sdc] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=DRIVER_OK ** Press ANY KEY to close this window *

 

Edited January 10, 20242 yr by un1ty

1 hour ago, un1ty said:

They were louder than need be, they (completely) failed without warning, they wouldn't even show up in bios eventually.

Understand, I have many times diak failure in 1.5T / 3T Seagate or WD disk which longtime ago, but just 2 or 3 times in next 8yrs.

11 hours ago, Vr2Io said:

Understand, I have many times diak failure in 1.5T / 3T Seagate or WD disk which longtime ago, but just 2 or 3 times in next 8yrs.

In my old NAS I'm still running 2 WD Red pro 4TB drives in raid 0 since late 2014. They have a online time of over 9 years without any failure, issue or sign of slowing down.