unRAID OS version 6.5.3-rc1 available - TESTERS NEEDED


Recommended Posts

Installation and Bug Reporting Instructions

 

This is a somewhat unusual release, meaning vs. 6.5.2 stable we only updated the kernel to latest patch release, but also changed a single kernel CONFIG setting that changes the kernel preemption model.  This change should not have any deleterious effect on the server, and in fact may improve performance in some areas, certainly in VM startup (see below).  However, we want to make this change in isolation and release to the Community in order to get testing on a wider range of hardware, especially to find out if parity operations are negatively affected.

 

The reason we are releasing this as 6.5.3 is because for upcoming unRAID OS 6.6 we are moving to the linux 4.16 kernel and will also be including a fairly large base package update, including updated Samba, Docker, LIbvirt and QEMU.  If we released this kernel CONFIG change along with all these other changes, and something is not working right, it could be very difficult to isolate.

 

Background: several users have reported, and we have verified, that as the number of cores assigned to a VM increases, the POST time required to start a VM increases seemingly exponentially with OVMF and at least one GPU / PCI device being passthrough.  Complicating matters, the issue only appears for certain Intel CPU families.  It took a lot of work by @eschultz in consultation with a couple linux kernel developers to figure out what was causing this issue.  It turns out that QEMU makes heavy use of a function associated with kernel CONFIG_PREEMPT_VOLUNTARY=yes to handle locking/unlocking of critical sections during VM startup.  Using our previous kernel setting CONFIG_PREEMPT=yes makes this function a NO-OP and thus introduces serious, unnecessary locking delays as CPU cores are initialized.  For core counts around 4-8 this delay is not that noticeable, but as the core count increases, VM start can take several minutes(!).

 

We are very interested in seeing reports regarding any performance issues not seen in 6.5.2 release.  As soon as we get this verified, we'll get this released to stable and get 6.6.0-rc1 out there.  Thank you!!

Version 6.5.3-rc1 2018-05-18

Summary:

In order to fix VM startup issue we need to change the linux kernel preemption model:

CONFIG_PREEMPT=no (previous setting)
This option reduces the latency of the kernel by making all kernel code (that is not executing in a critical section) preemptible. This allows reaction to interactive events by permitting a low priority process to be preempted involuntarily even if it is in kernel mode executing a system call and would otherwise not be about to reach a natural preemption point. This allows applications to run more 'smoothly' even when the system is under load, at the cost of slightly lower throughpu and a slight runtime overhead to kernel code.
Select this if you are building a kernel for a desktop or embedded system with latency requirements in the milliseconds range.
CONFIG_PREEMPT_VOLUNTARY=yes (new in this release)
This option reduces the latency of the kernel by adding more "explicit preemption points" to the kernel code. These new preemption points have been selected to reduce the maximum latency of rescheduling, providing faster application reactions, at the cost of slightly lower throughput.
This allows reaction to interactive events by allowing a low priority process to voluntarily preempt itself even if it is in kernel mode executing a system call. This allows applications to run more 'smoothly' even when the system is under load.
Select this if you are building a kernel for a desktop system.

Linux kernel:

  • version 4.14.41
    • set CONFIG_PREEMPT=no and CONFIG_PREEMPT_VOLUNTARY=yes
  • Like 1
Link to comment

Some notes from @eschultz:

 

Quote

All Haswell-arch CPUs I’ve tested (i7-4771, E3-1230 v3, and E5-2620 v3) seem immune regardless of how many vCPUs or ram is assigned

6.5.2:

  i7-8700K: 12 vcpu, 24GB ram --> ~90sec to post

  i5-3470:  4 vcpu, 24GB ram --> ~20sec to post

 

6.5.3-rc1:

  i7-8700K: 12 vcpu, 24GB ram --> ~4sec to post

  i5-3470:  4 vcpu, 24GB ram --> ~2sec to post

 

 

Link to comment

Gen 8 i5-8400 all 6 cores passed through.  Never noticed any issue with VM POST time before or after.

 

Parity Check begun.  Speeds consistent with 6.5.2 (23 drives).  Won't know for another 13 hours though the final average speed.

Link to comment

Ryzen Threadripper 1950X ,  OVMF ,  i440fx-2.11

6.5.2 stable:

vcpu 4,8,12,28 allocations and 32GB of memory, 1070 gpu passed through; posted three times with each set; measuring from hitting start in webui to Mint gui fully booted, ~30-33 seconds.

6.5.3-rc1:

vcpu 4 & 28 cores and 32GB of memory, 1070 gpu; posted three times with each set - same methods. Also ~30-33 seconds. 

 

Post times were probably ~2-4 second range, monitor kept going to sleep on vm shutdown making the measurement of "start" to logo difficult to measure. 

 

 

Parity check: 11 hours, 3mins.

 

Edited by Jcloud
removed questions; replaced with data.
Link to comment
1 hour ago, Squid said:

Gen 8 i5-8400 all 6 cores passed through.  Never noticed any issue with VM POST time before or after.

 

It's only an issue if you're passing through a physical GPU or other PCI device.  If it was just VNC then there was no OVMF startup delay.

Link to comment

Will see how the Parity check time is tomorrow when it finishes but looking like it will be the same time frame as the last check 3 days ago.  But I can give feedback on the VM startup time.  Very little of my setup really applies to the first post but am obliging.

 

Server spec: 

unRAID system:    unRAID server Pro, version 6.5.3-rc1
Model:    ASRock Dual X5 Xeons
Motherboard:    ASRock - EP2C602-4L/D16
Processor:    Intel® Xeon® CPU E5-2670 0 @ 2.60GHz

Cache pool: 2x PNY_CS2211_960GB_SSD

Array: 2 parity 13 data array all 6TB HGST NAS drives.

All VMs are seabios.

 

  1. 32Bit Windows WHSv1 can't tell any difference but it takes minutes (maybe10+) to start anyway - not really sure why but it is a blank screen for a long time and that wasn't reduced much if at all - image file is qcow2 drive so maybe that is why. Does have a 4 port Intel controller passed through for its storage drives root drive with C:\ and D:\ partitions are on image file on SSD cache pool.
  2. 32Bit Windows 7 seemed to boot faster.  I could usually see the black screen before the login screen but didn't after installing 6.5.3-rc1. Just the login screen nothing was  passed through to the VM.
  3. 64Bit Windows 7 seemed to boot faster.  Same reasons as 32bit above and nothing passed through.
  4. Have another 64Bit Win7 VM that has tuners passed through that I will test tomorrow after parity check is done.
  5. unRAID 6.5.1 VM (yes I have an unRAID VM running on a host unRAID server (host specs above) - my case has 48 HDD bays on two SAS expanders so why not consolidate two unRAID servers on one box).  This VM has two Happauge HVR-2250s passed through to it as well as a H310 HBA (LSI-9211-8i cross flashed) connected to one of the 24 bay SAS expanders in my case.  This seemed to boot faster to the unRAID boot screen and a little beyond.  Normally I see the PlopKExec screen displayed longer and after unRAID starts to book the "OK..."'s before the rest of the logging starts.  This time I barely saw the PlopKExec screen and didn't see the "OK..."s at all.  Could switch this VM to OVMF if you want.

 

OK parity is done and here are the times for this one and the one just a few days before on 6.5.1:

2018-05-19, 08:23:51    10 hr, 59 min, 52 sec    151.6 MB/s   OK   0

2018-05-15, 09:54:36    10 hr, 54 min, 35 sec    152.8 MB/s   OK   0

Edited by BobPhoenix
Link to comment
3 hours ago, killeriq said:

where do i find/change this setting?

 

i have 2xvm

1. 2x 1080ti passthrough on Linux

2. 1x R7 250x on Win10

 

thanks

The setting change has been made in the kernel. It's not a setting in any menu:)

 

By using 6.5.3rc1 you are using the new setting ?

Edited by nexusmaniac
Link to comment

First test on the old z400 workstation setup as a 2 users 1 cpu machine:

 

Model: Custom
M/B: Hewlett-Packard - 0B4Ch
CPU: Intel® Xeon® CPU W3520 @ 2.67GHz
HVM: Enabled
IOMMU: Enabled
Cache: 256 kB, 1024 kB, 8192 kB
Memory: 18 GB (max. installable capacity 24 GB)
Network: bond0: fault-tolerance (active-backup), mtu 1500 
 eth0: 1000 Mb/s, full duplex, mtu 1500
Kernel: Linux 4.14.41-unRAID x86_64
 
 
 
OSX vm on ovmf 4 cores 10gb ram gtx 1050 time to Tianocore screen: 12 seconds 
OSX vm on ovmf 7 cores 10gb ram gtx 1050 time to Tianocore screen: 13 seconds
 
Both times ram allocation appears to be near instantly. I also anticipate some of the time during bios post is spent waking the (older) monitor up, so it probably is faster to the bios screen in reality.
 
Cinebench
6.5.2: 198 average
6.3.1: 194 average 
 
 
 
I'll get the big multi-cpu server going shortly to run some tests shortly on large cpu counts.
 
Link to comment

?????

 

Server:

 

Model: DL580 G7

M/B: -

CPU: 2x Intel(R) Xeon(R) CPU E7- 4870  @ 2.40GHz

HVM: Enabled

IOMMU: Enabled

Cache: 320 kB, 320 kB, 2560 kB, 2560 kB, 30720 kB, 30720 kB

Memory: 64 GB Single-bit ECC (max. installable capacity 2048 GB)

Network: bond0: fault-tolerance (active-backup), mtu 1500 
 eth0: 1000 Mb/s, full duplex, mtu 1500 
 eth1: not connected
 eth2: not connected
 eth3: not connected

Kernel: Linux 4.14.41-unRAID x86_64

 

 

 

Boot times to tianocore logo

 

12 core vm with 24 gig of ram, and a gt 710 passed through

6.5.1/2: 1-2 minutes or so

6.5.3-rc1: 25-26 seconds 

 

40 core vm with 24 gig of ram, and a gt 710 passed through
6.5.1/2: 40-65 minutes (not a typo)

6.5.3-rc1: 59-70 seconds 

 

 

 

Benchmarking


12 core vm with 24 gig of ram, and a gt 710 passed through. 

 

cinebench on 6.5.1/2 
734
730
733
732


Cinebench on 6.5.3-rc1 
844
848
839

 

 

I was unable to run a parity check because my external enclosure is in the process of failing and I'm looking for a replacement.

 

GIGANTIC THANKS to @eschultz for working on this. This is a tremendous fix for something that greatly affected me , and will others more in the future. As someone who runs a 64 core vm, this is a colossal improvement in usability. I've had a few people turn their nose up and say "You have to pay for unRaid?" Yeah, and it's completely worth it.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

(now how about those multiple cache pools :) )

Link to comment

Just upgraded my server from a Xeon 2690 v4 to a threadripper 1950x. And one of my main reasons was the slow boot to Tianocore that I was getting with the 2690

on my VMS with a GPU passed through so its great to see this problem resolved.

A friend is having the 2690 and motherboard but hasn't collected from me as yet, so I thought I would do a quick test to see the difference.

 

Host

Xeon 2690 v4

ASRock x99 ws

32 gigs of ram

GTX 750ti

 

Guest - Win 10

8 cores

16 gigs ram

GTX  750 ti

 

Boot to Tianocore times

 

unRAID 6.5.2             72 seconds

unRAID 6.5.3-rc1:     18 seconds

 

 

I will test my new server later but and report back but I didn't have the slow boot times like I did on the Xeon.

But I would be interested to see if any benchmarks would be faster on 6.5.3-rc1 like the results @1812 had on his system.

 

Also, thank you @eschultz for all your hard work. 

Edited by gridrunner
Link to comment

I've not currently got any VM's on the server worth testing but I did do a parity check

 

6.5.3-rc1 - 2018-05-20, 19:51:00 10 hr, 35 min, 16 sec 105.0 MB/s OK 0
6.5.2-rc1 - 2018-05-04, 10:13:55 12 hr, 13 min, 54 sec 90.9 MB/s OK 0

 

100MB/s to 105MB/s so this is in the normal range. 

 

M/B: ASRock - Z77 Pro4-M
CPU: Intel® Core™ i7-3770 CPU @ 3.40GHz
HVM: Enabled
IOMMU: Enabled
Cache: 1024 kB, 128 kB, 8192 kB
Memory: 32 GB (max. installable capacity 32 GB)
Network: eth0: 1000 Mb/s, full duplex, mtu 1500
 
 
Link to comment

Host

Intel® Core™ i7-3770S CPU @ 3.10GHz

ASRock - H77 Pro4-M

32 gigs of ram

 

Guest - Win 10

OVMF

6 cores

16 gigs ram

GTX  750 ti

 

Boot to Tianocore times

 

unRAID 6.5.2             53 seconds

unRAID 6.5.3-rc1:     36 seconds

 

Novabench Results (No idea if Novabench is any good but tested with it for fun)

 

unRAID 6.5.2             1401

unRAID 6.5.3-rc1:     1512

 

Guest - OS X 10.13

OVMF

4 cores

8 gigs ram

Radeon 6450

 

Boot to Tianocore times

 

unRAID 6.5.2             1 minute 10 seconds

unRAID 6.5.3-rc1:     1 minute 11 seconds

 

Novabench Results (Novabench didn't recognize the CPU or RAM so the tests below should be taken with a grain of salt)

 

unRAID 6.5.2             402

unRAID 6.5.3-rc1:     411

Link to comment

VM 1: Windows 10 VM with OVMF and Q35 2.11 for Living Room. CPU pair 2/6 (Host Passthrough) with 4GB RAM. Passthrough nVidia GT 730 and Formosa USB Remote Control device (QEMU XHCI USB 3 Controller).

 

Boot to Tianocore times:

unRAID 6.5.2          5 seconds
unRAID 6.5.3-rc1   4 seconds

 

VM 2: Windows 10 VM with OVMF and Q35 2.11 for Master Bedroom Room. CPU pair 3/7 (Host Passthrough) with 4GB RAM. Passthrough nVidia GT 710 and Phillips USB Remote Control device (QEMU XHCI USB 3 Controller).

 

Boot to Tianocore times:

unRAID 6.5.2          7 seconds
unRAID 6.5.3-rc1   6 seconds

 

VM 3: Windows 10 VM with OVMF and Q35 2.11 for Development. CPU pair 2/6 and 3/7 (Host Passthrough) with 8GB RAM. Passthrough nVidia GTX 750 ti and Logitech Mouse/Keyboard (QEMU XHCI USB 3 Controller).

 

Boot to Tianocore times:

unRAID 6.5.2         12 seconds
unRAID 6.5.3-rc1    6 seconds

 

Hardware in my signature is up to date. I'll do a parity check tonight and post my results compared to previous unRAID version.

Link to comment

Just updated from 6.5.1 to 6.5.3-rc1 to see if this update fixed the long VM boot up times on this server. (Short answer - noticably faster)

 

Running on a Intel Xeon E5-2670-V3 (Yokohama server) listed below in sig. Previously it would blackscreen for approximately 1-2mins with a 960gtx and "8" cores being passed through before the OVMF bios would pop up and then load Win10 as normal. After the update the bios splash pop'd up almost immediately. I can see (at the moment) no other issues. 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.