limetech Posted May 18, 2018 Share Posted May 18, 2018 Installation and Bug Reporting Instructions This is a somewhat unusual release, meaning vs. 6.5.2 stable we only updated the kernel to latest patch release, but also changed a single kernel CONFIG setting that changes the kernel preemption model. This change should not have any deleterious effect on the server, and in fact may improve performance in some areas, certainly in VM startup (see below). However, we want to make this change in isolation and release to the Community in order to get testing on a wider range of hardware, especially to find out if parity operations are negatively affected. The reason we are releasing this as 6.5.3 is because for upcoming unRAID OS 6.6 we are moving to the linux 4.16 kernel and will also be including a fairly large base package update, including updated Samba, Docker, LIbvirt and QEMU. If we released this kernel CONFIG change along with all these other changes, and something is not working right, it could be very difficult to isolate. Background: several users have reported, and we have verified, that as the number of cores assigned to a VM increases, the POST time required to start a VM increases seemingly exponentially with OVMF and at least one GPU / PCI device being passthrough. Complicating matters, the issue only appears for certain Intel CPU families. It took a lot of work by @eschultz in consultation with a couple linux kernel developers to figure out what was causing this issue. It turns out that QEMU makes heavy use of a function associated with kernel CONFIG_PREEMPT_VOLUNTARY=yes to handle locking/unlocking of critical sections during VM startup. Using our previous kernel setting CONFIG_PREEMPT=yes makes this function a NO-OP and thus introduces serious, unnecessary locking delays as CPU cores are initialized. For core counts around 4-8 this delay is not that noticeable, but as the core count increases, VM start can take several minutes(!). We are very interested in seeing reports regarding any performance issues not seen in 6.5.2 release. As soon as we get this verified, we'll get this released to stable and get 6.6.0-rc1 out there. Thank you!! Version 6.5.3-rc1 2018-05-18 Summary: In order to fix VM startup issue we need to change the linux kernel preemption model: CONFIG_PREEMPT=no (previous setting) This option reduces the latency of the kernel by making all kernel code (that is not executing in a critical section) preemptible. This allows reaction to interactive events by permitting a low priority process to be preempted involuntarily even if it is in kernel mode executing a system call and would otherwise not be about to reach a natural preemption point. This allows applications to run more 'smoothly' even when the system is under load, at the cost of slightly lower throughpu and a slight runtime overhead to kernel code. Select this if you are building a kernel for a desktop or embedded system with latency requirements in the milliseconds range. CONFIG_PREEMPT_VOLUNTARY=yes (new in this release) This option reduces the latency of the kernel by adding more "explicit preemption points" to the kernel code. These new preemption points have been selected to reduce the maximum latency of rescheduling, providing faster application reactions, at the cost of slightly lower throughput. This allows reaction to interactive events by allowing a low priority process to voluntarily preempt itself even if it is in kernel mode executing a system call. This allows applications to run more 'smoothly' even when the system is under load. Select this if you are building a kernel for a desktop system. Linux kernel: version 4.14.41 set CONFIG_PREEMPT=no and CONFIG_PREEMPT_VOLUNTARY=yes 1 Quote Link to comment
SimonF Posted May 18, 2018 Share Posted May 18, 2018 Updated no noticeable change so far in start up, but only have 8 cores. Quote Link to comment
limetech Posted May 18, 2018 Author Share Posted May 18, 2018 Some notes from @eschultz: Quote All Haswell-arch CPUs I’ve tested (i7-4771, E3-1230 v3, and E5-2620 v3) seem immune regardless of how many vCPUs or ram is assigned 6.5.2: i7-8700K: 12 vcpu, 24GB ram --> ~90sec to post i5-3470: 4 vcpu, 24GB ram --> ~20sec to post 6.5.3-rc1: i7-8700K: 12 vcpu, 24GB ram --> ~4sec to post i5-3470: 4 vcpu, 24GB ram --> ~2sec to post Quote Link to comment
Squid Posted May 18, 2018 Share Posted May 18, 2018 Gen 8 i5-8400 all 6 cores passed through. Never noticed any issue with VM POST time before or after. Parity Check begun. Speeds consistent with 6.5.2 (23 drives). Won't know for another 13 hours though the final average speed. Quote Link to comment
Jcloud Posted May 18, 2018 Share Posted May 18, 2018 (edited) Ryzen Threadripper 1950X , OVMF , i440fx-2.11 6.5.2 stable: vcpu 4,8,12,28 allocations and 32GB of memory, 1070 gpu passed through; posted three times with each set; measuring from hitting start in webui to Mint gui fully booted, ~30-33 seconds. 6.5.3-rc1: vcpu 4 & 28 cores and 32GB of memory, 1070 gpu; posted three times with each set - same methods. Also ~30-33 seconds. Post times were probably ~2-4 second range, monitor kept going to sleep on vm shutdown making the measurement of "start" to logo difficult to measure. Parity check: 11 hours, 3mins. Edited May 20, 2018 by Jcloud removed questions; replaced with data. Quote Link to comment
eschultz Posted May 18, 2018 Share Posted May 18, 2018 1 hour ago, Squid said: Gen 8 i5-8400 all 6 cores passed through. Never noticed any issue with VM POST time before or after. It's only an issue if you're passing through a physical GPU or other PCI device. If it was just VNC then there was no OVMF startup delay. Quote Link to comment
Squid Posted May 18, 2018 Share Posted May 18, 2018 It's only an issue if you're passing through a physical GPU or other PCI device. If it was just VNC then there was no OVMF startup delay.Oh ok. Using seabios and GPU / usb passthru so I never would've been affected. Quote Link to comment
BobPhoenix Posted May 19, 2018 Share Posted May 19, 2018 Would seabios and pass through of HBA/TVTuners/USB help? Quote Link to comment
limetech Posted May 19, 2018 Author Share Posted May 19, 2018 21 minutes ago, BobPhoenix said: Would seabios and pass through of HBA/TVTuners/USB help? Sure! Every data point is appreciated. Quote Link to comment
BobPhoenix Posted May 19, 2018 Share Posted May 19, 2018 (edited) Will see how the Parity check time is tomorrow when it finishes but looking like it will be the same time frame as the last check 3 days ago. But I can give feedback on the VM startup time. Very little of my setup really applies to the first post but am obliging. Server spec: unRAID system: unRAID server Pro, version 6.5.3-rc1 Model: ASRock Dual X5 Xeons Motherboard: ASRock - EP2C602-4L/D16 Processor: Intel® Xeon® CPU E5-2670 0 @ 2.60GHz Cache pool: 2x PNY_CS2211_960GB_SSD Array: 2 parity 13 data array all 6TB HGST NAS drives. All VMs are seabios. 32Bit Windows WHSv1 can't tell any difference but it takes minutes (maybe10+) to start anyway - not really sure why but it is a blank screen for a long time and that wasn't reduced much if at all - image file is qcow2 drive so maybe that is why. Does have a 4 port Intel controller passed through for its storage drives root drive with C:\ and D:\ partitions are on image file on SSD cache pool. 32Bit Windows 7 seemed to boot faster. I could usually see the black screen before the login screen but didn't after installing 6.5.3-rc1. Just the login screen nothing was passed through to the VM. 64Bit Windows 7 seemed to boot faster. Same reasons as 32bit above and nothing passed through. Have another 64Bit Win7 VM that has tuners passed through that I will test tomorrow after parity check is done. unRAID 6.5.1 VM (yes I have an unRAID VM running on a host unRAID server (host specs above) - my case has 48 HDD bays on two SAS expanders so why not consolidate two unRAID servers on one box). This VM has two Happauge HVR-2250s passed through to it as well as a H310 HBA (LSI-9211-8i cross flashed) connected to one of the 24 bay SAS expanders in my case. This seemed to boot faster to the unRAID boot screen and a little beyond. Normally I see the PlopKExec screen displayed longer and after unRAID starts to book the "OK..."'s before the rest of the logging starts. This time I barely saw the PlopKExec screen and didn't see the "OK..."s at all. Could switch this VM to OVMF if you want. OK parity is done and here are the times for this one and the one just a few days before on 6.5.1: 2018-05-19, 08:23:51 10 hr, 59 min, 52 sec 151.6 MB/s OK 0 2018-05-15, 09:54:36 10 hr, 54 min, 35 sec 152.8 MB/s OK 0 Edited May 19, 2018 by BobPhoenix Quote Link to comment
killeriq Posted May 19, 2018 Share Posted May 19, 2018 where do i find/change this setting? i have 2xvm 1. 2x 1080ti passthrough on Linux 2. 1x R7 250x on Win10 thanks Quote Link to comment
nexusmaniac Posted May 19, 2018 Share Posted May 19, 2018 (edited) 3 hours ago, killeriq said: where do i find/change this setting? i have 2xvm 1. 2x 1080ti passthrough on Linux 2. 1x R7 250x on Win10 thanks The setting change has been made in the kernel. It's not a setting in any menu:) By using 6.5.3rc1 you are using the new setting ? Edited May 19, 2018 by nexusmaniac Quote Link to comment
1812 Posted May 19, 2018 Share Posted May 19, 2018 First test on the old z400 workstation setup as a 2 users 1 cpu machine: Model: Custom M/B: Hewlett-Packard - 0B4Ch CPU: Intel® Xeon® CPU W3520 @ 2.67GHz HVM: Enabled IOMMU: Enabled Cache: 256 kB, 1024 kB, 8192 kB Memory: 18 GB (max. installable capacity 24 GB) Network: bond0: fault-tolerance (active-backup), mtu 1500 eth0: 1000 Mb/s, full duplex, mtu 1500 Kernel: Linux 4.14.41-unRAID x86_64 OSX vm on ovmf 4 cores 10gb ram gtx 1050 time to Tianocore screen: 12 seconds OSX vm on ovmf 7 cores 10gb ram gtx 1050 time to Tianocore screen: 13 seconds Both times ram allocation appears to be near instantly. I also anticipate some of the time during bios post is spent waking the (older) monitor up, so it probably is faster to the bios screen in reality. Cinebench 6.5.2: 198 average 6.3.1: 194 average I'll get the big multi-cpu server going shortly to run some tests shortly on large cpu counts. Quote Link to comment
killeriq Posted May 19, 2018 Share Posted May 19, 2018 3 hours ago, nexusmaniac said: The setting change has been made in the kernel. It's not a setting in any menu:) By using 6.5.3rc1 you are using the new setting ? aah thought its somewhere in some file to edit to compare with/without that setting Quote Link to comment
Squid Posted May 19, 2018 Share Posted May 19, 2018 Parity Check completed. Average was actually a couple MB/s faster, but well within the usual margin of error. Quote Link to comment
dlandon Posted May 20, 2018 Share Posted May 20, 2018 Not very scientific, but VMs appear to start quicker and other things seem to be a bit more responsive like Dockers. Quote Link to comment
1812 Posted May 20, 2018 Share Posted May 20, 2018 ????? Server: Model: DL580 G7 M/B: - CPU: 2x Intel(R) Xeon(R) CPU E7- 4870 @ 2.40GHz HVM: Enabled IOMMU: Enabled Cache: 320 kB, 320 kB, 2560 kB, 2560 kB, 30720 kB, 30720 kB Memory: 64 GB Single-bit ECC (max. installable capacity 2048 GB) Network: bond0: fault-tolerance (active-backup), mtu 1500 eth0: 1000 Mb/s, full duplex, mtu 1500 eth1: not connected eth2: not connected eth3: not connected Kernel: Linux 4.14.41-unRAID x86_64 Boot times to tianocore logo 12 core vm with 24 gig of ram, and a gt 710 passed through 6.5.1/2: 1-2 minutes or so 6.5.3-rc1: 25-26 seconds 40 core vm with 24 gig of ram, and a gt 710 passed through 6.5.1/2: 40-65 minutes (not a typo) 6.5.3-rc1: 59-70 seconds Benchmarking 12 core vm with 24 gig of ram, and a gt 710 passed through. cinebench on 6.5.1/2 734 730 733 732 Cinebench on 6.5.3-rc1 844 848 839 I was unable to run a parity check because my external enclosure is in the process of failing and I'm looking for a replacement. GIGANTIC THANKS to @eschultz for working on this. This is a tremendous fix for something that greatly affected me , and will others more in the future. As someone who runs a 64 core vm, this is a colossal improvement in usability. I've had a few people turn their nose up and say "You have to pay for unRaid?" Yeah, and it's completely worth it. (now how about those multiple cache pools ) Quote Link to comment
SpaceInvaderOne Posted May 20, 2018 Share Posted May 20, 2018 (edited) Just upgraded my server from a Xeon 2690 v4 to a threadripper 1950x. And one of my main reasons was the slow boot to Tianocore that I was getting with the 2690 on my VMS with a GPU passed through so its great to see this problem resolved. A friend is having the 2690 and motherboard but hasn't collected from me as yet, so I thought I would do a quick test to see the difference. Host Xeon 2690 v4 ASRock x99 ws 32 gigs of ram GTX 750ti Guest - Win 10 8 cores 16 gigs ram GTX 750 ti Boot to Tianocore times unRAID 6.5.2 72 seconds unRAID 6.5.3-rc1: 18 seconds I will test my new server later but and report back but I didn't have the slow boot times like I did on the Xeon. But I would be interested to see if any benchmarks would be faster on 6.5.3-rc1 like the results @1812 had on his system. Also, thank you @eschultz for all your hard work. Edited May 20, 2018 by gridrunner Quote Link to comment
MowMdown Posted May 20, 2018 Share Posted May 20, 2018 unRAID 6.5.2 ~ 50 seconds unRAID 6.5.3-rc1 ~ 25 seconds Cut boot time roughly in half for me on my Xeon E3-1241v3 cpu passing my GTX 970 though. ? Quote Link to comment
eek Posted May 20, 2018 Share Posted May 20, 2018 I've not currently got any VM's on the server worth testing but I did do a parity check 6.5.3-rc1 - 2018-05-20, 19:51:00 10 hr, 35 min, 16 sec 105.0 MB/s OK 0 6.5.2-rc1 - 2018-05-04, 10:13:55 12 hr, 13 min, 54 sec 90.9 MB/s OK 0 100MB/s to 105MB/s so this is in the normal range. M/B: ASRock - Z77 Pro4-M CPU: Intel® Core™ i7-3770 CPU @ 3.40GHz HVM: Enabled IOMMU: Enabled Cache: 1024 kB, 128 kB, 8192 kB Memory: 32 GB (max. installable capacity 32 GB) Network: eth0: 1000 Mb/s, full duplex, mtu 1500 Quote Link to comment
archedraft Posted May 20, 2018 Share Posted May 20, 2018 Host Intel® Core™ i7-3770S CPU @ 3.10GHz ASRock - H77 Pro4-M 32 gigs of ram Guest - Win 10 OVMF 6 cores 16 gigs ram GTX 750 ti Boot to Tianocore times unRAID 6.5.2 53 seconds unRAID 6.5.3-rc1: 36 seconds Novabench Results (No idea if Novabench is any good but tested with it for fun) unRAID 6.5.2 1401 unRAID 6.5.3-rc1: 1512 Guest - OS X 10.13 OVMF 4 cores 8 gigs ram Radeon 6450 Boot to Tianocore times unRAID 6.5.2 1 minute 10 seconds unRAID 6.5.3-rc1: 1 minute 11 seconds Novabench Results (Novabench didn't recognize the CPU or RAM so the tests below should be taken with a grain of salt) unRAID 6.5.2 402 unRAID 6.5.3-rc1: 411 Quote Link to comment
GHunter Posted May 20, 2018 Share Posted May 20, 2018 VM 1: Windows 10 VM with OVMF and Q35 2.11 for Living Room. CPU pair 2/6 (Host Passthrough) with 4GB RAM. Passthrough nVidia GT 730 and Formosa USB Remote Control device (QEMU XHCI USB 3 Controller). Boot to Tianocore times: unRAID 6.5.2 5 seconds unRAID 6.5.3-rc1 4 seconds VM 2: Windows 10 VM with OVMF and Q35 2.11 for Master Bedroom Room. CPU pair 3/7 (Host Passthrough) with 4GB RAM. Passthrough nVidia GT 710 and Phillips USB Remote Control device (QEMU XHCI USB 3 Controller). Boot to Tianocore times: unRAID 6.5.2 7 seconds unRAID 6.5.3-rc1 6 seconds VM 3: Windows 10 VM with OVMF and Q35 2.11 for Development. CPU pair 2/6 and 3/7 (Host Passthrough) with 8GB RAM. Passthrough nVidia GTX 750 ti and Logitech Mouse/Keyboard (QEMU XHCI USB 3 Controller). Boot to Tianocore times: unRAID 6.5.2 12 seconds unRAID 6.5.3-rc1 6 seconds Hardware in my signature is up to date. I'll do a parity check tonight and post my results compared to previous unRAID version. Quote Link to comment
ryoko227 Posted May 21, 2018 Share Posted May 21, 2018 Just updated from 6.5.1 to 6.5.3-rc1 to see if this update fixed the long VM boot up times on this server. (Short answer - noticably faster) Running on a Intel Xeon E5-2670-V3 (Yokohama server) listed below in sig. Previously it would blackscreen for approximately 1-2mins with a 960gtx and "8" cores being passed through before the OVMF bios would pop up and then load Win10 as normal. After the update the bios splash pop'd up almost immediately. I can see (at the moment) no other issues. Quote Link to comment
pederm Posted May 21, 2018 Share Posted May 21, 2018 Installed 6.5.3-rc1 on my Ryzen system, apparently no change to boot-time of my Ubuntu Bionic VM with GTX1050-passthrough. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.