snailbrain Posted February 25, 2019 Author Share Posted February 25, 2019 3 hours ago, 1812 said: what guide did you follow in setting up the windows vm? at this point, I'd make a new windows 10 vm and see if the issues are still persistent to better isolate where the source problem is, as up to this point, there have been wild guesses and standard optimization solutions presented. also, take your main image off the cache and put it on another fast drive via unassigned devices. Thanks, I've ordered an SSD for that purpose. We'll see how it goes. I didn't follow much of a guide - but seemed self explanatory and i've tweaked it by reading the forum to wher to i am now - great help from everyone. I am a bit disappointed i need to pin and isolate 1 core for emu. but as long as it works. The main issue i had was with HTEP, disabling this helped massively. 3 hours ago, bastl said: Another thing i noticed you're using 2 cache devices, right? I guess they are mirrored (RAID1). Keep in mind that the controller these devices are connected to has to handle the write operations twice and also the calculations for the parity. Is this a onboard sata controller directly connected to the CPU or connected to the chipset sharing the PCIe lanes with other devices like USB controllers, or even NVME devices. Or do you use an extra RAID card where these disks are connected to? Which slot on the board? How is this PCIe slot connceted? Directly or over chipset? Tons of questions. And you might get an idea which direction i try to point you. Lets say your M.2 slot for your E drive is connected to the chipset which is limited to 4 PCIe lanes to the CPU and at the same time you using a raid card also sharing these 4 lanes, I guess you get the idea the chipset becomes the bottleneck here and can cause issues. Check your motherboards manual and check which M.2 slots and which PCIe slots are connected where and try to eliminate bottlenecks. Thanks again. Yes they are mirrored (or however unraid does it). The parity drives are on the motherboard, the cache drives are on the raid controller (with 4 other 3tb wd reds). 2x 2TB drives are also on the motherboard + a 120gb ssd for temp storage. The VM has the nvme controller + usb controller (4x 3.0 ports) passed through to them and they are not shared (when acs is disabled they are in their own group). However the Raid card is shared with various other devices.. Yeh i understand much of the sharing of lanes etc (- before unraid for several years i used to have my windows desktop as a VM in ESXI, only i was using an AMD card as they were the only ones you could passthrough (at the time at least). I guess one thing I could try is swap my raid card with my intel network card. The intel network card is in it's own group, the raid controller is shared with other stuff. It's currently passed through to a pfsense VM. The Intel card sharing with some other stuff might not hurt as much and giving the raid card the full lane may help? I played with the "cache" (for ram cache?) in Tips and Tweaks plugin and this has definitely helped a lot (as suggested by someone else in here) but the transfer rates are abysmal and still get spikes - performance is almost like there is no point having a cache drive though. I assumed that by having the cache drive, when I want to copy something onto my array (and use the cache), i'll be getting 10gbit/sec transfer rates.. (or max out the write speeds of my 2x 25" ssds, probably over 400MB/sec. Quote Link to comment
snailbrain Posted February 25, 2019 Author Share Posted February 25, 2019 Update: Swapping the Raid card into it's own group has definitely improved things. Copying onto the Array now (onto a disk that is solely on the raid card and onto a disk on the motherboard [although parities are both on the motherboard]) without the cache drive seems pretty good. I do see the occasional spike but it's tolerable. Also the CPU usage is not major in unraid... is it recommended to put the parity drives onto the lsi raid card? BUT - copying to a cache share, even though it's copying really rally slow (slower than copying directly onto the array), uses the emu pin at 100% and the 3 cores i have for unraid.. they all max out in red. I then start to see the spikes again.. Copying onto the vdisk (c drive which is on the cache) is still fine except slow. and i only see the emu core at 100%, not every other unraid core. This leaves me to believe that unraid is doing something with the files that it is not doing with vdisk - for example, like the Intergrity file checksum checker (but i dont have that on, and it works fine when copying to the normal array). The ssds are cheapo though - A400 Kingston. Quote Link to comment
snailbrain Posted February 26, 2019 Author Share Posted February 26, 2019 Update: I got another SSD and using that for the C drive on windows now (instead of the cache). Same issue - although everything has improved since i started the threads. The issue now is just when using the cache drive(s), the 3 unraid cores (full cores + HT) - 6 threads) i've reserved max out at 100% and i get unusable windows. Quote Link to comment
1812 Posted February 26, 2019 Share Posted February 26, 2019 8 minutes ago, snailbrain said: Update: I got another SSD and using that for the C drive on windows now (instead of the cache). Same issue - although everything has improved since i started the threads. The issue now is just when using the cache drive(s), the 3 unraid cores (full cores + HT) - 6 threads) i've reserved max out at 100% and i get unusable windows. post new diagnostics and your windows xml Quote Link to comment
snailbrain Posted February 26, 2019 Author Share Posted February 26, 2019 Thanks. <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='7'> <name>WinMain</name> <uuid>7af4209a-1db7-2d25-02df-14750ef934ae</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>16252928</memory> <currentMemory unit='KiB'>16252928</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='8'/> <vcpupin vcpu='1' cpuset='9'/> <vcpupin vcpu='2' cpuset='10'/> <vcpupin vcpu='3' cpuset='11'/> <vcpupin vcpu='4' cpuset='12'/> <vcpupin vcpu='5' cpuset='13'/> <vcpupin vcpu='6' cpuset='14'/> <vcpupin vcpu='7' cpuset='15'/> <emulatorpin cpuset='6'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-3.0'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/7af4209a-1db7-2d25-02df-14750ef934ae_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='8' threads='1'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/disk/by-id/ata-PNY_CS900_240GB_SSD_PNY0219000039020481A'/> <backingStore/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='ide' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='sata' index='0'> <alias name='sata0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:bc:cd:4c'/> <source bridge='br0'/> <target dev='vnet1'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/2'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-7-WinMain/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x20' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom file='/mnt/user/Downloads/bios/Gigabyte2.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x20' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x22' slot='0x00' function='0x3'/> </source> <alias name='hostdev3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain> urhpg8-diagnostics-20190226-1206.zip Quote Link to comment
snailbrain Posted February 26, 2019 Author Share Posted February 26, 2019 (edited) I think could be related to networking (virtual). Downloading a game on steam (at 40MB/second), - Steam is using 50% of the 4 cores (+threads) in unraid the VM is using 60-70% in total. Edited February 26, 2019 by snailbrain Quote Link to comment
bastl Posted February 26, 2019 Share Posted February 26, 2019 You're downloading to the second passed through SSD, right? Does the same happen if you download a bigger file to the C drive? Quote Link to comment
snailbrain Posted February 26, 2019 Author Share Posted February 26, 2019 Hi thanks. Yes. but i just checked with the C drive, same thing. almost maxing out all 4 cores. note: the NVME i have passed through the entire controller (e drive) and the C drive is now a passed through by dev/disk/by-id/xxxx Also getting slow down now when my temp ssd (120gb ssd unassigned) copies (moves) my download torrents to the array (ignoring cache). I'm wondering if the 6x On Board Sata ports are the cause. I'm using all 6 (and 6 on the raid card) on the Motherboard I have 2x 6TB WD Red (2x Parity) 1x 120GB SSD (temp drive for torrents [they give moved onto array but not through cache, when completed]) 1x 240GB SSD (C Drive) 2x Toshiba 2TB Mechanical (on Array) on Raid Controller 4x 3TB WD Red (on array) 2x 1TB Kingston SSD (cache) I've ordered a spf cable for my raid controller and will connected the 2x Parity to it (or the SSDs) Wondering if: the motherboard ports are saturated (but they not transferring that fast..) the Network card is now in a shared slot (but it's a 4 port gigabit, and only using 1 of the ports for WAN and other for LAN (passed through to pfsense). The motherboard onboard lan is also being used for Unraid itself (and VMs/Dockers etc). Quote Link to comment
snailbrain Posted February 26, 2019 Author Share Posted February 26, 2019 (edited) Here is an example with more info: I'm copying from my nvme to a share which IS using the cache. Apart from it grinding to a halt, it's also using masses of CPU. The first 3 cores are used by unraid and are 100% during the attempted copy. Strangly in "top" it does not show what is using the CPU (is that normal or does it help diagnose) the terminal window there was inactive but same result (doesn;t show anything in top using 300%.. etc Edited February 26, 2019 by snailbrain Quote Link to comment
bastl Posted February 27, 2019 Share Posted February 27, 2019 @snailbrain Are you using some sort of IDS like Snort ur Suricata on Pfsense to check your network traffic? This would explain why you see these CPU spikes. I'am using Snort on an separated Pfsense box as my main router. Snort is very heavy on the CPU if you have set it up to check all your Lan traffic. Try to reproduce the issue without the Pfsense running if you can. Quote Link to comment
snailbrain Posted February 27, 2019 Author Share Posted February 27, 2019 (edited) 12 hours ago, bastl said: @snailbrain Are you using some sort of IDS like Snort ur Suricata on Pfsense to check your network traffic? This would explain why you see these CPU spikes. I'am using Snort on an separated Pfsense box as my main router. Snort is very heavy on the CPU if you have set it up to check all your Lan traffic. Try to reproduce the issue without the Pfsense running if you can. Hi and thanks again. No i'm not using those. I've also tried closing down the VMs and same issue. it's also going through the virtual switch so not touching pfsense atm - pretty much. here is copying from nvme to cached share (6-15 are isolated. and windows is using 8-15. emu on 6. Edited February 27, 2019 by snailbrain Quote Link to comment
snailbrain Posted February 28, 2019 Author Share Posted February 28, 2019 (edited) UPDATE: I disabled the virtual network adapter in windows, and passed through one of the ports on my intel quad 1gb pcie card and connected it to my switch. It works like a dream - i still see some high CPU usage for the unraid cores, but everything works great (except limited to 100MB/sec). BUT, unfortunatly, my pfsense box will not boot if i don't pass through the whole card for some reason, even though the card is stubbed and i can select each port to pass through... in anycase - this leaves me to believe that the problem is related to: 1. Caching from the VM (i've played with the numbers in the tips and tweaks plugin but either it becomes unusably slow or makes jerking worse) 2. The Virtual LAN adapter. (note: i also got a bit of an improvement by moving around the cores used by the VM. I noticed that cpu 8 (4th core), even though was isolated and nothing was pinned to it, and the VM was not running, was still using 1-4%, and have no idea why - so i used this as emu pin, and used cpu 6 as the 1st core of the VM. This definitely helped a bit. Edited February 28, 2019 by snailbrain Quote Link to comment
JesterEE Posted December 24, 2019 Share Posted December 24, 2019 @snailbrain did you ever get completely to root cause? I have a 95% decent experience on my QEMU 4.1 i440fx Windows 10 VM, but I am getting audio glitching somewhat frequently through my NVIDIA GTX 1060 HDMI audio passthrough similar to what you have in your video. It's really jarring, and kind of killing the enjoyable experience, so I want it gone! I've done most of the same configuration you have, except I'm still using a vdisk for OS loaded on an unassigned device and still using the virtual network interface. Your experience is the closest I've seen to what i have been experiencing and you've done a bunch more testing than I have ... though I'm getting close 😂. Anything you can share would be great! Thanks -JesterEE Quote Link to comment
bastl Posted December 24, 2019 Share Posted December 24, 2019 @JesterEE Have you tried using the MSI_util to fix the audio stuttering/glitching? https://wiki.unraid.net/index.php/UnRAID_Manual_6#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support Quote Link to comment
JesterEE Posted December 25, 2019 Share Posted December 25, 2019 @bastl Yes I have. Thanks! I don't want to both necropost AND hijack a thread. I'd still like to follow-up with @snailbrain here, but for working with my own configuration woes, I have opened an new thread. Quote Link to comment
tsawind Posted November 5, 2020 Share Posted November 5, 2020 I too have been battling similar latency issues for over 2 years now, I just call it the unRAID stutter... Mine is the worst when I write to my array without a cache drive, but dual parity, I believe that the parity calculations overwhelm the cpu and cause major stutter. This is an old link roughly showing my system: https://pcpartpicker.com/b/6vGG3C This has been helpful, I just have one question. What is htep and where do I find it? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.