[6.6.6] [BOUNTY] Ryzen 2700x - [VIDEO] Latency, stuttering, STRANGE issues


Recommended Posts

3 hours ago, 1812 said:

what guide did you follow in setting up the windows vm? at this point, I'd make a new windows 10 vm and see if the issues are still persistent to better isolate where the source problem is, as up to this point, there have been wild guesses and standard optimization solutions presented.

 

also, take your main image off the cache and put it on another fast drive via unassigned devices.

Thanks, I've ordered an SSD for that purpose. We'll see how it goes.

I didn't follow much of a guide - but seemed self explanatory and i've tweaked it by reading the forum to wher to i am now - great help from everyone.

 

I am a bit disappointed i need to pin and isolate 1 core for emu. but as long as it works.
The main issue i had was with HTEP, disabling this helped massively.
 

3 hours ago, bastl said:

Another thing i noticed you're using 2 cache devices, right? I guess they are mirrored (RAID1). Keep in mind that the controller these devices are connected to has to handle the write operations twice and also the calculations for the parity. Is this a onboard sata controller directly connected to the CPU or connected to the chipset sharing the PCIe lanes with other devices like USB controllers, or even NVME devices. Or do you use an extra RAID card where these disks are connected to? Which slot on the board? How is this PCIe slot connceted? Directly or over chipset? Tons of questions. And you might get an idea which direction i try to point you. Lets say your M.2 slot for your E drive is connected to the chipset which is limited to 4 PCIe lanes to the CPU and at the same time you using a raid card also sharing these 4 lanes, I guess you get the idea the chipset becomes the bottleneck here and can cause issues. Check your motherboards manual and check which M.2 slots and which PCIe slots are connected where and try to eliminate bottlenecks.


Thanks again.

Yes they are mirrored (or however unraid does it).

The parity drives are on the motherboard, the cache drives are on the raid controller (with 4 other 3tb wd reds). 2x 2TB drives are also on the motherboard + a 120gb ssd for temp storage.

The VM has the nvme controller + usb controller (4x 3.0 ports) passed through to them and they are not shared (when acs is disabled they are in their own group).

However the Raid card is shared with various other devices..

Yeh i understand much of the sharing of lanes etc (- before unraid for several years i used to have my windows desktop as a VM in ESXI, only i was using an AMD card as they were the only ones you could passthrough (at the time at least).

I guess one thing I could try is swap my raid card with my intel network card. The intel network card is in it's own group, the raid controller is shared with other stuff. It's currently passed through to a pfsense VM.
The Intel card sharing with some other stuff might not hurt as much and giving the raid card the full lane may help?

I played with the "cache" (for ram cache?) in Tips and Tweaks plugin and this has definitely helped a lot (as suggested by someone else in here) but the transfer rates are abysmal and still get spikes  - performance is almost like there is no point having a cache drive though. 

I assumed that by having the cache drive, when I want to copy something onto my array (and use the cache), i'll be getting 10gbit/sec transfer rates..  (or max out the write speeds of my 2x 25" ssds, probably over 400MB/sec.

 

Link to comment

Update:

Swapping the Raid card into it's own group has definitely improved things.

 

Copying onto the Array now (onto a disk that is solely on the raid card and onto a disk on the motherboard [although parities are both on the motherboard]) without the cache drive seems pretty good. I do see the occasional spike but it's tolerable.

Also the CPU usage is not major in unraid... is it recommended to put the parity drives onto the lsi raid card?

BUT - copying to a cache share, even though it's copying really rally slow (slower than copying directly onto the array), uses the emu pin at 100% and the 3 cores i have for unraid.. they all max out in red.
I then start to see the spikes again..

 

Copying onto the vdisk (c drive which is on the cache) is still fine except slow. and i only see the emu core at 100%, not every other unraid core.

This leaves me to believe that unraid is doing something with the files that it is not doing with vdisk - for example, like the Intergrity file checksum checker (but i dont have that on, and it works fine when copying to the normal array).

The ssds are cheapo though - A400 Kingston.

Link to comment

Update:

I got another SSD and using that for the C drive on windows now (instead of the cache).

Same issue - although everything has improved since i started the threads.

The issue now is just when using the cache drive(s), the 3 unraid cores (full cores + HT) - 6 threads) i've reserved max out at 100% and i get unusable windows.

Link to comment
8 minutes ago, snailbrain said:

Update:

I got another SSD and using that for the C drive on windows now (instead of the cache).

Same issue - although everything has improved since i started the threads.

The issue now is just when using the cache drive(s), the 3 unraid cores (full cores + HT) - 6 threads) i've reserved max out at 100% and i get unusable windows.

post new diagnostics and your windows xml

Link to comment

Thanks.

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='7'>
  <name>WinMain</name>
  <uuid>7af4209a-1db7-2d25-02df-14750ef934ae</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>16252928</memory>
  <currentMemory unit='KiB'>16252928</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='8'/>
    <vcpupin vcpu='1' cpuset='9'/>
    <vcpupin vcpu='2' cpuset='10'/>
    <vcpupin vcpu='3' cpuset='11'/>
    <vcpupin vcpu='4' cpuset='12'/>
    <vcpupin vcpu='5' cpuset='13'/>
    <vcpupin vcpu='6' cpuset='14'/>
    <vcpupin vcpu='7' cpuset='15'/>
    <emulatorpin cpuset='6'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-3.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/7af4209a-1db7-2d25-02df-14750ef934ae_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='8' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/disk/by-id/ata-PNY_CS900_240GB_SSD_PNY0219000039020481A'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='sata0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:bc:cd:4c'/>
      <source bridge='br0'/>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-7-WinMain/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x20' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/user/Downloads/bios/Gigabyte2.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x20' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x22' slot='0x00' function='0x3'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

 

urhpg8-diagnostics-20190226-1206.zip

Link to comment

Hi thanks.

 

Yes. but i just checked with the C drive, same thing. almost maxing out all 4 cores.

note: the NVME i have passed through the entire controller (e drive)

and the C drive is now a passed through by dev/disk/by-id/xxxx

Also getting slow down now when my temp ssd (120gb ssd unassigned) copies (moves) my download torrents to the array (ignoring cache). 

I'm wondering if the 6x On Board Sata ports are the cause. I'm using all 6 (and 6 on the raid card)

on the Motherboard I have

2x 6TB WD Red (2x Parity)
1x 120GB SSD (temp drive for torrents [they give moved onto array but not through cache, when completed])

1x 240GB SSD (C Drive)
2x Toshiba 2TB Mechanical (on Array)

on Raid Controller

4x 3TB WD Red (on array)
2x 1TB Kingston SSD (cache)

I've ordered a spf cable for my raid controller and will connected the 2x Parity to it (or the SSDs)

Wondering if:
the motherboard ports are saturated (but they not transferring that fast..)

the Network card is now in a shared slot (but it's a 4 port gigabit, and only using 1 of the ports for WAN and other for LAN (passed through to pfsense).  The motherboard onboard lan is also  being used for Unraid itself (and VMs/Dockers etc).
 

Link to comment

Here is an example with more info:

 

I'm copying from my nvme to a share which IS using the cache. Apart from it grinding to a halt, it's also using masses of CPU.

 

The first 3 cores are used by unraid and are 100% during the attempted copy. Strangly in "top" it does not show what is using the CPU (is that normal or does it help diagnose)

image.thumb.png.d322a189cf6d34be2864872d36963fb1.png

 

the terminal window there was inactive :D but same result (doesn;t show anything in top using 300%.. etc

Edited by snailbrain
Link to comment

@snailbrain Are you using some sort of IDS like Snort ur Suricata on Pfsense to check your network traffic? This would explain why you see these CPU spikes. I'am using Snort on an separated Pfsense box as my main router. Snort is very heavy on the CPU if you have set it up to check all your Lan traffic. Try to reproduce the issue without the Pfsense running if you can. 

Link to comment
12 hours ago, bastl said:

@snailbrain Are you using some sort of IDS like Snort ur Suricata on Pfsense to check your network traffic? This would explain why you see these CPU spikes. I'am using Snort on an separated Pfsense box as my main router. Snort is very heavy on the CPU if you have set it up to check all your Lan traffic. Try to reproduce the issue without the Pfsense running if you can. 

Hi and thanks again. No i'm not using those. I've also tried closing down the VMs and same issue.

it's also going through the virtual switch so not touching pfsense atm - pretty much.
 

here is copying from nvme to cached share (6-15 are isolated. and windows is using 8-15. emu on 6.

image.thumb.png.43130b1220c262ad5b48575517c6ae7a.png

 

 

Edited by snailbrain
Link to comment

UPDATE:

 

I disabled the virtual network adapter in windows, and passed through one of the ports on my intel quad 1gb pcie card and connected it to my switch.

It works like a dream - i still see some high CPU usage for the unraid cores, but everything works great (except limited to 100MB/sec).
BUT, unfortunatly, my pfsense box will not boot if i don't pass through the whole card for some reason, even though the card is stubbed and i can select each port to pass through...

in anycase - this leaves me to believe that the problem is related to:

 

1. Caching from the VM (i've played with the numbers in the tips and tweaks plugin but either it becomes unusably slow or makes jerking worse)

2. The Virtual LAN adapter.

 

(note: i also got a bit of an improvement by moving around the cores used by the VM. I noticed that cpu 8 (4th core), even though was isolated and nothing was pinned to it, and the VM was not running, was still using 1-4%, and have no idea why - so i used this as emu pin, and used cpu 6 as the 1st core of the VM. This definitely helped a bit.
 

Edited by snailbrain
Link to comment
  • 9 months later...

@snailbrain did you ever get completely to root cause? I have a 95% decent experience on my QEMU 4.1 i440fx Windows 10 VM, but I am getting audio glitching somewhat frequently through my NVIDIA GTX 1060 HDMI audio passthrough similar to what you have in your video. It's really jarring, and kind of killing the enjoyable experience, so I want it gone! I've done most of the same configuration you have, except I'm still using a vdisk for OS loaded on an unassigned device and still using the virtual network interface. Your experience is the closest I've seen to what i have been experiencing and you've done a bunch more testing than I have ... though I'm getting close 😂.

 

Anything you can share would be great!

 

Thanks

-JesterEE

Link to comment
  • 10 months later...

I too have been battling similar latency issues for over 2 years now, I just call it the unRAID stutter... Mine is the worst when I write to my array without a cache drive, but dual parity, I believe that the parity calculations overwhelm the cpu and cause major stutter. This is an old link roughly showing my system: https://pcpartpicker.com/b/6vGG3C

This has been helpful, I just have one question. What is htep and where do I find it?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.