ChewbaccaBG

I moved to Proxmox and all problems disappeared.

Well... I regret to say this but... without *ANY* tweaks at all, and with 8 less cores allocated to the Windows 10 VM (16 host cores, no pinning, no isolation etc) - it travels at WARP 12. The same config under unRAID yielded terrible performance... Hopefully unRAID will catch up *soon*. Now, onto the tweaks.

Well there seems to be some weird issues with unRAID ... I was expecting a "multitasking monster" Windows 10 VM on a Threadripper 3960x 32G ram, nv 1070 gpu, 1TB nvme... nomatter what I did --- it couldn't stand up to a 2015 laptop (core i7 4c4t). As soon as I start something that uses a lot of ram (6-8 GB) everything slows down. DPC rises... Tried everything I could think of w/o much (if any) success. The old laptop just blows the VM away like a sports car. Not sure what the problem is... I'll try to run PVE and see if it's the same there.

Tried everything I could think of - performance was s**t, comparing to a 2015 laptop (4core2thread) ... the laptop was rocking it, the VM - not so much. Decided to postpone the VM setup for now. Will try again in 6 months.

For some reason I actually got better performance without CPU isolation. Still testing.

Yeah but won't i440fx have issues with passed-through NICs / GPUs ?

Finally - network works!

That's how I started. No interface detected - install complete -> usb flash copied the patch, ran the commands exactly -> patched -> compiled -> reboot -> no networking. Are you sure you are doing this on 12.1-RELEASE, and not on 12.0 or something else ?

There was already the same <interface> in the config, I've added the "alias" from your config - still no network adapter. Are you doing this on a 12.1-RELEASE, or on other? When installing it won't see the virtio network adapter even if I select configure ipv4 and configure dhcp - it says "no network adapter". Here's the current config: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='4'> <name>FreeBSD 12.1 DEV</name> <uuid>adece69a-e678-9280-0e59-988c2d78d910</uuid> <metadata> <vmtemplate xmlns="unraid" name="FreeBSD" icon="freebsd.png" os="freebsd"/> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>16</vcpu> <cputune> <vcpupin vcpu='0' cpuset='6'/> <vcpupin vcpu='1' cpuset='30'/> <vcpupin vcpu='2' cpuset='7'/> <vcpupin vcpu='3' cpuset='31'/> <vcpupin vcpu='4' cpuset='8'/> <vcpupin vcpu='5' cpuset='32'/> <vcpupin vcpu='6' cpuset='9'/> <vcpupin vcpu='7' cpuset='33'/> <vcpupin vcpu='8' cpuset='10'/> <vcpupin vcpu='9' cpuset='34'/> <vcpupin vcpu='10' cpuset='11'/> <vcpupin vcpu='11' cpuset='35'/> <vcpupin vcpu='12' cpuset='12'/> <vcpupin vcpu='13' cpuset='36'/> <vcpupin vcpu='14' cpuset='13'/> <vcpupin vcpu='15' cpuset='37'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-4.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/adece69a-e678-9280-0e59-988c2d78d910_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='16' threads='1'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/FreeBSD 12.1 DEV/vdisk1.img' index='1'/> <backingStore/> <target dev='hdc' bus='sata'/> <boot order='1'/> <alias name='sata0-0-2'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xb'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xc'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <filesystem type='mount' accessmode='passthrough'> <source dir='/mnt/user/isos'/> <target dir='isos'/> <alias name='fs0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </filesystem> <interface type='bridge'> <mac address='52:54:00:33:19:b1'/> <source bridge='br0'/> <target dev='vnet1'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/1'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/1'> <source path='/dev/pts/1'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-4-FreeBSD 12.1 DEV/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x48' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom file='/boot/vBIOS/ASUS-ROG-STRIX-nVidia-GTX1050Ti-GP107-KVM.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x1a2c'/> <product id='0x2d23'/> <address bus='1' device='2'/> </source> <alias name='hostdev1'/> <address type='usb' bus='0' port='1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x8564'/> <product id='0x1000'/> <address bus='3' device='5'/> </source> <alias name='hostdev2'/> <address type='usb' bus='0' port='2'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain>

During the install - it doesn't detect the network interface, so I continue the install without it. Compiled kernel with 2 patches - rebooted - no network (I'm not passing PCIe NICs, I'm trying to get the br0/vmbr0 working). Here's the output of pciconf -lv:

Q35 4.2 OVMF - managed to install with a GPU attached. Patched & compiled the kernel with device netmap disabled, rebooted - still no network through br0 or virbr0 - no network device detected at all. Compiling with the 2nd patch added (netmap still disabled) - no network. Compiling with netmap enabled + both patches -> compiled fine, still no vmbr0 / br0 networking devices detected.

Here's how I install FreeBSD on PVE: @jonp It finds the VirtIO disk, finds the VirtIO network. Not much else I can say about this... It just works. And here's the config of the VM I just installed: root@vh1:~# cat /etc/pve/nodes/vh2/qemu-server/107.conf bios: ovmf bootdisk: virtio0 cores: 4 cpu: host efidisk0: zfspool_1:vm-107-disk-0,size=1M ide2: local:iso/FreeBSD-12.1-RELEASE-amd64-disc1.iso,media=cdrom machine: q35 memory: 1024 name: TESTBSD net0: virtio=52:1E:B0:FC:4E:14,bridge=vmbr0,firewall=1 numa: 0 ostype: other scsihw: virtio-scsi-single smbios1: uuid=14e06ec5-9453-459c-89d2-ef36ae1c026e sockets: 1 virtio0: local:107/vm-107-disk-0.raw,iothread=1,size=32G vmgenid: 5300f588-c646-4c62-9913-a062563b7eae root@vh1:~# (/etc/pve is a cluster filesystem so can be accessed from either node) Also the running process: root@vh2:~# ps fax |grep 107 |grep "kvm" 22649 ? Sl 0:21 /usr/bin/kvm -id 107 -name TESTBSD -chardev socket,id=qmp,path=/var/run/qemu-server/107.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/107.pid -daemonize -smbios type=1,uuid=14e06ec5-9453-459c-89d2-ef36ae1c026e -drive if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd -drive if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/zvol/zfspool_1/vm-107-disk-0 -smp 4,sockets=1,cores=4,maxcpus=4 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/107.vnc,password -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi -m 1024 -object iothread,id=iothread-virtio0 -device vmgenid,guid=5300f588-c646-4c62-9913-a062563b7eae -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device VGA,id=vga,bus=pcie.0,addr=0x1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:8478cb83180 -drive file=/var/lib/vz/template/iso/FreeBSD-12.1-RELEASE-amd64-disc1.iso,if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -drive file=/var/lib/vz/images/107/vm-107-disk-0.raw,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,iothread=iothread-virtio0,bootindex=100 -netdev type=tap,id=net0,ifname=tap107i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=52:1E:B0:FC:4E:14,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -machine type=q35+pve1 root@vh2:~# Here's the detected virtio devices: Feel free to reproduce.

It has a gui the same way unRAID has a gui. I did show you a config of a FreeBSD VM running on pve. You can just install PVE somewhere and run a VM and test this yourself. I can take you through the process but I can tell you right now - I ran a FreeBSD VM from the first try, from the first time I used PVE. Just select Q35, everything else is.... well.. self-explanatory.

@jonp So... the big question - why was unRAID affected and PVE - unaffected? I spent few hours today scrolling through kernel / qemu kvm changes - I couldn't see a single thing related to bsd. Perhaps I'm wrong... Still, it's worth checking.

I won't argue with you, I just wish to see unRAID being able to handle all the major OSes @ VM, and not just because I need FreeBSD As for the FreeBSD use, I think most of its use is on bare-metal, not VM.

If your job needs it, you'll have to btw bsd not so evil There's a reason when performance is needed - freebsd is used, especially in networking. If you take the time - you'll see freebsd network stack being ported on god knows how many projects.

Thank you man, I'll try this as soon as I get home.

Let me guess - you've never used FreeBSD, or at least not past it's EOL. When you do - you'll understand why this is important.

Can you provide more details on that? I really need to run a new VM *SOON* so I can do my job... I've seen the kernel patch but can you apply it on a 12.1 install, or do I have to install 11.2 and patch and update the source tree and patch again etc etc ?

Regretfully - no. It's end of life, our production servers have already moved on 12.1. You can temporarily "get away with it" if you already had a VM running with everything installed, but now since EOL it's a pain in the *** to install anything (especially on a freshly installed VM). There's a "compat" port you can install on the newer servers that can use software compiled on an older version. But that is - if you've had a VM to work with. Right now I can install 11.2 but most of the ports I have to install won't install due to dependencies. If you know how the ports tree works on FreeBSD you'll understand.

@jonp please let your dev team know about the issue. If they can figure out why PVE doesn't have the same problem as unRAID, it'd be great. I doubt we'll see any official action from the FreeBSD devs on this (at least anytime soon). Some of us really need FreeBSD working in order to do our jobs. I'm currently running a Trial unRAID and I have to decide whether or not to buy. If I can't do my job I won't have an incentive to get unRAID... and I really like it.

Try changing the QPI link to 6,400 MT/s after swapping the CPUs (1 on 1 off so you can POST), then add the 2nd CPU. http://valid.x86.fr/x546z7 -- should work. Check the bios versions & etc from there.

msconfig -> boot -> advanced options -> Number of processors set to 24 (total provided to the VM). Page file -> disabled MSI enabled on: More tweaks to come soon, I'm still reading a ton of posts on reddit and the libvirt manuals, testing and getting some feedback from LatencyMon.

I've pasted a VM config on the 1st page, it's OVMF. As for the changes - PVE is open source, I suppose it wouldn't be hard to search for any kernel/qemu changes... https://git.proxmox.com/

Disabled mem-ballooning, added iothreads & emulatorpin, config @ 1st post updated. Tried following some of the tips in: https://mathiashueber.com/performance-tweaks-gaming-on-virtual-machines/ https://mathiashueber.com/qemu-troubleshooting-errors-gpu-passthrough-vm Also, for some reason the task manager @ the windows guest shows "L1 Cache: N/A". Any ideas? More testing tomorrow.