ChewbaccaBG
-
Posts
71 -
Joined
-
Last visited
Content Type
Profiles
Forums
Downloads
Store
Gallery
Bug Reports
Documentation
Landing
Posts posted by ChewbaccaBG
-
-
Well... I regret to say this but... without *ANY* tweaks at all, and with 8 less cores allocated to the Windows 10 VM (16 host cores, no pinning, no isolation etc) - it travels at WARP 12. The same config under unRAID yielded terrible performance...
Hopefully unRAID will catch up *soon*.
Now, onto the tweaks.
-
Well there seems to be some weird issues with unRAID ... I was expecting a "multitasking monster" Windows 10 VM on a Threadripper 3960x 32G ram, nv 1070 gpu, 1TB nvme... nomatter what I did --- it couldn't stand up to a 2015 laptop (core i7 4c4t). As soon as I start something that uses a lot of ram (6-8 GB) everything slows down. DPC rises... Tried everything I could think of w/o much (if any) success. The old laptop just blows the VM away like a sports car.
Not sure what the problem is... I'll try to run PVE and see if it's the same there.
-
Tried everything I could think of - performance was s**t, comparing to a 2015 laptop (4core2thread) ... the laptop was rocking it, the VM - not so much.
Decided to postpone the VM setup for now. Will try again in 6 months.
-
For some reason I actually got better performance without CPU isolation. Still testing.
-
8 hours ago, WashingtonMatt said:
More progress on the FreeBSD side. Patch attached in bug report.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241774#c35
It really sounds like i440fx should be working on unRAID...Yeah but won't i440fx have issues with passed-through NICs / GPUs ?
-
Finally - network works!
-
1 minute ago, WashingtonMatt said:
I believe the alias was added when I added a second e1000 interface for troubleshooting. It should work. At this point all I can recommend is start over and apply first patch and kernel without Netmap.That's how I started. No interface detected - install complete -> usb flash copied the patch, ran the commands exactly -> patched -> compiled -> reboot -> no networking.
Are you sure you are doing this on 12.1-RELEASE, and not on 12.0 or something else ?
-
9 minutes ago, WashingtonMatt said:
I have not tried the second patch, so my kernel is without NETMAP. Since networking is not configured during setup, you do have to manually set it up after the custom kernel is installed.
<interface type='bridge'> <mac address='52:54:00:70:6f:f4'/> <source bridge='br0'/> <target dev='vnet1'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface>
vtpcim0@pci0:1:0:0: class=0x020000 card=0x11001af4 chip=0x10411af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio network device' class = network subclass = ethernet
ifconfig lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6> ether 52:54:00:70:6f:f4 inet 192.168.0.12 netmask 0xffffff00 broadcast 192.168.0.255 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
There was already the same <interface> in the config, I've added the "alias" from your config - still no network adapter.
Are you doing this on a 12.1-RELEASE, or on other? When installing it won't see the virtio network adapter even if I select configure ipv4 and configure dhcp - it says "no network adapter".
Here's the current config:
<?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='4'> <name>FreeBSD 12.1 DEV</name> <uuid>adece69a-e678-9280-0e59-988c2d78d910</uuid> <metadata> <vmtemplate xmlns="unraid" name="FreeBSD" icon="freebsd.png" os="freebsd"/> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>16</vcpu> <cputune> <vcpupin vcpu='0' cpuset='6'/> <vcpupin vcpu='1' cpuset='30'/> <vcpupin vcpu='2' cpuset='7'/> <vcpupin vcpu='3' cpuset='31'/> <vcpupin vcpu='4' cpuset='8'/> <vcpupin vcpu='5' cpuset='32'/> <vcpupin vcpu='6' cpuset='9'/> <vcpupin vcpu='7' cpuset='33'/> <vcpupin vcpu='8' cpuset='10'/> <vcpupin vcpu='9' cpuset='34'/> <vcpupin vcpu='10' cpuset='11'/> <vcpupin vcpu='11' cpuset='35'/> <vcpupin vcpu='12' cpuset='12'/> <vcpupin vcpu='13' cpuset='36'/> <vcpupin vcpu='14' cpuset='13'/> <vcpupin vcpu='15' cpuset='37'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-4.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/adece69a-e678-9280-0e59-988c2d78d910_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='16' threads='1'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/FreeBSD 12.1 DEV/vdisk1.img' index='1'/> <backingStore/> <target dev='hdc' bus='sata'/> <boot order='1'/> <alias name='sata0-0-2'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xb'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xc'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <filesystem type='mount' accessmode='passthrough'> <source dir='/mnt/user/isos'/> <target dir='isos'/> <alias name='fs0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </filesystem> <interface type='bridge'> <mac address='52:54:00:33:19:b1'/> <source bridge='br0'/> <target dev='vnet1'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/1'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/1'> <source path='/dev/pts/1'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-4-FreeBSD 12.1 DEV/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x48' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom file='/boot/vBIOS/ASUS-ROG-STRIX-nVidia-GTX1050Ti-GP107-KVM.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x1a2c'/> <product id='0x2d23'/> <address bus='1' device='2'/> </source> <alias name='hostdev1'/> <address type='usb' bus='0' port='1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x8564'/> <product id='0x1000'/> <address bus='3' device='5'/> </source> <alias name='hostdev2'/> <address type='usb' bus='0' port='2'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain>
-
34 minutes ago, WashingtonMatt said:
Did you configure DHCP, or set up IP address? What's output of ifconfig and pciconf -lv ?
During the install - it doesn't detect the network interface, so I continue the install without it.
Compiled kernel with 2 patches - rebooted - no network (I'm not passing PCIe NICs, I'm trying to get the br0/vmbr0 working).
Here's the output of pciconf -lv:
-
19 hours ago, WashingtonMatt said:
It's pretty painless, and of course not vetted by anyone at this point.
- Download the 12.1 DVD ISO
- SATA disk
- Be sure to install SRC
- Apply patch per instructions
- Comment out NETMAP from kernel
- Build/Install custom kernel
- Done
And yes it's going to seriously suck to get a new VM built back to how i had it on 11.2 .
Edit: I should add that I used USB flash drive to copy patch to VM
Q35 4.2 OVMF - managed to install with a GPU attached.
Patched & compiled the kernel with device netmap disabled, rebooted - still no network through br0 or virbr0 - no network device detected at all.
Compiling with the 2nd patch added (netmap still disabled) - no network.
Compiling with netmap enabled + both patches -> compiled fine, still no vmbr0 / br0 networking devices detected.
-
Here's how I install FreeBSD on PVE:
@jonp It finds the VirtIO disk, finds the VirtIO network.
Not much else I can say about this... It just works.
And here's the config of the VM I just installed:
root@vh1:~# cat /etc/pve/nodes/vh2/qemu-server/107.conf
bios: ovmf
bootdisk: virtio0
cores: 4
cpu: host
efidisk0: zfspool_1:vm-107-disk-0,size=1M
ide2: local:iso/FreeBSD-12.1-RELEASE-amd64-disc1.iso,media=cdrom
machine: q35
memory: 1024
name: TESTBSD
net0: virtio=52:1E:B0:FC:4E:14,bridge=vmbr0,firewall=1
numa: 0
ostype: other
scsihw: virtio-scsi-single
smbios1: uuid=14e06ec5-9453-459c-89d2-ef36ae1c026e
sockets: 1
virtio0: local:107/vm-107-disk-0.raw,iothread=1,size=32G
vmgenid: 5300f588-c646-4c62-9913-a062563b7eae
root@vh1:~#(/etc/pve is a cluster filesystem so can be accessed from either node)
Also the running process:
root@vh2:~# ps fax |grep 107 |grep "kvm"
22649 ? Sl 0:21 /usr/bin/kvm -id 107 -name TESTBSD -chardev socket,id=qmp,path=/var/run/qemu-server/107.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/107.pid -daemonize -smbios type=1,uuid=14e06ec5-9453-459c-89d2-ef36ae1c026e -drive if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd -drive if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/zvol/zfspool_1/vm-107-disk-0 -smp 4,sockets=1,cores=4,maxcpus=4 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/107.vnc,password -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi -m 1024 -object iothread,id=iothread-virtio0 -device vmgenid,guid=5300f588-c646-4c62-9913-a062563b7eae -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device VGA,id=vga,bus=pcie.0,addr=0x1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:8478cb83180 -drive file=/var/lib/vz/template/iso/FreeBSD-12.1-RELEASE-amd64-disc1.iso,if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -drive file=/var/lib/vz/images/107/vm-107-disk-0.raw,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,iothread=iothread-virtio0,bootindex=100 -netdev type=tap,id=net0,ifname=tap107i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=52:1E:B0:FC:4E:14,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -machine type=q35+pve1
root@vh2:~#Here's the detected virtio devices:
Feel free to reproduce.
-
5 minutes ago, jonp said:
Does ProxMox have a GUI? If so, can you show me the screen(s) for your configured FreeBSD VM? Can you post those details here?
Sent from my Pixel 3 XL using Tapatalk
It has a gui the same way unRAID has a gui. I did show you a config of a FreeBSD VM running on pve. You can just install PVE somewhere and run a VM and test this yourself. I can take you through the process but I can tell you right now - I ran a FreeBSD VM from the first try, from the first time I used PVE. Just select Q35, everything else is.... well.. self-explanatory.
-
@jonp So... the big question - why was unRAID affected and PVE - unaffected? I spent few hours today scrolling through kernel / qemu kvm changes - I couldn't see a single thing related to bsd. Perhaps I'm wrong... Still, it's worth checking.
-
24 minutes ago, jonp said:
Meh, not to start a war of the OSes, but I don't think FreeBSD's major calling card is networking performance. Here's a study from 2019 showing FreeBSD holding its own, but not dominating Linux in any capacity: https://www.phoronix.com/scan.php?page=article&item=windows-linux-10gbe&num=4
And unless I'm mistaken, even Cisco UCS systems that support 40GBe+ connectivity don't even support FreeBSD.
The main benefit for FreeBSD over Linux is it's licensing model. CDDL allows developers to advance the platform without having to contribute any improvements to the code base back to the FreeBSD maintainers (if they don't want to). This is why so many companies make hardware appliances that use FreeBSD for their base OS (they can fine tune their product for maximum performance and then charge a premium for their efforts without allowing someone to just rip off their code that they spent time creating). CDDL also allows for compatibility with Oracle's licensing on ZFS, which is why Oracle has never sued FreeNAS (and frankly I bet that was the deciding factor that led FreeNAS to use FreeBSD in the first place).
Don't let any of this come across as me negating your reasons for needing FreeBSD though. A job's a job and if you need it for your job, well, then you need it. That said, I can't imagine the FreeBSD developers can really afford to let this issue linger. It's not like Linux KVM is a small platform anymore and I have to imagine the vast majority of FreeBSD users are using it in a VM (not bare metal). I could be wrong.
I won't argue with you, I just wish to see unRAID being able to handle all the major OSes @ VM, and not just because I need FreeBSD As for the FreeBSD use, I think most of its use is on bare-metal, not VM.
-
1 hour ago, BRiT said:
And I hope I never need to. 😝
If your job needs it, you'll have to btw bsd not so evil There's a reason when performance is needed - freebsd is used, especially in networking. If you take the time - you'll see freebsd network stack being ported on god knows how many projects.
-
8 minutes ago, WashingtonMatt said:
It's pretty painless, and of course not vetted by anyone at this point.
- Download the 12.1 DVD ISO
- SATA disk
- Be sure to install SRC
- Apply patch per instructions
- Comment out NETMAP from kernel
- Build/Install custom kernel
- Done
And yes it's going to seriously suck to get a new VM built back to how i had it on 11.2 .
Edit: I should add that I used USB flash drive to copy patch to VM
Thank you man, I'll try this as soon as I get home.
-
6 minutes ago, BRiT said:
That just means you wont get updates, but depending on what you need to do a job, it still might be suitable, like if you only need an environment to write scripts that would be enough. But being likely large changes from 11.3 to 12.1 I can see that not being enough for production level.
Let me guess - you've never used FreeBSD, or at least not past it's EOL. When you do - you'll understand why this is important.
-
Just now, WashingtonMatt said:
For what it's worth, I was able to get 12.1 working with virtio NIC using TommyP's patch .
Can you provide more details on that? I really need to run a new VM *SOON* so I can do my job... I've seen the kernel patch but can you apply it on a 12.1 install, or do I have to install 11.2 and patch and update the source tree and patch again etc etc ?
-
11 minutes ago, BRiT said:
And you can't use a version of FreeBSD that worked before?
Regretfully - no. It's end of life, our production servers have already moved on 12.1. You can temporarily "get away with it" if you already had a VM running with everything installed, but now since EOL it's a pain in the *** to install anything (especially on a freshly installed VM). There's a "compat" port you can install on the newer servers that can use software compiled on an older version. But that is - if you've had a VM to work with. Right now I can install 11.2 but most of the ports I have to install won't install due to dependencies. If you know how the ports tree works on FreeBSD you'll understand.
-
@jonp please let your dev team know about the issue. If they can figure out why PVE doesn't have the same problem as unRAID, it'd be great.
I doubt we'll see any official action from the FreeBSD devs on this (at least anytime soon). Some of us really need FreeBSD working in order to do our jobs. I'm currently running a Trial unRAID and I have to decide whether or not to buy. If I can't do my job I won't have an incentive to get unRAID... and I really like it.
-
Try changing the QPI link to 6,400 MT/s after swapping the CPUs (1 on 1 off so you can POST), then add the 2nd CPU.
http://valid.x86.fr/x546z7 -- should work. Check the bios versions & etc from there.
-
-
8 hours ago, jonp said:
Been trying to get FreeBSD working in a VM with various configs/methods, but I'm failing to do so successfully. I'm wondering if Proxmox downgraded their kernel or quirked either it or QEMU to absolve themselves from this. In addition, are you sure on Proxmox the VM is loading via OVMF as opposed to a legacy BIOS? That might be another thing for us to try. In the end, I'm not sure if we're going to be able to do anything in the short term to resolve this.
I've pasted a VM config on the 1st page, it's OVMF. As for the changes - PVE is open source, I suppose it wouldn't be hard to search for any kernel/qemu changes...
-
Disabled mem-ballooning, added iothreads & emulatorpin, config @ 1st post updated.
Tried following some of the tips in:
https://mathiashueber.com/performance-tweaks-gaming-on-virtual-machines/
https://mathiashueber.com/qemu-troubleshooting-errors-gpu-passthrough-vm
Also, for some reason the task manager @ the windows guest shows "L1 Cache: N/A". Any ideas?
More testing tomorrow.
ASUS TRX40-PRO & Threadripper 3960x
in Motherboards and CPUs
Posted
I moved to Proxmox and all problems disappeared.