jaiinus

Issue: Minecraft server has extremely poor performance (8-12TPS) with 7 players on a fresh world that's largely pregenerated (5,000x5,000 pregenerated). Suspect configuration issues with Ubuntu 20.04 VM. Unraid Hardware: Ryzen 2700x on MSI B450 A-Pro (4 cores/8thread pinned) Sabrent Rocket PCIE3.0 1TB NVME SSD (firmware RKT303.2) 48GB RAM (26gb allocated to VM) Verizon FIOS 100/100 connection Software: Latest (6.8.3) Unraid with QEMU 4.2.0 Ubuntu 20.04 LTS and Minecraft 1.16.4 running Paper (minimal mods i.e. world border/coreprotect, performance tweaked) Debugging Images (/proc/cpuinfo difference between host/guest, disk IO test from windows guest, timing report from minecraft): https://imgur.com/a/Tg63GL0 Java launch options java -Xms20G -Xmx20G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=200 -XX:+UnlockExperimentalVMOptions -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=40 -XX:G1HeapRegionSize=8M -XX:G1ReservePercent=20 -XX:G1HeapWastePercent=5 -XX:G1MixedGCCountTarget=4 -XX:InitiatingHeapOccupancyPercent=15 -XX:G1MixedGCLiveThresholdPercent=90 -XX:G1RSetUpdatingPauseTimePercent=5 -XX:SurvivorRatio=32 -XX:+PerfDisableSharedMem -XX:MaxTenuringThreshold=1 -Dusing.aikars.flags=https://mcflags.emc.gs -Daikars.new.flags=true -jar paper.jar nogui VM config below (I added 2 cores/4threads but doubt it will affect lag problems): <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='25'> <name>Ajax Minecraft</name> <uuid>57bee4d5-05f1-0962-f6f3-76f245013119</uuid> <metadata> <vmtemplate xmlns="unraid" name="Ubuntu" icon="ubuntu.png" os="ubuntu"/> </metadata> <memory unit='KiB'>27262976</memory> <currentMemory unit='KiB'>27262976</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>12</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='10'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='11'/> <vcpupin vcpu='4' cpuset='4'/> <vcpupin vcpu='5' cpuset='12'/> <vcpupin vcpu='6' cpuset='5'/> <vcpupin vcpu='7' cpuset='13'/> <vcpupin vcpu='8' cpuset='6'/> <vcpupin vcpu='9' cpuset='14'/> <vcpupin vcpu='10' cpuset='7'/> <vcpupin vcpu='11' cpuset='15'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-4.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/57bee4d5-05f1-0962-f6f3-76f245013119_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='6' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/Ubuntu/vdisk1.img' index='1'/> <backingStore/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </disk> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x12'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x14'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:b6:7c:73'/> <source bridge='br0'/> <target dev='vnet1'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/1'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/1'> <source path='/dev/pts/1'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-25-Ajax Minecraft/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <graphics type='vnc' port='5901' autoport='yes' websocket='5701' listen='0.0.0.0' keymap='en-us'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain> Disk speed test below from guest Ubuntu 20.04 (compare stats with https://www.legitreviews.com/sabrent-rocket-1tb-ssd-review-tlc-nand-flash_217101/4): fio --loops=3 --size=100m --filename=/home/fiotest.tmp --stonewall --ioengine=libaio --direct=1 \ --name=Seqread --bs=1m --rw=read \ --name=Seqwrite --bs=1m --rw=write \ --name=512Kread --bs=512k --rw=randread \ --name=512Kwrite --bs=512k --rw=randwrite \ --name=4kQD32read --bs=4k --iodepth=32 --rw=randread \ --name=4kQD32write --bs=4k --iodepth=32 --rw=randwrite Seqread: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 Seqwrite: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 512Kread: (g=2): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1 512Kwrite: (g=3): rw=randwrite, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1 4kQD32read: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 4kQD32write: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 fio-3.16 Starting 6 processes Seqread: (groupid=0, jobs=1): err= 0: pid=2638: Sat Dec 19 18:15:11 2020 read: IOPS=2307, BW=2308MiB/s (2420MB/s)(300MiB/130msec) slat (usec): min=63, max=433, avg=116.73, stdev=55.16 clat (usec): min=6, max=630, avg=252.30, stdev=83.59 lat (usec): min=302, max=865, avg=411.92, stdev=61.76 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 88], 10.00th=[ 106], 20.00th=[ 196], | 30.00th=[ 239], 40.00th=[ 255], 50.00th=[ 269], 60.00th=[ 281], | 70.00th=[ 293], 80.00th=[ 310], 90.00th=[ 338], 95.00th=[ 367], | 99.00th=[ 392], 99.50th=[ 441], 99.90th=[ 627], 99.95th=[ 627], | 99.99th=[ 627] lat (usec) : 10=0.67%, 100=7.33%, 250=27.33%, 500=64.33%, 750=0.33% cpu : usr=0.00%, sys=48.06%, ctx=299, majf=0, minf=269 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=300,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Seqwrite: (groupid=1, jobs=1): err= 0: pid=2639: Sat Dec 19 18:15:11 2020 write: IOPS=1675, BW=1676MiB/s (1757MB/s)(300MiB/179msec); 0 zone resets slat (usec): min=125, max=438, avg=192.45, stdev=31.50 clat (usec): min=245, max=1822, avg=373.94, stdev=103.29 lat (usec): min=438, max=2029, avg=576.84, stdev=106.32 clat percentiles (usec): | 1.00th=[ 253], 5.00th=[ 281], 10.00th=[ 302], 20.00th=[ 322], | 30.00th=[ 343], 40.00th=[ 355], 50.00th=[ 363], 60.00th=[ 379], | 70.00th=[ 392], 80.00th=[ 408], 90.00th=[ 433], 95.00th=[ 457], | 99.00th=[ 652], 99.50th=[ 660], 99.90th=[ 1827], 99.95th=[ 1827], | 99.99th=[ 1827] lat (usec) : 250=0.67%, 500=96.67%, 750=2.33% lat (msec) : 2=0.33% cpu : usr=5.06%, sys=41.57%, ctx=301, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,300,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 512Kread: (groupid=2, jobs=1): err= 0: pid=2640: Sat Dec 19 18:15:11 2020 read: IOPS=2816, BW=1408MiB/s (1477MB/s)(300MiB/213msec) slat (usec): min=39, max=1904, avg=68.17, stdev=79.36 clat (usec): min=8, max=7372, avg=251.34, stdev=424.70 lat (usec): min=212, max=7467, avg=335.02, stdev=461.42 clat percentiles (usec): | 1.00th=[ 133], 5.00th=[ 159], 10.00th=[ 176], 20.00th=[ 190], | 30.00th=[ 198], 40.00th=[ 204], 50.00th=[ 212], 60.00th=[ 219], | 70.00th=[ 227], 80.00th=[ 235], 90.00th=[ 253], 95.00th=[ 277], | 99.00th=[ 1565], 99.50th=[ 1958], 99.90th=[ 7373], 99.95th=[ 7373], | 99.99th=[ 7373] lat (usec) : 10=0.17%, 100=0.17%, 250=88.50%, 500=9.83% lat (msec) : 2=0.83%, 4=0.17%, 10=0.33% cpu : usr=5.19%, sys=36.32%, ctx=600, majf=0, minf=140 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=600,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 512Kwrite: (groupid=3, jobs=1): err= 0: pid=2641: Sat Dec 19 18:15:11 2020 write: IOPS=2608, BW=1304MiB/s (1368MB/s)(300MiB/230msec); 0 zone resets slat (usec): min=62, max=198, avg=95.80, stdev=15.92 clat (usec): min=6, max=5299, avg=252.15, stdev=212.23 lat (usec): min=281, max=5394, avg=366.47, stdev=209.98 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 194], 10.00th=[ 204], 20.00th=[ 219], | 30.00th=[ 231], 40.00th=[ 239], 50.00th=[ 247], 60.00th=[ 253], | 70.00th=[ 260], 80.00th=[ 269], 90.00th=[ 281], 95.00th=[ 293], | 99.00th=[ 334], 99.50th=[ 371], 99.90th=[ 5276], 99.95th=[ 5276], | 99.99th=[ 5276] lat (usec) : 10=0.33%, 100=1.33%, 250=53.83%, 500=44.17% lat (msec) : 2=0.17%, 10=0.17% cpu : usr=7.42%, sys=33.19%, ctx=601, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,600,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 4kQD32read: (groupid=4, jobs=1): err= 0: pid=2642: Sat Dec 19 18:15:11 2020 read: IOPS=10.8k, BW=42.3MiB/s (44.3MB/s)(300MiB/7094msec) slat (usec): min=23, max=1966, avg=43.91, stdev=22.55 clat (usec): min=91, max=7795, avg=2877.85, stdev=316.86 lat (usec): min=127, max=7845, avg=2927.08, stdev=321.19 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2507], 10.00th=[ 2573], 20.00th=[ 2638], | 30.00th=[ 2737], 40.00th=[ 2802], 50.00th=[ 2835], 60.00th=[ 2868], | 70.00th=[ 2933], 80.00th=[ 3064], 90.00th=[ 3228], 95.00th=[ 3359], | 99.00th=[ 3916], 99.50th=[ 4293], 99.90th=[ 6063], 99.95th=[ 6456], | 99.99th=[ 7701] bw ( KiB/s): min=27470, max=42524, per=69.89%, avg=30267.00, stdev=3827.95, samples=13 iops : min= 6867, max=10631, avg=7566.23, stdev=957.12, samples=13 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.06%, 4=98.97%, 10=0.92% cpu : usr=3.14%, sys=96.76%, ctx=202, majf=0, minf=42 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=76800,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 4kQD32write: (groupid=5, jobs=1): err= 0: pid=2643: Sat Dec 19 18:15:11 2020 write: IOPS=10.8k, BW=42.3MiB/s (44.4MB/s)(300MiB/7091msec); 0 zone resets slat (usec): min=24, max=4689, avg=46.08, stdev=28.13 clat (usec): min=98, max=8286, avg=2877.20, stdev=338.32 lat (usec): min=143, max=8339, avg=2928.69, stdev=342.57 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2507], 10.00th=[ 2573], 20.00th=[ 2638], | 30.00th=[ 2737], 40.00th=[ 2802], 50.00th=[ 2868], 60.00th=[ 2868], | 70.00th=[ 2933], 80.00th=[ 2999], 90.00th=[ 3195], 95.00th=[ 3359], | 99.00th=[ 4080], 99.50th=[ 4490], 99.90th=[ 7046], 99.95th=[ 7504], | 99.99th=[ 8225] bw ( KiB/s): min=41104, max=46488, per=99.94%, avg=43296.00, stdev=1566.64, samples=14 iops : min=10276, max=11622, avg=10824.00, stdev=391.66, samples=14 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.05%, 4=98.78%, 10=1.14% cpu : usr=2.64%, sys=97.32%, ctx=155, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=0,76800,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): READ: bw=2308MiB/s (2420MB/s), 2308MiB/s-2308MiB/s (2420MB/s-2420MB/s), io=300MiB (315MB), run=130-130msec Run status group 1 (all jobs): WRITE: bw=1676MiB/s (1757MB/s), 1676MiB/s-1676MiB/s (1757MB/s-1757MB/s), io=300MiB (315MB), run=179-179msec Run status group 2 (all jobs): READ: bw=1408MiB/s (1477MB/s), 1408MiB/s-1408MiB/s (1477MB/s-1477MB/s), io=300MiB (315MB), run=213-213msec Run status group 3 (all jobs): WRITE: bw=1304MiB/s (1368MB/s), 1304MiB/s-1304MiB/s (1368MB/s-1368MB/s), io=300MiB (315MB), run=230-230msec Run status group 4 (all jobs): READ: bw=42.3MiB/s (44.3MB/s), 42.3MiB/s-42.3MiB/s (44.3MB/s-44.3MB/s), io=300MiB (315MB), run=7094-7094msec Run status group 5 (all jobs): WRITE: bw=42.3MiB/s (44.4MB/s), 42.3MiB/s-42.3MiB/s (44.4MB/s-44.4MB/s), io=300MiB (315MB), run=7091-7091msec Disk stats (read/write): dm-0: ios=77700/77535, merge=0/0, ticks=7868/7892, in_queue=15760, util=90.60%, aggrios=78000/78906, aggrmerge=0/7, aggrticks=7353/7813, aggrin_queue=216, aggrutil=90.30% vda: ios=78000/78906, merge=0/7, ticks=7353/7813, in_queue=216, util=90.30% Long form: I'm hosting for a friend and community I'm in. The configuration for the java/minecraft server was ported over from an ubuntu system running on rented xeon hardware with much worse single core performance than what my 2700x should be giving. He turned off and cut down settings and the performance is still at 8-10 TPS on a FRESH world with a mostly pregenerated world (5,000^2 pregenerated). Performance improved a bit when we had the two guys trying to find world border stop but it's still way under what is expected considering the Xeon based server had large builds and multiple farms running 18-20TPS consistently with 5-7 players online. I suspect there's some configuration issue. I used the GUI to setup the guest OS and did not install any additional drivers. I can see /proc/cpuinfo on the Ubuntu guest shows all threads pinned at maximum, non-boosted speed (3.6GHZ) but the Host shows it boosting to the proper speed (4.3ghz) on whatever thread is getting slammed before it moves to another thread and boosts to 4.3GHZ. I can see on Unraid dashboard that HT is working as the maxed out thread sometimes is on the HT and not the "base" thread. I can also see that disk stats seem low from the guest OS? Even then the performance should be more than enough for minecraft at over 40MB/s random write/read.