Crimson Unraider

I have had some issues with improper shutdowns and the server starting parity after every shutdown, then I had a cache drive fail so I replaced it and during parity I had another fail. I have done the following: 1. Started a new config, I preserved the Array but built the cache pool with new drives. I added my process pool back with no changes. 2. I reinstalled my dockers using the previous apps reinstall. 3. I had some freezing issues so I deleted the docker image and started over again. 4. After reinstalling all again it seemed to be fine for a couple days. Today, I started having issues with dockers running slow and refusing to stop (example SAB went from download speeds in the 200-300M range to 1-2K range) , I get a server error while trying to stop the container. I stopped the docker service and tried to restart it but it shows yes but also shows it is not running, next I tried to stop the array and restart but array won't stop because cache target is busy. This server has been running pretty solid for a few years now, I have recently upgraded my internet to 2.5G fiber and changed the router to accommodate 2.5G. My backup server is running perfect after this change. I'm having a hard time finding the issue with this one so any help would be greatly appreciated. Seems like after a reboot it is more stable but the dashboard takes a long time to load (all adblockers are disabled for that site) and I can't update the dockers because I get server errors trying to stop the dockers Thanks in advance Phil crimson-diagnostics-20230301-0032.zip

Seems like after a reboot it is more stable but the dashboard takes a long time to load (all adblockers are disabled for that site) and I can't update the dockers because I get server errors trying to stop the dockers

I have had some issues with improper shutdowns and the server starting parity after every shutdown, then I had a cache drive fail so I replaced it and during parity I had another fail. I have done the following: 1. Started a new config, I preserved the Array but built the cache pool with new drives. I added my process pool back with no changes. 2. I reinstalled my dockers using the previous apps reinstall. 3. I had some freezing issues so I deleted the docker image and started over again. 4. After reinstalling all again it seemed to be fine for a couple days. Today, I started having issues with dockers running slow and refusing to stop (example SAB went from download speeds in the 200-300M range to 1-2K range) , I get a server error while trying to stop the container. I stopped the docker service and tried to restart it but it shows yes but also shows it is not running, next I tried to stop the array and restart but array won't stop because cache target is busy. This server has been running pretty solid for a few years now, I have recently upgraded my internet to 2.5G fiber and changed the router to accommodate 2.5G. My backup server is running perfect after this change. I'm having a hard time finding the issue with this one so any help would be greatly appreciated. Thanks in advance Phil crimson-diagnostics-20230301-0032.zip

Jorge, I changed back to 1G nic for awile and I still had the same issue. Then I swatched back to the 10Gbe nic and changed the Docker custom network type -> ipvlan and it has been running pretty good for the last couple days. I will let it run like this for a few more days but I think that was the problem. Also, I just upgraded to 6.11.0 so that may have also been the fix. Thanks for the help, I'll let you know.

Ok I'll try it and let you know. Thanks for the quick response. It won't detect both lans because I disabled the 1G nic in the Bios. Can I configure with the 1G and the 10G on the same network? My switch is an unmanaged switch with 2 10G and 8 1G ports.

Hey guys, I could use a little help. I have been getting a lot of random issues in the last week, High CPU usage, all threads stuck 100%, Slow access to files, Plex locking up, other docker randomly freezing and I get Server Error (no code) when I try to stop or restart the docker. When I reboot the server using the reboot button it takes more than 20 min and when it comes back up it starts a Parity check due to unclean shut down. I stopped docker and ran mover, but mover is really slow also. The only warnings I get from fix common problems is my Network adapters are configured with a non-standard MTU of 9014. I have a 10Gbs nic installed and onboard nic disabled but it gives the warning for both eth0 and eth1. Only eth0 shows on network settings. I tried downgrading to 6.10.2 but it didn't help. VM seems to be fine. My diagnostics are attached but it is after restart due to frozen GUI. crimson-diagnostics-20220917-0138.zip

Woke up today and everything is working. Strange?

I set up a pfsense router and now I can't access Unraid Dashboard. I booted in GUI mode, changed ip address to 10.xx.xx.xx to match the router ip scheme. Restarted and now I can't get to dashboard from network or GUI Mode. Tried starting in GUI Safe mode but just get black screen with cursor blink. In normal boot and GUI mode, I can access all shares from the network using the new ip, all autostart dockers and vms are working but I have no way of accessing the Dashboard to control anything. Following another thread I pulled the USB and copied all files then deleted the config/ssh and ssl folders, rebooted and still no luck. I also tried editing the network.cfg to old ip and swapped back to the old router and still had the same issue. I can access the internet from the unraid GUI. Any assistance would be greatly appreciated. crimson-diagnostics-20220228-1600.zip

When I tried to update the Bios before I tried to use the 6900 the update failed and it bricked the motherboard. I had to buy an eprom programmer to get it back so I'm reluctant to try version F3a again. I've had no issues with Nvidia, the 127 error only started after adding the 6900 so I'm staying with Nvidia for now. Thanks for your assistance.

It seems like I spoke too soon. Started having issues again while playing Far Cry 6 and Doom Eternal had crashing and then getting the 127 error while trying to restart VM, so I tried a bunch of stuff including fresh Windows installs and upgrading Unraid to Version: 6.10.0-rc2 and getting AMD Vender Reset Plugin. Nothing worked, I went back to my GTX 1660 TI and still had issues until I did another Windows install. Seems to be stable now, been playing Far Cry for a couple days with no crashing. I returned the RX 6900 XT and have a RTX 3070 coming (not ideal but can't afford anything better with today's price gouging), I'm going to stay away from AMD GPU, I had issues years ago on bare metal Windows and switched the Nvidia, probably should have stayed.

Yes, no issues, this is my XML. <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='1'> <name>Windows 11</name> <uuid>6ca3b9c9-ee9d-a847-5843-230278bbd541</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>32505856</memory> <currentMemory unit='KiB'>32505856</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>16</vcpu> <cputune> <vcpupin vcpu='0' cpuset='8'/> <vcpupin vcpu='1' cpuset='24'/> <vcpupin vcpu='2' cpuset='9'/> <vcpupin vcpu='3' cpuset='25'/> <vcpupin vcpu='4' cpuset='10'/> <vcpupin vcpu='5' cpuset='26'/> <vcpupin vcpu='6' cpuset='11'/> <vcpupin vcpu='7' cpuset='27'/> <vcpupin vcpu='8' cpuset='12'/> <vcpupin vcpu='9' cpuset='28'/> <vcpupin vcpu='10' cpuset='13'/> <vcpupin vcpu='11' cpuset='29'/> <vcpupin vcpu='12' cpuset='14'/> <vcpupin vcpu='13' cpuset='30'/> <vcpupin vcpu='14' cpuset='15'/> <vcpupin vcpu='15' cpuset='31'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-5.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/6ca3b9c9-ee9d-a847-5843-230278bbd541_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='8' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso' index='2'/> <backingStore/> <target dev='hdb' bus='sata'/> <readonly/> <alias name='sata0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/Windows 11/spaces_win_clover.img' index='1'/> <backingStore/> <target dev='hdc' bus='sata'/> <boot order='1'/> <alias name='sata0-0-2'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xb'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xc'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xd'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xe'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xf'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='9' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:e8:0e:f9'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio-net'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-Windows 11/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom file='/mnt/user/isos/vbios/Powercolor.RX6900XT.16384.201214.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0a' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x41' slot='0x00' function='0x0'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x43' slot='0x00' function='0x0'/> </source> <alias name='hostdev3'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain>

I have had the VM running with the new config for about 12 hours now and it seems stable. Thanks for your assistance, I will try playing Far Cry 6 after work to see if I still have a problem.

Ghost, 1st, thanks for the quick response and advice. I made the adjustments you suggested and added the USB controller card. This is my log now, do you see any other issues? I had to go in and edit the xml every time I saved because the pcie keeps changing the audio to bus 5. I will do some testing tonight and see if it is sorted. -nodefaults \ -chardev socket,id=charmonitor,fd=32,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=localtime \ -no-hpet \ -no-shutdown \ -boot strict=on \ -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -device pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \ -device pcie-root-port,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 \ -device pcie-root-port,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x4 \ -device pcie-root-port,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x5 \ -device pcie-root-port,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x6 \ -device pcie-root-port,port=0xf,chassis=8,id=pci.8,bus=pcie.0,addr=0x1.0x7 \ -device pcie-pci-bridge,id=pci.9,bus=pci.1,addr=0x0 \ -device qemu-xhci,p2=15,p3=15,id=usb,bus=pcie.0,addr=0x7 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \ -blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.190-1.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \ -device ide-cd,bus=ide.1,drive=libvirt-2-format,id=sata0-0-1 \ -blockdev '{"driver":"file","filename":"/mnt/user/domains/Windows 11/spaces_win_clover.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \ -device ide-hd,bus=ide.2,drive=libvirt-1-format,id=sata0-0-2,bootindex=1,write-cache=on \ -netdev tap,fd=34,id=hostnet0 \ -device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:e8:0e:f9,bus=pci.3,addr=0x0 \ -chardev pty,id=charserial0 \ -device isa-serial,chardev=charserial0,id=serial0 \ -chardev socket,id=charchannel0,fd=35,server,nowait \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \ -device usb-tablet,id=input0,bus=usb.0,port=1 \ -device vfio-pci,host=0000:0a:00.0,id=hostdev0,bus=pci.4,multifunction=on,addr=0x0,romfile=/mnt/user/isos/vbios/Powercolor.RX6900XT.16384.201214.rom \ -device vfio-pci,host=0000:0a:00.1,id=hostdev1,bus=pci.4,addr=0x0.0x1 \ -device vfio-pci,host=0000:41:00.0,id=hostdev2,bus=pci.6,addr=0x0 \ -device vfio-pci,host=0000:43:00.0,id=hostdev3,bus=pci.7,addr=0x0 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on 2021-12-22 22:00:49.913+0000: Domain id=4 is tainted: high-privileges 2021-12-22 22:00:49.913+0000: Domain id=4 is tainted: host-cpu char device redirected to /dev/pts/0 (label charserial0) crimson-diagnostics-20211222-1707.zip

Ok I will look at those, I have been running the USB controller like that and noticed the error but it has been working so I didn't change it. I can't passthrough the sata so I will add a pcie usb and passthrough that instead of passing the motherboard usb controller. I added the RX 6900 XL to the vfio now and rebooted, I ran the diagnostics after the libvirt crashed so there were no XML's, see attached (this one is after a reboot). crimson-diagnostics-20211222-0701.zip

All, I could use a little help, My system has been very stable until I recently replaced my GTX 1660 TI with a RX 6900 XT for use with my Window 11 VM. I setup the VM as a Q35 machine, all runs well for awhile but since the change I have been experiencing random crashes on my VM mainly during gaming and every time it happens it only affects the VM, Dockers still run fine. When I go to VM manager and stop it, I am unable to restart it until I reboot the server. I had some issues getting it going at first but then I followed Zeus83's thread about passthrough hints (all except, I had to keep CSM enabled to use my NVME on the VM, and I only passthrough two items, the GPU and Sound) and I got it working. Any assistance would be greatly appreciated, Diagnostics are attached. crimson-diagnostics-20211221-1956.zip

John_M, Thanks for the help, all my problems were linked to the failing CPU cooler. I had to wait on parts but I went all in on water cooling kit and now I'm averaging 37 C and all is running fine. 🤪

I found the reason for the shutdowns. My cpu cooler has failed and my cpu os overheating. I brought the PC up in the bios, after about 10 min it shut down. I noticed the cpu temp was 93 c.

I opened the log as soon as I could at boot and captured this before it shut off again.

It just shut down again

So, another update, the nvme I put in was reporting 57 degrees C. When I clicked on it to see the info page the server turned off. I removed the nvme and restarted, the system started a parity check due to unclean shutdown but I stopped it until after troubleshooting. I don't want it to crash in the middle of parity. When I pulled the nvme it was warm to touch but not hot. When I removed the nvme I put the samsung ssd in the main cache pool. I moved it earlier to try to separate libvrt from the other traffic when the slow down first started, that didn't help so I went back to one pool. I also noticed that it is taking more than 5 min to boot up now. crimson-diagnostics-20210405-0846.zip

Ok I changed the cables on all 4 of the Kingston SSDs, see attached Diagnostics after boot. Also, my ssds are all in an icy dock, I checked the power connectors but there are only two feeding the 6 ssds. Also, while I had it open I added another nvme for the second cache. crimson-diagnostics-20210405-0713.zip

John, I'm ready to get started on this again, I'm trying to find which drive has the cable problem, I have 4 of those drives in the pool. I'm pretty new to Unraid and I'm just curious where you found that.

So I stubbed the 1660 and the Sanity check errors stopped. But it is still really slow and locking up. I think it might be network related, task manager keeps showing "System Interrupts" excessive cpu usage when the freezing happens. I googled it and "System Interrupts" usually means a hardware issue and most say it is likely caused by a nic or external device. The only things plugged in is usb keyboard/mouse and an Xbox controller wireless adapter. When I disable the windows network adapter I see less system interrupts but now I can't play most of my games. I think I need to walk away for the night.

I have 2 GPU's, I use my GTX 1070 with Plex and Emby. Can I stub the 1660 in vfio for use with my VM and keep the Nvidia driver for the 1070?

I have 2 Windows VMs, one on cache drive and on on nvme (followed Space Invaders's guide), I normally only use the nvme but I left the cache one on for testing. Both worked fine in 6.8.3 and both are extremely slow after upgrade. I noticed in task manager the "System Interrupts" was using over 60% of my CPU randomly on both VMs. It is setup as a gaming VM using Nvidia 1660 TI with 16G of ram, I thought that the GPU passthrough may be the problem so I tried VNC and it was still slow. I noticed this error while running GPU statistics plugin repeating and filling my log file so I uninstalled the plugin and the error stopped. I wasn't able to get the diagnostics after the random crashes but I did get the attached after running a VM. My system log shows these errors after I start the VM (the errors repeat until I shut down the VM). I checked my Bios and it is the newest, HVM and IOMMU are Enabled M/B: Gigabyte Technology Co., Ltd. X399 AORUS PRO-CF Version Default string BIOS: American Megatrends Inc. Version F2. Dated: 12/11/2019 CPU: AMD Ryzen Threadripper 2950X 16-Core @ 3500 MHz Memory: 128 GiB DDR4 (max. installable capacity 512 GiB) crimson-diagnostics-20210403-1538.zip