March 29, 20251 yr Hi Guys, I'm finally giving up trying to solve my problem on my own. I can't figure out what's happening and hope you guys can help me. My main issue is continuous and random freezes of my UNRAID server (7.0.0). It happens anywhere between right after a reboot up to 17 hours of runtime. I think it may have something to do with my Ubuntu VM, but more on that later. First, I’d like to introduce my system. I initially set up UNRAID on an i7 7700k with 4x 8GB DDR4 RAM modules, no GPU, and the following drives: Array Drive: SAMSUNG_MZNTD128HAGM-00000_S15YNYAD903274 (sdf) Parity Drive: Samsung_SSD_850_EVO_250GB_S2R6NX0J711276K (sde) Pool Drives: ST12000NM0127_ZJV5H5P2 – 12 TB (sdh) Samsung_Flash_Drive_FIT_0321922080002714-0:0 – 128 GB (sdb) Samsung_SSD_850_EVO_500GB_S2RBNX0J511298P (sdg) Samsung_SSD_960_EVO_500GB_S3EUNX0J126088A (nvme1n1) Flash Drive: Crappy 1GB No Name USB 2.0 Stick As for software, I was running an Ubuntu Server 24.10 VM, with the vdisk stored on the two Samsung SSDs in the pool. This Ubuntu Server runs CapRover. CapRover is a SaaS platform based on Docker, so I’m running multiple Docker containers there for web hosting and app development. It’s the only VM I have. Additionally, I run some Docker apps natively via UNRAID: AdGuard Home, JDownloader2, Nginx Proxy Manager, Plex Media Server, and three game servers (Enshrouded, Satisfactory, and Valheim). However, the game servers have been stopped for testing and haven’t been running since. As for plugins, I’ve only installed a few basic ones like: Backup/Restore Appdata, Fix Common Problems, Tips and Tweaks, Unassigned Devices, User Scripts, VM Backup, rclone, and of course, Community Applications. That was my initial setup — which already caused the random freezes. Then I started trying to figure out what’s going on. I read that I should set up a Syslog server to catch logs if a kernel panic happens. I did that, but no kernel panics were ever logged. Then I read that it should log them, so I thought: "Maybe there's an issue with the drives or something..." — so I set up a Syslog server on my Home Assistant instance running natively on a Raspberry Pi. Everything is being logged — except kernel panics. It just stops at the last regular log line and then continues again after a reboot. Then I thought: "Okay, maybe it’s just old hardware. Maybe I'm running too many things. Also, I’d like to run those game servers efficiently, and Plex 4K transcoding is also pretty heavy on performance. Let’s upgrade this thing." So I went out and grabbed some new gear. And here’s the current setup: CPU: Intel i5 13600k Mainboard: MSI MEG Z690 UNIFY RAM: 2x 16GB DDR5 6000 GPU: GeForce RTX 4700 12GB Drives (some reused from the old setup): Array Drive: Lexar_SSD_NM620_1TB_ND4342R003031P1110 – 1 TB (nvme2n1) Pool Drives: ST3000VN000-1HJ166_W6A17W07 – 3 TB (sdc) Samsung_SSD_870_QVO_1TB_S5RRNF0T454321J – 1 TB (sdd) ST12000NM0127_ZJV5H5P2 – 12 TB (sdh) Samsung_Flash_Drive_FIT_0321922080002714-0:0 – 128 GB (sdb) Flash Drive: Flash_Drive_FIT – 64.2 GB (sda) (new Samsung FIT Plus USB 3.1 64GB Stick, MUF-64AB/APC) Unmounted Drive: WD_BLACK_SN850X_1000GB_23051C802731 (nvme0n1) Some of the old drives are currently not in use. Along with the hardware change, I also changed some assignments. The UNRAID server now runs from the new USB stick, which I restored via “Flash Backup” and then upgraded to 7.0.1. I also changed the array drive — the old one was connected through a broken SATA-to-NVME adapter (the plastic part of the connector broke, but the contacts were still fine). Here’s what I did: Removed the parity drive Added the new Lexar drive Copied files via UNRAID shares Removed the old Samsung NVME Since then, I’ve been having issues where the UNRAID web UI won’t come up after a reboot. I always need to restart the UNRAID nginx manually to get it back — but that’s another story. I also moved the Ubuntu VM: Removed both 500GB Samsung SSDs Added a single Samsung SSD as a pool device Copied the VM content via rclone Adjusted the vdisk location in VM settings The Ubuntu VM booted just fine afterwards. I also added a Windows 11 VM (directly using the unmounted WD-Black NVME), but this VM is barely used. Finally, I added AnythingLLM and Jellyfin as additional Docker containers on UNRAID, and installed a few more plugins: Nvidia Driver, GPU Statistics, and Unraid Patch. What did I try to fix the issue? As mentioned, I changed the hardware and some drives. I experimented with the Ubuntu VM since I feel like the issue might be there (just a hunch). Tried a bunch of things: CPU pinning Different network models RAM sizes Double-checked all virtualization settings in BIOS What didn't I try? Stopping the Ubuntu VM and just letting the system run. The thing is — as I mentioned earlier — I use the VM for development. I connect via VS Code Remote SSH to work directly on the domain with SSL enabled. Also, the VM hosts multiple client websites, which need to stay online. Still, I plan to shut it down tonight and see if UNRAID is still alive in the morning. (So far, it hasn’t survived a single night.) What would help me? Well, obviously: fixing the issue. But also: Getting Syslog to actually log kernel panics would be a massive step forward And if possible: a quick fix for why the UNRAID web UI isn’t available after reboot and needs a manual nginx restart Thanks a lot in advance. Robert singularity-diagnostics-20250329-1407.zip
March 29, 20251 yr Community Expert Quote As for software, I was running an Ubuntu Server 24.10 VM, with the vdisk stored on the two Samsung SSDs in the pool. This Ubuntu Server runs CapRover. CapRover is a SaaS platform based on Docker, so I’m running multiple Docker containers there for web hosting and app development. It’s the only VM I have. unraid doesn't support sas multipath, you may be expiring sas back plan disk disconnect with unraid. unraid really wants back planes to run in IT mode and be read as disks. --Alot here I think i misread something... Edited March 29, 20251 yr by bmartino1 Data - Typo
March 29, 20251 yr Community Expert Summary of Findings So Far: AreaObservationRecommendation License Key: /boot/config/Basic.key errors repeat hundreds of timesCheck USB stick integrity — try a different brand/model and run fsck on it VM Disk Interface: Ubuntu VM using IDE interfaceSwitch to VirtIO for better performance and fewer I/O stalls MemoryNo OOM or memory issues at snapshot timeStill monitor long-term usage (especially CapRover + Docker) NTP Bind Errors: Multiple ntpd port binding failuresEnsure only one NTP service is running; check port 123 conflicts Syslog Logging: No crash data during freezeMay indicate hard freeze or USB I/O failure halting log writes is your usb licensed or did you try moving the usb to a usb 2.0 slot. ther have been isseus with unraid and usb 3.0 ports... Here are a few important observations from the last part of the syslog.txt Licensing/Key Issues: Quote emhttpd: error: get_key_info, 674: Invalid argument (22): stat: /boot/config/Basic.key This line is being repeated hundreds of times, suggesting that the Unraid server is having trouble accessing the USB flash drive where the license key should be stored. This could mean: The USB stick is failing or intermittently disconnecting File corruption on the USB stick The file /boot/config/Basic.key is missing or unreadable Run a filesystem check on the USB drive. Consider replacing it if issues persist, especially since you mentioned it's a brand-new stick — it may still be faulty. Flash drive also need a UUID. samsung fit may not have a true uuid second you seem to have a ntp issues taht may be cascading to a time difereianlt and count problem... NTP Time Sync Failing Quote ntpd: bind(...) failed: Address already in use ntpd: cannot bind address ... This shows ntpd (Network Time Protocol daemon) failing to bind to ports — could indicate a misconfigured network, or possibly something else already binding port 123 (or duplicate services from plugins or docker containers). You have FCP error: Quote Fix Common Problems: Other Warning: Unassigned Devices Plus not installed Fix Common Problems: Other Warning: Background notifications not enabled -Nothing critical here, but useful for general housekeeping. So thing to look into: Whether your USB drive is consistently mounted (/boot) Any deeper kernel errors or out-of-memory (OOM) events VM and Docker-specific logs for clues of what might be freezing the system as the system is most likely frezing try to log or do somethign with the flash drive when it is disconectd... You may be hitting OOM and issues with the VM... So now lets review your vm per the diag: caprover-host.txt log (Ubuntu VM) Quote VM Configuration (CapRover/Ubuntu Host) QEMU version: 9.1.0 Libvirt version: 10.7.0 Memory: 8GB allocated (8589934592 bytes) vCPUs: 12 (1 socket, 6 cores, 2 threads) Boot mode: OVMF (UEFI) Disk Interface: IDE (with rotation rate 1 = SSD emulation) Disk File: /mnt/ssd-storage/domains/caprover-host/vdisk1.img Network: VirtIO-net (52:54:00:86:bc:eb) Acceleration: KVM Potential Concerns or Red Flags Disk Interface: IDE IDE is old and inefficient. If the Ubuntu VM is doing heavy I/O (CapRover + Docker), this can be a major performance bottleneck or even cause freezing under load. Recommendation: Switch to VirtIO disk interface in the VM settings. Make sure the virtio drivers are installed in Ubuntu. Memory Allocation: 8GB Combined with other dockers + another Windows VM + Unraid overhead, you may be hitting memory exhaustion, especially if Ubuntu or Docker apps aren’t memory-limited. Let’s verify in the system meminfo.txt if there’s memory pressure or OOM events. VM Storage Location: /mnt/ssd-storage/... Ensure this pool is stable. If the pool gets unmounted or remounted mid-run (due to drive issues), the VM will hang/freeze. so some disk layout and setups may be concerneing... as Diag show: Total System Memory: 32 GB Free: ~12.2 GB Available: ~25.1 GB Used by Ubuntu VM: 8 GB (as configured) Swap: Not configured (an aditonal plugin not always needed) No red flags here regarding total memory usage at the time the diagnostics were captured — memory usage looks healthy, with ~12 GB free and ~14 GB cached. However… Risk of Memory Exhaustion Still Exists This snapshot doesn't confirm what happens right before a freeze. If your Ubuntu VM (CapRover, Docker containers) spikes in RAM or consumes memory over time, and you're running other VMs or Docker containers, you could still hit memory pressure or OOM (Out Of Memory) conditions. Let’s check syslog for any OOM killer messages, which would confirm that. -No OOM killer events were found in the system logs — that’s a good sign. It means the system hasn't hit critical memory pressure that would trigger the kernel to kill processes. So,Less likly to be the VM and memory realted..
March 30, 20251 yr Community Expert Mar 29 12:32:53 SINGULARITY emhttpd: error: get_key_info, 674: Invalid argument (22): stat: /boot/config/Basic.key This means that key is invalid, just delete that file to stop the spam, you are now using the Plus key.
March 30, 20251 yr Author Hi guys, and thanks for your replies. Good news so far: the server is still up and running smoothly since I shut down the Ubuntu VM. So, as I suspected, the freezes are caused by the VM. The other mentioned problems aren't that critical — I’ll deal with them later — but thanks a lot for pointing them out. I’ve also deleted the basic key file to stop the log spam. However, I was really surprised that the diagnostics said my Ubuntu VM is using an IDE interface — which is definitely not the case. I set up the VM with a SATA interface and checked the "SSD" option. I’ll still try your recommendation and switch the interface to VirtIO, but first I increased the max RAM to 20480MB and set a kernel memory limit in Ubuntu to 19968MB, based on these instructions: https://wiki.ubuntu.com/Kernel/KernelBootParameters Let’s see if that makes any difference. I’ll also add my current configuration of the Ubuntu VM below: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='3'> <name>caprover-host</name> <uuid>c18eeea4-c0d9-ea73-63b5-2313ff0bfa04</uuid> <metadata> <vmtemplate xmlns="unraid" name="Ubuntu" icon="ubuntu.png" os="ubuntu" webui="" storage="default"/> </metadata> <memory unit='KiB'>20971520</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>12</vcpu> <cputune> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='1'/> <vcpupin vcpu='2' cpuset='2'/> <vcpupin vcpu='3' cpuset='3'/> <vcpupin vcpu='4' cpuset='4'/> <vcpupin vcpu='5' cpuset='5'/> <vcpupin vcpu='6' cpuset='6'/> <vcpupin vcpu='7' cpuset='7'/> <vcpupin vcpu='8' cpuset='16'/> <vcpupin vcpu='9' cpuset='17'/> <vcpupin vcpu='10' cpuset='18'/> <vcpupin vcpu='11' cpuset='19'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-9.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/c18eeea4-c0d9-ea73-63b5-2313ff0bfa04_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' clusters='1' cores='6' threads='2'/> <cache mode='passthrough'/> </cpu> <clock offset='localtime'> <timer name='hpet' present='no'/> <timer name='hypervclock' present='no'/> <timer name='pit' tickpolicy='delay'/> <timer name='rtc' tickpolicy='catchup'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback' discard='unmap'/> <source file='/mnt/ssd-storage/domains/caprover-host/vdisk1.img' index='1'/> <backingStore/> <target dev='hdc' bus='sata' rotation_rate='1'/> <serial>vdisk1</serial> <boot order='1'/> <alias name='sata0-0-2'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <controller type='sata' index='0'> <alias name='sata0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:86:bc:eb'/> <source bridge='br0'/> <target dev='vnet2'/> <model type='virtio-net'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/run/libvirt/qemu/channel/3-caprover-host/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <graphics type='vnc' port='5900' autoport='yes' websocket='5700' listen='0.0.0.0' sharePolicy='ignore'> <listen type='address' address='0.0.0.0'/> </graphics> <audio id='1' type='none'/> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain> Little update after posting my response here but still wait for approvement. The Ubuntu kernel memory limit didn't work. System crashed again. I've now switched to the VirtIO interface for the disc. Here is the current meminfo of the Ubuntu VM: MemTotal: 9650796 kB MemFree: 2692456 kB MemAvailable: 5691676 kB Buffers: 301180 kB Cached: 2577108 kB SwapCached: 0 kB Active: 4868664 kB Inactive: 1182072 kB Active(anon): 3188104 kB Inactive(anon): 0 kB Active(file): 1680560 kB Inactive(file): 1182072 kB Unevictable: 25504 kB Mlocked: 25504 kB SwapTotal: 3755004 kB SwapFree: 3755004 kB Zswap: 0 kB Zswapped: 0 kB Dirty: 184 kB Writeback: 4 kB AnonPages: 3198164 kB Mapped: 1177104 kB Shmem: 8752 kB KReclaimable: 449268 kB Slab: 628708 kB SReclaimable: 449268 kB SUnreclaim: 179440 kB KernelStack: 20608 kB PageTables: 82120 kB SecPageTables: 0 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 8580400 kB Committed_AS: 8727292 kB VmallocTotal: 34359738367 kB VmallocUsed: 40452 kB VmallocChunk: 0 kB Percpu: 8256 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB Unaccepted: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 261056 kB DirectMap2M: 4976640 kB DirectMap1G: 7340032 kB Edited March 30, 20251 yr by justRobsi
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.