November 20, 20232 yr Hello, I can no longer log in to the WebUI but it works via ssh I have started diagnostics under ssh and have the following output: root@U-YK:~# diagnostics Starting diagnostics collection... tail: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device tr: write error: No space left on device sed: couldn't flush <unknown>: No space left on device sed: couldn't flush <unknown>: No space left on device awk: cmd. line:1: (FILENAME=/tower-diagnostics-20231120-1557/logs/syslog.txt FNR=63) fatal: print to "standard output" failed: No space left on device sed: couldn't flush <unknown>: No space left on device tail: error writing 'standard output': No space left on device done. ZIP file '/boot/logs/tower-diagnostics-20231120-1557.zip' created. root@U-YK:~# root@U-YK:~# df Filesystem 1K-blocks Used Available Use% Mounted on rootfs 16309572 16309572 0 100% / tmpfs 32768 992 31776 4% /run /dev/sda1 62640288 896704 61743584 2% /boot overlay 16309572 16309572 0 100% /lib overlay 16309572 16309572 0 100% /usr devtmpfs 8192 0 8192 0% /dev tmpfs 16323332 0 16323332 0% /dev/shm tmpfs 131072 38320 92752 30% /var/log tmpfs 1024 0 1024 0% /mnt/disks tmpfs 1024 0 1024 0% /mnt/remotes tmpfs 1024 0 1024 0% /mnt/addons tmpfs 1024 0 1024 0% /mnt/rootshare /dev/md1p1 3906540608 2782381212 1124159396 72% /mnt/disk1 /dev/md2p1 1953036628 324607340 1628429288 17% /mnt/disk2 shfs 5859577236 3106988552 2752588684 54% /mnt/user0 shfs 5859577236 3106988552 2752588684 54% /mnt/user /dev/sdd1 11718850556 8895632708 2823217848 76% /mnt/disks/Elements /dev/loop2 41943040 20638672 20187872 51% /var/lib/docker /dev/loop3 1048576 9488 920912 2% /etc/libvirt tmpfs 3264664 0 3264664 0% /run/user/0 root@U-YK:~# Can anyone help here? Thanks a lot Edited November 20, 20232 yr by powerforum75
November 20, 20232 yr Community Expert 1 hour ago, powerforum75 said: rootfs 16309572 16309572 0 100% / roots is completely full, reboot in safe mode and see if you can find what's filling it up.
November 21, 20232 yr Author After a reboot I checked everything, it is due to a windows VM. How can I find out what the cause is? The VM has been there for some time!
November 21, 20232 yr Community Expert Kind of unusual, post the VM XML to see if there's anything we can see.
November 21, 20232 yr Author Is a windows 11 VM running on an NVMe. <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Win11</name> <uuid>c94cb20e-ec12-a5ef-fc67-e68c085bxxx</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/> </metadata> <memory unit='KiB'>12582912</memory> <currentMemory unit='KiB'>12582912</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='5'/> <vcpupin vcpu='2' cpuset='6'/> <vcpupin vcpu='3' cpuset='7'/> <vcpupin vcpu='4' cpuset='8'/> <vcpupin vcpu='5' cpuset='9'/> <vcpupin vcpu='6' cpuset='10'/> <vcpupin vcpu='7' cpuset='11'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-7.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader> <nvram>/etc/libvirt/qemu/nvram/c94cb20e-ec12-a5ef-fc67-e68c085b8df2_VARS-pure-efi-tpm.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='4' threads='2'/> <cache mode='passthrough'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.229-1.iso'/> <target dev='hdb' bus='sata'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:bd:cf:39'/> <source bridge='br0'/> <model type='virtio-net'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='qemu-vdagent'> <source> <clipboard copypaste='yes'/> <mouse mode='client'/> </source> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='4'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <tpm model='tpm-tis'> <backend type='emulator' version='2.0' persistent_state='yes'/> </tpm> <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='de'> <listen type='address' address='0.0.0.0'/> </graphics> <audio id='1' type='none'/> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <boot order='2'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> </domain>
November 21, 20232 yr Community Expert Edited your post to add code tags, makes it more readable, I don't see anything out of the ordinary, and don't remember a VM filling up rootfs before, but maybe someone else can see something.
November 21, 20232 yr Author When I start the VM, rootfs goes up. But how can I see where it writes and what it writes? root@U-YK:~# df -hT /root/ Filesystem Type Size Used Avail Use% Mounted on rootfs rootfs 16G 4.9G 11G 32% / root@U-YK:~# ls -la total 20 drwx--x--- 3 root root 200 Nov 21 17:47 ./ drwxr-xr-x 21 root root 440 Nov 21 12:25 ../ -rw------- 1 root root 3705 Nov 21 17:47 .bash_history -rw-r--r-- 1 root root 316 Aug 31 19:57 .bash_profile -rwxr-xr-x 1 root root 60 Aug 31 19:57 .bashrc* drwxr-xr-x 3 root root 60 Aug 31 19:57 .config/ lrwxrwxrwx 1 root root 30 Aug 31 19:57 .docker -> /boot/config/plugins/dockerMan/ lrwxrwxrwx 1 root root 21 Aug 31 19:57 .ssh -> /boot/config/ssh/root/ -rw-r--r-- 1 root root 59 Nov 21 12:25 .tmux.conf -rw-r--r-- 1 root root 215 Nov 21 12:58 .wget-hsts root@U-YK:~#
November 22, 20232 yr Author Solution I have found the error. 🤦♂️ It was the Urbackup client on the Windows VM that referred to one of my Unraid share directories in the file backup (I added it). But why it filled up the rootfs directory I don't know!
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.