• [6.10.0RC2] VM doesn't start because has no space for logs (but does have).


    NLS
    • Minor

    So I try to start a general use Windows VM I have, that was running fine days ago (also in RC2), with error

    "Unable to write to file /var/log/libvirt/qemu/Windows.log: No space left on device"

     

    Not sure where /var/log is, but I have plenty gigabytes empty both in my boot stick and my Cache disk.

    Clicking "logs" in the VM menu, doesn't list any huge log.

     

    I doubt the issue relates to RC2, it just happens I use RC2. VM was working fine a few days ago (in RC2 and before).

     

    Any ideas?

     




    User Feedback

    Recommended Comments

    I'm having the same issue. Here are my diagnostics

     

    And running the command du -h -d 1 /var/log  initially showed this after rebooting:

    0       /var/log/pwfail
    4.0K    /var/log/swtpm
    20K     /var/log/samba
    0       /var/log/plugins
    0       /var/log/pkgtools
    0       /var/log/nginx
    0       /var/log/nfsd
    12K     /var/log/libvirt
    328K    /var/log

     

     

    BUT it showed this just after shutting down a VM I couldn't get to pass through again and had been showing 84 degrees C! (I've been having problems trying to pass through VM's on my new motherboard/setup, though I also added a cache drive and may have messed things up when trying to move around info and to separate VM info onto the new drive.

    and after doing the command line that you requested earlier now shows (after shutting down the VM but not rebooting yet)

    0       /var/log/pwfail
    0       /var/log/swtpm
    20K     /var/log/samba
    0       /var/log/plugins
    0       /var/log/pkgtools
    0       /var/log/nginx
    0       /var/log/nfsd
    7.6M    /var/log/libvirt
    128M    /var/log

    rushtallica-diagnostics-20211119-1008.zip

    Link to comment

    Well log bar in dashboard does show 100%.

     

    Attached are my diags.

      

    root@quasar-ultima:/var/log# ls -l
    total 75236
    -rw-r--r-- 1 root   root    24200 Nov 11 09:21 Xorg.0.log
    -rw------- 1 root   root        0 Oct 19 00:05 btmp
    -rw-r--r-- 1 root   root        0 Apr 28  2021 cron
    -rw-r--r-- 1 root   root        0 Apr 28  2021 debug
    -rw-r--r-- 1 root   root    70702 Nov 11 09:20 dmesg
    -rw-r--r-- 1 root   root     8192 Nov 22 22:40 docker.log
    -rw-r--r-- 1 root   root        0 Feb 13  2021 faillog
    -rw-rw-rw- 1 root   root      618 Nov 11 09:21 fluxbox.log
    -rw-r--r-- 1 root   root        0 Apr  8  2000 lastlog
    drwxr-xr-x 3 root   root      120 Nov 11 09:22 libvirt/
    -rw-r--r-- 1 root   root     4096 Nov 23 00:20 maillog
    -rw-r--r-- 1 root   root        0 Nov 11 09:20 mcelog
    -rw-r--r-- 1 root   root        0 Apr 28  2021 messages
    drwxr-xr-x 2 root   root       40 Jun 15 21:38 nfsd/
    drwxr-x--- 2 nobody root       80 Nov 13 04:40 nginx/
    lrwxrwxrwx 1 root   root       24 Oct 19 00:06 packages -> ../lib/pkgtools/packages/
    drwxr-xr-x 5 root   root      100 Nov 13 02:09 pkgtools/
    drwxr-xr-x 2 root   root      600 Nov 22 16:14 plugins/
    drwxr-xr-x 2 root   root       40 Nov 11 10:18 pwfail/
    lrwxrwxrwx 1 root   root       25 Oct 19 00:05 removed_packages -> pkgtools/removed_packages/
    lrwxrwxrwx 1 root   root       24 Oct 19 00:05 removed_scripts -> pkgtools/removed_scripts/
    lrwxrwxrwx 1 root   root       34 Nov 13 02:09 removed_uninstall_scripts -> pkgtools/removed_uninstall_scripts/
    drwxr-xr-x 3 root   root      160 Nov 11 09:21 samba/
    lrwxrwxrwx 1 root   root       23 Oct 19 00:06 scripts -> ../lib/pkgtools/scripts/
    -rw-r--r-- 1 root   root        0 Apr 28  2021 secure
    lrwxrwxrwx 1 root   root       21 Oct 19 00:06 setup -> ../lib/pkgtools/setup/
    -rw-r--r-- 1 root   root        0 Apr 28  2021 spooler
    drwxr-xr-x 3 root   root       60 Oct 11 19:01 swtpm/
    -rw-r--r-- 1 root   root        0 Nov 23 02:23 syslog
    -rw-r--r-- 1 root   root 70582272 Nov 13 04:37 syslog.1
    -rw-r--r-- 1 root   root  6334751 Nov 12 03:22 syslog.2
    -rw-r--r-- 1 root   root        0 Nov 11 09:20 vfio-pci
    -rw-rw-r-- 1 root   utmp     7680 Nov 11 09:21 wtmp
    root@quasar-ultima:/var/log#  du -h -d 1 /var/log
    0       /var/log/pwfail
    0       /var/log/swtpm
    20K     /var/log/samba
    0       /var/log/plugins
    0       /var/log/pkgtools
    55M     /var/log/nginx
    0       /var/log/nfsd
    20K     /var/log/libvirt
    128M    /var/log

     

    Looks like syslog is filling up the space?

    I deleted syslog files and the VM fired up ok.

    But WHO is filling syslog and why syslog is not managed some way to not fill up the quota of the directory?

     

    quasar-ultima-diagnostics-20211123-0222.zip

    Edited by NLS
    Link to comment
    6 hours ago, NLS said:

    But WHO is filling syslog

     

    Nov 12 08:04:47 quasar-ultima rpc.mountd[6064]: refused mount request from 172.17.2.111 for /rootshare (/): not exported

    and other paths.

     

    also plenty of

    Nov 12 08:04:49 quasar-ultima nginx: 2021/11/12 08:04:49 [crit] 10941#10941: ngx_slab_alloc() failed: no memory
    Nov 12 08:04:49 quasar-ultima nginx: 2021/11/12 08:04:49 [error] 10941#10941: shpool alloc failed
    Nov 12 08:04:49 quasar-ultima nginx: 2021/11/12 08:04:49 [error] 10941#10941: nchan: Out of shared memory while allocating message of size 8328. Increase nchan_max_reserved_memory.
    Nov 12 08:04:49 quasar-ultima nginx: 2021/11/12 08:04:49 [error] 10941#10941: *821458 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost"

     

    Link to comment

    I'm having the same issue (on Unraid 6.10.0 rc2) with mine filling up. I have to restart the server to get the log cleared. Mine has to do with the vbios misconfigured for passthrough which results in this error
    2021-12-27T03:40:02.510435Z qemu-system-x86_64: vfio_region_write(0000:21:00.0:region1+0xe5340, 0x0,1) failed: Device or resource busy

    over and over

    When you try to restart the VM after the log is dumped over and over it errors out with this

    Unable to write to file /var/log/libvirt/qemu/Windows 11.log: No space left on device

     

    I am amazed that we do not have a way to stop it from flooding the log so fast.

    Link to comment
    On 12/26/2021 at 7:53 PM, TPNuts said:

    I'm having the same issue (on Unraid 6.10.0 rc2) with mine filling up. I have to restart the server to get the log cleared. Mine has to do with the vbios misconfigured for passthrough which results in this error
    2021-12-27T03:40:02.510435Z qemu-system-x86_64: vfio_region_write(0000:21:00.0:region1+0xe5340, 0x0,1) failed: Device or resource busy

    over and over

    When you try to restart the VM after the log is dumped over and over it errors out with this

    Unable to write to file /var/log/libvirt/qemu/Windows 11.log: No space left on device

     

    I am amazed that we do not have a way to stop it from flooding the log so fast.

    I'm having this exact same issue. Did you ever find a fix?

     

    I deleted my libvirt file and rebooted to see if that would reset something, but still running into this same issue with the exact error message as you.

    Edited by BBLV
    Link to comment

    I'm on RC3. Tried RC4 but didn't like it. This is a dual boot machine. Running Windows natively has no problems.

     

    Link to comment

    FYI I updated to RC4, created a new Windows 11 VM and it fired up right away with VNC - no error messages (I'm remote right now, but will try assigning my new 3090 GPU as soon as I get home later today).

     

    Update: switched over from VNC to the 3090 GPU and the new Windows 11 VM fired up immediately. Loving RC4 so far!

    Edited by BBLV
    Link to comment

    Has anyone resolved this issue? I'm still having the problem. VM won't start and after awhile fix common problems says no space left. I tried creating a separate Windows 11 VM and still get the same issue. Also, now when I try to log into my native Windows 11 I get a diagnosing computer error! After rebooting everything works okay in Windows 11 native.

    Link to comment
    6 minutes ago, tjw said:

    Has anyone resolved this issue? I'm still having the problem. VM won't start and after awhile fix common problems says no space left. I tried creating a separate Windows 11 VM and still get the same issue. Also, now when I try to log into my native Windows 11 I get a diagnosing computer error! After rebooting everything works okay in Windows 11 native.

    You update to RC4? I had the issue in RC2, can’t remember if I did troubleshooting in RC3, and RC4 just works for me (I created a brand new Windows 11 install via VM/VNC from scratch on a freshly reformatted NVME drive after updating to RC4 FWIW).

    Edited by BBLV
    Link to comment
    55 minutes ago, BBLV said:

    You update to RC4? I had the issue in RC2, can’t remember if I did troubleshooting in RC3, and RC4 just works for me (I created a brand new Windows 11 install via VM/VNC from scratch on a freshly reformatted NVME drive after updating to RC4 FWIW).

    I'll try it again tonight. Strange though, it was working on rc3 for the past several weeks.

    Link to comment

    Updated to RC4 but no joy! I'm starting to get frustrated. This is the only forum or thread I can find that mentions this problem.

    Link to comment

    I got it figured out. I had both monitors plugged in to the video card. When I pugged the second monitor into the motherboard everything is working as expected. I just don't have dual monitors on the VM. So Happy!

    Link to comment
    4 hours ago, tjw said:

    I got it figured out. I had both monitors plugged in to the video card. When I pugged the second monitor into the motherboard everything is working as expected. I just don't have dual monitors on the VM. So Happy!

    Weird! Glad you figured it out.

    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.