libvirt.img is in-use, cannot mount


PR.

Recommended Posts

Hi,

 

Bit of an issue, I was having a problem with one core on my CPU being maxed out so I was stopping dockers trying to trace it (eventually found it was the directory caching plugin).

 

During that I set 'Enable VM's' to off, but now I can't turn it back on, the error "root: /mnt/user/system/libvirt/libvirt.img is in-use, cannot mount" is showing up in the logs.

 

I'm sure a reboot would fix the issue but wondering if there's another way?

 

Thanks

 

 

Link to comment
  • 7 months later...
  • 6 months later...
59 minutes ago, barf said:

fuser -c /mnt/user/system/libvirt/libvirt.img

ps -ef | grep pid that's using it

 

Mine was docker.  I had to stop all dockers and was able to restart kvm.

 

Did you have any docker container running that related to running VMs?  Cannot think of any other reason that libvirt.img would be in use by a docker container.

Link to comment
  • 2 weeks later...
  • 1 month later...
  • 1 year later...

Did this ever get fixed?   I've got an issue with KVM not starting and doing the above fuser -c /mnt/user/system/libvirt/libvirt.img replies with a large number of processes.  Without killing them all and restarting the array/docker/server is there any other options? 

image.png

Link to comment
  • 1 month later...
On 10/25/2020 at 11:30 PM, isaw said:

Did this ever get fixed?   I've got an issue with KVM not starting and doing the above fuser -c /mnt/user/system/libvirt/libvirt.img replies with a large number of processes.  Without killing them all and restarting the array/docker/server is there any other options? 

Hi,

Same here. When I try to restart the Libvirt service I get the following error:

Dec 21 23:45:51 MOZART root: /mnt/user/system/libvirt/libvirt.img is in-use, cannot mount

... which seems to be caused by many processes:

root@MOZART:~# fuser -c /mnt/user/system/libvirt/libvirt.img
/mnt/user/system/libvirt/libvirt.img:   328   329   374cm   406   512   542m   611m   667m   726m  3948  5304c 10517c 10555c 12015 12857 12916c 13025c 13929 14060 16265 16802c 17139 17540c 17582 18255 18271c 18278c 18312 18522c 18523c 18524c 18525c 18526c 18686 18822 19091 19232c 20250 20251 20252 20253 21427c 21678 21706 21707 22021c 22564c 22580c 22581c 22805 22812m 22870m 22871m 23030m 23501 23505 23509 23526 23527 23528 23529 23530 23546m 23557e 23559e 24224 24225 24226 24246 24247 24248 24249 25821 26019c 26908 27102 27344c 27861 28083c 29489m 29531c 29601 30878 30918 30919 30978cm 31046m 32746

I have the virt-manager Docker container installed but the issue occurred while it was stopped.

I confirm that stopping the array and starting it again fixes the issue.

Best,

OP

Link to comment
  • 8 months later...

Going to bump this as I think it needs to be upgraded to a bug.

When disabling VmManager via settings it will stop, but reenabling required the docker service to stop also (the .img file is being accessed by all Dockers).  Once docker is stopped VM can be restarted, then Docker restarted.

Link to comment
On 8/31/2021 at 12:36 AM, isaw said:

When disabling VmManager via settings it will stop, but reenabling required the docker service to stop also (the .img file is being accessed by all Dockers).  Once docker is stopped VM can be restarted, then Docker restarted.

 

No reason I can think of that the libvirt.img file should ever be accessed by docker containers (in fact it should not even be visible to them I would have thought!). 

 

I have just checked on my own live system  running 6.9.2 and see similar behaviour.  My guess is they disappear when stopping the VM service as libvirt.iimg then gets unmounted.  Feels like feedback is needed from someone closer to @limetech as to whether this is a bug or expected behaviour and if it is expected behaviour the rationale so we understand what is happening.

 

EDIT:  Just checked on my development system running 6.10.0 rc2d and am not seeing this behaviour.  Not sure if that means this is a fixed bug or the development system is not running whatever triggers the behaviour. Might need  temporarily put the live system on that release to see if the behaviour disappears.

 

EDIT2:  Just brought my live system online with 6.10.0 rc2d and still see this behaviour.

 

Link to comment
  • 1 month later...
  • 2 months later...

The same happened to me after stopping the VM service, renaming the "domains" share to "vdisks" and restarting the VM service.

 

What I tried:

 

root@thoth:~# umount /mnt/cache/libvirt/libvirt.img
umount: /mnt/cache/libvirt/libvirt.img: not mounted.
root@thoth:~# /usr/local/sbin/mount_image '/mnt/cache/libvirt/libvirt.img' /etc/libvirt 1
/mnt/cache/libvirt/libvirt.img is in-use, cannot mount
root@thoth:~# fuser -c /mnt/cache/libvirt/libvirt.img
/mnt/cache/libvirt/libvirt.img:  6030m  6055m  7314m  7574m  8400m  8520m  8521m  8523m  8528m  8529m  8554m  8557m  8581m  8596m  8646m  8647m  8648m  8649m  8729m  8841m  8842m  8845m  9040m  9045m 12060m 12144m 16103m 16106m 16107m 16109m 16110m 16285m 16301m 16314m 16338m 16339m 16374m 16400m 16424cm 16446m 16455m 16479m 16488m 16658m 16659m 16662m 16663m 16678m 16893m 17537m 18805m 18818m 19679m 19680m 19681m 19682m 19683m 19684m 19685m 19686m 19687m 19688m 19689m 19690m 19691m 31769m 31843m 32152m 32155m 32195cm 32321m
root@thoth:~# ps -p 6030
  PID TTY          TIME CMD
 6030 ?        00:00:39 dockerd
root@thoth:~# ps -p 6055
  PID TTY          TIME CMD
 6055 ?        00:03:22 containerd
root@thoth:~# ps -p 7314
  PID TTY          TIME CMD
 7314 ?        00:00:00 s6-svscan
root@thoth:~# ps -p 7574
  PID TTY          TIME CMD
 7574 ?        00:00:00 s6-supervise
root@thoth:~# ps -p 8400
  PID TTY          TIME CMD
 8400 ?        00:00:00 dumb-init
root@thoth:~# ps -p 32321
  PID TTY          TIME CMD
32321 ?        00:00:06 Plex Tuner Serv
root@thoth:~# 

 

Then while and after stopping the docker service:

root@thoth:~# fuser -c /mnt/cache/libvirt/libvirt.img
/mnt/cache/libvirt/libvirt.img:  5651m  6030m  6055m
root@thoth:~# ps -p 5651
  PID TTY          TIME CMD
 5651 ?        00:00:00 s6-sync
root@thoth:~# ps -p 6030
  PID TTY          TIME CMD
 6030 ?        00:00:39 dockerd
root@thoth:~# ps -p 6055
  PID TTY          TIME CMD
 6055 ?        00:03:23 containerd
root@thoth:~# fuser -c /mnt/cache/libvirt/libvirt.img
/mnt/cache/libvirt/libvirt.img:  5651m  6030m  6055m
root@thoth:~# fuser -c /mnt/cache/libvirt/libvirt.img
root@thoth:~# 

 

 

Then I started the docker service again:

fuser -c /mnt/cache/libvirt/libvirt.img
/mnt/cache/libvirt/libvirt.img:  8869m  8893m  9644m  9788m 10066m 10067m 10071m 10072m 10088m 10096m 10197m 10198m 10199m 10200m 10201m 10202m 10203m 10204m 10205m 10206m 10207m 10208m 10209m 10210m 10271m 10630m 10788m 10789m 10790m 10793m 10794m 10806m 10807m 10825m 10841m 10842m 10843m 10844m 10925m 11227m 11259m 11262m 11325m 11472cm 11474m 11570m 11571m 11572m 11729m 11745m 11754m 11755m 11757m 11762m 11826m 11863cm

 

And started successfully the VM service, too (yes completely the same output):

fuser -c /mnt/cache/libvirt/libvirt.img
/mnt/cache/libvirt/libvirt.img:  8869m  8893m  9644m  9788m 10066m 10067m 10071m 10072m 10088m 10096m 10197m 10198m 10199m 10200m 10201m 10202m 10203m 10204m 10205m 10206m 10207m 10208m 10209m 10210m 10271m 10630m 10788m 10789m 10790m 10793m 10794m 10806m 10807m 10825m 10841m 10842m 10843m 10844m 10925m 11227m 11259m 11262m 11325m 11472cm 11474m 11570m 11571m 11572m 11729m 11745m 11754m 11755m 11757m 11762m 11826m 11863cm

 

I'd say it has nothing to do with the output of fuser. Must be something else wrong.

Link to comment
  • 4 weeks later...
On 12/28/2021 at 8:35 PM, mgutt said:

If someone else has this problem, please post output of these commands:

fuser -s "/mnt/user/system/libvirt/libvirt.img"
losetup -j "/mnt/user/system/libvirt/libvirt.img"

 

These two checks are part of /usr/local/sbin/in_use which is the reason for the "is in-use, cannot mount" error.

 

I have this problem also. The but none of the commands render any output:

:~# fuser -s "/mnt/user/system/libvirt/libvirt.img"
:~# losetup -j "/mnt/user/system/libvirt/libvirt.img"
:~#

 

I still have the problem where I see:

~# fuser -c /mnt/user/system/libvirt/libvirt.img
/mnt/user/system/libvirt/libvirt.img: 12659c 14997 14999 15000 15051m 15619 15621 15623 15625 15627 17069c 17136 17525 18758 19036 19298 19856m 19980m 20180m 20538 20705 20824m 21309 21751 21986 21988 21989 22042 22043 22044 22045 22046 22047 22048 22049 22052 22053 22055 22056 22057 22059 22060 22061 22062 22063 22064 22066 22067 22068 22070 22071 22072 22073 22074 22075 22076 22077 22078 22079 23509m 23942 24433 24434 24456 24457 24458 24459 25140c 25219c

 

Which correspond to these processes:

php-fpm: master process (/etc/php7/php-fpm.conf)
nginx: master process /usr/sbin/nginx -c /config/nginx/nginx.conf
/usr/bin/python3 /usr/bin/fail2ban-client -x -f start
influxd
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: cache manager process
/usr/bin/mariadbd --basedir=/usr --datadir=/config/databases --plugin-dir=/u
/usr/bin/python /usr/bin/supervisord -c /etc/supervisor.conf -n
/usr/bin/python /usr/bin/supervisord -c /etc/supervisor.conf -n
/usr/bin/python /usr/bin/supervisord -c /etc/supervisor.conf -n
/usr/bin/python /usr/bin/supervisord -c /etc/supervisor.conf -n
/usr/bin/python /usr/bin/supervisord -c /etc/supervisor.conf -n
/usr/bin/mono --debug /usr/lib/sonarr/bin/Sonarr.exe -nobrowser -data=/confi
/app/bin/Readarr --nobrowser --data=/config
/usr/lib/radarr/bin/Radarr -nobrowser -data=/config
/usr/bin/python /usr/bin/supervisord -c /etc/supervisor.conf -n
/opt/ombi/Ombi --storage /config --host http://*:3579
/usr/lib/lidarr/bin/Lidarr -nobrowser -data=/config
python3 /app/bin/nzbhydra2wrapperPy3.py --nobrowser --datafolder /config
java -Xmx256M -DfromWrapper -XX:TieredStopAtLevel=1 -noverify -XX:+HeapDumpO
nginx: master process /usr/sbin/nginx -c /config/nginx/nginx.conf
php-fpm: master process (/etc/php7/php-fpm.conf)
python3 /app/tautulli/Tautulli.py --datadir /config
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
/usr/bin/python3 -u /app/bazarr/bin/bazarr/main.py --no-update --config /con
/usr/sbin/python3 /usr/lib/sabnzbd/SABnzbd.py --daemon --config-file /config
php-fpm: master process (/etc/php7/php-fpm.conf)
nginx: master process /usr/sbin/nginx -c /config/nginx/nginx.conf
nginx: worker process
nginx: worker process
nginx: worker process
nginx: worker process
/usr/bin/python /usr/bin/deluged -c /config -L info -l /config/deluged.log
deluge-web

 

Attaching my diagnostics from after I tried reboot which didn't help. 

I then tried stopping the Docker service and the start the VM service and then it did come up. But...

...starting my daily driver Win10 VM did not work. I get "Cannot get interface MTU on 'virbr0': No such device."

 

Searched some more and found this:

:~# virsh net-start default
Network default started

 

This made it possible to start the VM without the error. I can use TeamViewer to use the VM, but I get "Failed to connect to server" in the VNC Remote window

diagnostics-20220124-1513.zip

Link to comment

I too am having this problem.  Stopped VM Manager in settings in order to move all my VMs to a new NVMe drive installed just for VMs.  When I started the VM Manager I ran in to this problem.

 

Log Entry:

root: /mnt/apps/system/libvirt/libvirt.img is in-use, cannot mount

 

Output of requested commends:

:~# fuser -s "/mnt/user/system/libvirt/libvirt.img"
:~#
:~# losetup -j "/mnt/user/system/libvirt/libvirt.img"
:~#
:~#  fuser -c /mnt/user/system/libvirt/libvirt.img 
/mnt/user/system/libvirt/libvirt.img:  3604  3766m  6260c  6578c  8214  8215  8367  9023 10355 12704 12705 12706 12767m 13068 13069 13070 13071 13072 14181m 16701m 17871 17873 17905 17906 17907 17908 20952 32970c 40041
:~# 

 

Those PIDs also corresponds to every docker container I have running at the time.  As my server is in use at this time I cannot reboot or stop docker to try to fix.

 

Here is my Diagnostics.  Need anything else just ask.  Happy to help any way I can.  I won't be able to try to get VM Manager back up for a couple hours and I will check here before I try just in case there is anything you want me to do.

rudder2-server-diagnostics-20220125-1619.zip

Link to comment

A friend of mine had this problem and I tried many things to bypass this problem without rebooting, but I was not successful.

 

As the others posted, these commands don't return anything:

root@Tower:~# fuser -s "/mnt/user/system/libvirt/libvirt.img"
root@Tower:~# losetup -j "/mnt/user/system/libvirt/libvirt.img"

 

But the libvirt.img is still attached as block device (even after the VM service has been stopped):

root@Tower:~# losetup -l
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                             DIO LOG-SEC
/dev/loop1         0      0         1  1 /boot/bzmodules                         0     512
/dev/loop2         0      0         1  0 /mnt/cache/system/libvirt/libvirt.img   1     512
/dev/loop0         0      0         1  1 /boot/bzfirmware                        0     512
root@Tower:~# losetup -a
/dev/loop1: [2049]:11 (/boot/bzmodules)
/dev/loop2: [66305]:536871053 (/mnt/cache/system/libvirt/libvirt.img)
/dev/loop0: [2049]:9 (/boot/bzfirmware)
root@Tower:~# dmsetup info
No devices found

 

Nothing is using this loop device as all commands return nothing:

fuser -c /dev/loop2
fuser -f /dev/loop2
lsof | grep loop2

 

No error detaching the block device:

losetup -d /dev/loop2

 

dmsetup fails as well:

root@Tower:~# dmsetup remove --force loop2
device-mapper: table ioctl on loop2  failed: No such device or address
device-mapper: reload ioctl on loop2  failed: No such device or address
device-mapper: remove ioctl on loop2  failed: No such device or address

 

 

I was able to mount/read/unmount it:

root@Tower:~# fdisk /dev/loop2 -l
Disk /dev/loop2: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@Tower:~# blkid | grep loop2
/dev/loop2: UUID="2791640b-f2f7-4b5d-be92-b7715a2268b7" UUID_SUB="73b862f5-df27-432d-9f14-d46bdffe239f" BLOCK_SIZE="4096" TYPE="btrfs"
root@Tower:~# mkdir /mnt/disks/libvirt
root@Tower:~# mount /dev/loop2 /mnt/disks/libvirt
root@Tower:~# ls /mnt/disks/libvirt/
hooks/              qemu-lockd.conf        virtlockd.conf      virtqemud.conf
libvirt-admin.conf  qemu.conf              virtlogd.conf       virtsecretd.conf
libvirt.conf        secrets/               virtnetworkd.conf   virtstoraged.conf
libvirtd.conf       virt-login-shell.conf  virtnodedevd.conf
nwfilter/           virtchd.conf           virtnwfilterd.conf
qemu/               virtinterfaced.conf    virtproxyd.conf
root@Tower:~# mountpoint /mnt/disks/libvirt
/mnt/disks/libvirt is a mountpoint
root@Tower:~# umount /mnt/disks/libvirt
root@Tower:~# rmdir /mnt/disks/libvirt

 

Deleting the file does not help:

root@Tower:~# cp -a /mnt/cache/system/libvirt/libvirt.img /mnt/cache/system/libvirt/libvirt.img.copy
root@Tower:~# rm /mnt/cache/system/libvirt/libvirt.img
root@Tower:~# losetup -l
NAME     SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                                      DIO LOG-SEC
/dev/loop1       0      0         1  1 /boot/bzmodules                                  0     512
/dev/loop2       0      0         1  0 /mnt/cache/system/libvirt/libvirt.img (deleted)  1     512
/dev/loop0       0      0         1  1 /boot/bzfirmware                                 0     512

 

Later I found a method to get more verbose output, but I'm not sure if "No such file or directory" was returned as I deleted the file?!

root@Tower:~# LOOPDEV_DEBUG=all losetup -vd /dev/loop2
31598: loopdev:      CXT: [0x7ffe409acdd0]: initialize context
31598: loopdev:      CXT: [0x7ffe409acdd0]: init: ignore ioctls
31598: loopdev:      CXT: [0x7ffe409acdd0]: init: loop-control detected 
31598: loopdev:      CXT: [0x7ffe409acdd0]: /dev/loop2 name assigned
31598: loopdev:      CXT: [0x7ffe409acdd0]: open /dev/loop2 [ro]: No such file or directory
31598: loopdev:      CXT: [0x7ffe409acdd0]: device removed
31598: loopdev:      CXT: [0x7ffe409acdd0]: de-initialize
31598: loopdev:      CXT: [0x7ffe409acdd0]: closing old open fd
31598: loopdev:     ITER: [0x7ffe409acfa8]: de-initialize

 

So the next user which has this problem should try to execute those two commands (while VM service has been stopped), loopX must be replaced by the output of "losetup -l":

losetup -l
LOOPDEV_DEBUG=all losetup -vd /dev/loopX

 

By that we can verify if it's "No such file" in any case.

 

Another thing which I was not able to test is to stop the array and to check how it does influence the "losetup -l" output. And the next time I like to test if the installation of kpartx and the command "kpartx -d /dev/loopX" could help.

  • Thanks 1
Link to comment

Just encountered the same problem, libvirt is located on my ZFS pool, this is my losetup output:

 

root@unRAID:~# losetup -l
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                       DIO LOG-SEC
/dev/loop1         0      0         1  1 /boot/bzmodules                   0     512
/dev/loop2         0      0         1  0 /ZFSPOOL/vi/libvirt/libvirt.img   1     512
/dev/loop0         0      0         1  1 /boot/bzfirmware                  0     512
root@unRAID:~# LOOPDEV_DEBUG=all losetup -vd /dev/loop2
24542: loopdev:      CXT: [0x7fffbefab7a0]: initialize context
24542: loopdev:      CXT: [0x7fffbefab7a0]: init: ignore ioctls
24542: loopdev:      CXT: [0x7fffbefab7a0]: init: loop-control detected
24542: loopdev:      CXT: [0x7fffbefab7a0]: /dev/loop2 name assigned
24542: loopdev:      CXT: [0x7fffbefab7a0]: open /dev/loop2 [ro]: No such file or directory
24542: loopdev:      CXT: [0x7fffbefab7a0]: device removed
24542: loopdev:      CXT: [0x7fffbefab7a0]: de-initialize
24542: loopdev:      CXT: [0x7fffbefab7a0]: closing old open fd
24542: loopdev:     ITER: [0x7fffbefab978]: de-initialize

 

  • Like 1
Link to comment
  • 2 weeks later...
On 1/30/2022 at 10:08 PM, mgutt said:

So the next user which has this problem should try to execute those two commands (while VM service has been stopped), loopX must be replaced by the output of "losetup -l":

losetup -l
LOOPDEV_DEBUG=all losetup -vd /dev/loopX

 

By that we can verify if it's "No such file" in any case.

 

Another thing which I was not able to test is to stop the array and to check how it does influence the "losetup -l" output. And the next time I like to test if the installation of kpartx and the command "kpartx -d /dev/loopX" could help.

 

I encoutered the issue again today.

 

Here is the output of what you suggested:

 

BEFORE STOPPING THE ARRAY:

root@MOZART:~# losetup -l
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                             DIO LOG-SEC
/dev/loop1         0      0         1  1 /boot/bzfirmware                        0     512
/dev/loop2         0      0         1  0 /mnt/cache/system/libvirt/libvirt.img   1     512
/dev/loop0         0      0         1  1 /boot/bzmodules                         0     512
/dev/loop3         0      0         1  0 /mnt/cache/system/docker/docker.img     1     512
root@MOZART:~# LOOPDEV_DEBUG=all losetup -vd /dev/loop2
24350: loopdev:      CXT: [0x7fffc39eaf60]: initialize context
24350: loopdev:      CXT: [0x7fffc39eaf60]: init: ignore ioctls
24350: loopdev:      CXT: [0x7fffc39eaf60]: init: loop-control detected 
24350: loopdev:      CXT: [0x7fffc39eaf60]: /dev/loop2 name assigned
24350: loopdev:      CXT: [0x7fffc39eaf60]: open /dev/loop2 [ro]: No such file or directory
24350: loopdev:      CXT: [0x7fffc39eaf60]: device removed
24350: loopdev:      CXT: [0x7fffc39eaf60]: de-initialize
24350: loopdev:      CXT: [0x7fffc39eaf60]: closing old open fd
24350: loopdev:     ITER: [0x7fffc39eb0f0]: de-initialize
root@MOZART:~# 

 

AFTER STOPPING THE ARRAY:

root@MOZART:~# losetup -l
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE        DIO LOG-SEC
/dev/loop1         0      0         1  1 /boot/bzfirmware   0     512
/dev/loop0         0      0         1  1 /boot/bzmodules    0     512
root@MOZART:~# 

 

I hope this helps and that @limetech will be able to fix this issue.

 

Best,

OP

Link to comment
  • 3 months later...

Happened to me today too.

 

Restarted array / unraid.

Docker came up and so did a few VMs but some failed.

Tried to start manually, failed --> error, VNC no available port 

Looked in syslog --> Failed to add multicast for WSDD: Address already in use

Disabled VM service

Disabled Docker

re-enable VM service -- worked

re-enabled docker -- worked

Link to comment

Bumping this thread, Having the same issue today. decided to try restarting my vm service as my 2 vm's where SUPER slow (which i think is due to libvirt image being on disk in the array)(took 24hr+ for a windows 2016 install to finalize)
Set "Enable VMs" to yes and check log to see the same output of libvirt in use, and like all before, pids line up with my dockers.

Link to comment
  • 2 months later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.