Docker and VMs Won't Start

[jessetn]

Was having some weird issues with a few containers, ultimately didn't really have time to dive in troubleshooting so I rebooted the server.

Parity check started automatically after reboot, now docker and vm service both won't start. So I disable both while parity check ran.

2 days later after parity check finishes and 100k+ error corrections, Docker and VM services still won't start.

Jul  1 08:28:54 kernel: BTRFS info (device loop2): using crc32c (crc32c-intel) checksum algorithm
Jul  1 08:28:54 kernel: BTRFS info (device loop2): using free space tree
Jul  1 08:28:54 kernel: BTRFS error (device loop2): parent transid verify failed on logical 30556160 mirror 1 wanted 937 found 935
Jul  1 08:28:54 kernel: BTRFS error (device loop2): parent transid verify failed on logical 30556160 mirror 2 wanted 937 found 935
Jul  1 08:28:54 kernel: BTRFS warning (device loop2): failed to read root (objectid=4): -5
Jul  1 08:28:54 kernel: BTRFS error (device loop2): open_ctree failed
Jul  1 08:28:54 root: mount: /etc/libvirt: wrong fs type, bad option, bad superblock on /dev/loop2, missing codepage or helper program, or other error.
Jul  1 08:28:54 root:        dmesg(1) may have more information after failed mount system call.
Jul  1 08:28:54 root: mount error

 

Pretty sure I'll need to recreate docker.img, I've had to before, but this is the first time libvirt also won't start. 
I'm aware I had/have a lidarr configuration problem which is the cause of my docker image being large. Just going to end up removing it when I get docker working again.

diagnostics-20240701-0915.zip

[JorgeB]

Librvirt image is corrupt, do you have a backup?

[JorgeB]

Docker image is also corrupt, but that one can easily be recreated.

[jessetn]

16 minutes ago, JorgeB said:

Librvirt image is corrupt, do you have a backup?

I don't have a backup of the libvirt.img but I do of the actual VM.

From the logs, is there a way to tell how the corruption happened so I can prevent it in the future? Didn't have any power loss or drive failures

[JorgeB]

33 minutes ago, jessetn said:

is there a way to tell how the corruption happened so I can prevent it in the future?

Looks like there were some lost writes, likely to the underlying device.

 

33 minutes ago, jessetn said:

I don't have a backup of the libvirt.img but I do of the actual VM.

If the vdisk is OK, you should be able to use a new libvirt image, then create a VM with the same settings and point to the existing vdisk.

 

Recommend keeping a backup of that file for the future.

[jessetn]

5 hours ago, JorgeB said:

Looks like there were some lost writes, likely to the underlying device.

 

If the vdisk is OK, you should be able to use a new libvirt image, then create a VM with the same settings and point to the existing vdisk.

 

Recommend keeping a backup of that file for the future.

Thank you for the help. Recreated the docker.img just fine, and after a bit of fiddling with VM settings, got my VM's back up using their vdisks.

 

For future me or other's reference, follow JorgeB's advice and make sure docker.img and libvirt.img's are backed up as well, not just appdata or your VM disks!