I'm running into a problem on my server getting docker and VM's to start.

 

Yesterday all of my docker containers and VM's hard crashed and I wasn't able to get them restarted.

 

Every time I tried I got a generic 403 error and neither the containers or VM's would start.

 

I tried to reboot my server a couple times and the docker service fails to start. It currently look like the the VM's are working but It took a couple reboots and I'm pretty sure my Windows ones got corrupted. 

 

I've had a similar issue in the past and rebooting the server always fixed it, but it hasn't any more.

 

Because I've had similar issue docker containers before I swapped over to a new flash drive around 2 weeks ago. This is the first time I've had an issue on the flash drive.

 

I have two 32 Gb sticks of ram in my server and I booted the server with one stick in at a time to see if it was a ram issue. It didn't appear to have any impact.

 

I've attached a diagnostic file that was taken right after reboot when the docker services failed to start.

lab-diagnostics-20241117-1406.zip

Will review diag soon...

Based on your description, it sounds like there could be a corruption or misconfiguration issue affecting Docker, your flash drive, or the disk subsystem. Here’s how to systematically address the issue:

 

1. Analyze the Diagnostic File

If you have access to the diagnostic file:

Look for errors in:

/var/log/docker.log

/var/log/syslog

/boot/config/plugins/dockerMan/templates-user

Anything related to libvirt for VMs.

Common errors include:

Corruption in the Docker image file (e.g., docker.img).

Issues with the flash drive not mounting correctly.

Filesystem errors on the array/cache drive.

If you need help analyzing the log file, let me know, and I can guide you further.

 

2. Inspect the Docker Configuration

Check the Docker Image File:

The Docker service failing to start can often point to a corrupted docker.img file:

Go to Settings > Docker in the Unraid web interface.

Check the location of the Docker image file. If it's stored on a cache or array drive, ensure the drive is mounted and healthy.

If the file seems corrupted, delete it:

Stop the Docker service.

Delete docker.img from the cache drive.

Recreate it via Settings > Docker.

Preserve Container Data:

If you delete the docker.img, you won’t lose your app configurations, as they’re stored in /mnt/user/appdata. After recreating docker.img, re-add containers using the previous templates in Docker > Add Container.

 

3. Check Filesystem Health

The Docker and VM failures could stem from underlying filesystem issues. To check:

Stop the array.
BTRFS:

btrfs scrub start /mnt/cache

XFS:

xfs_repair -n /dev/sdX

then reboot unraid...

 

4. Verify Flash Drive Integrity

Your flash drive is critical to the server's operation. Problems with the flash drive can cause services to fail:

Connect the flash drive to another computer.

Run a filesystem check (e.g., chkdsk on Windows or fsck on Linux).

Reinstall Unraid onto the flash drive if issues are found:

Backup your current flash drive (especially config folder).

Create a new Unraid installation.

Restore the config folder to the new installation.

 

5. Investigate VM Issues

If Windows VMs have been corrupted, the following steps can help:

Check libvirt.img:

Navigate to Settings > VM Manager and check the path to libvirt.img.

If corrupted, you can recreate the image. Stop the VM service first, then delete and recreate it.

Restore VM Configurations:

Backup /mnt/user/system/libvirt/libvirt.img.

Recreate the VMs using the same disk images (vdisks).

Verify vdisk Health:

Use qemu-img check to inspect the health of vdisk files:

qemu-img check /mnt/user/domains/<VM_Name>/vdisk1.img

6. run memtest at unraid boot...
 

6. Check Logs for Hardware Issues

The hard crashes you described might be related to hardware:

Memory (RAM):

Use MemTest86 to check RAM for errors.

Storage:

Run SMART tests on all drives via Main > Disk > SMART Test.

Power Supply:

Verify the PSU is sufficient for your server’s configuration.
*Will post what i find if any soon...

 

7. Final Steps: Rebuild Services

Once Docker and VM services are running:

Re-add containers through Docker > Add Container.

Test each VM to confirm they boot correctly.

 

Potential Hardware / Disk failure is in the logs. I believe your systems is die due to btrfs error on disk.

Nov 17 12:06:04 Lab kernel: verify_parent_transid: 718 callbacks suppressed
Nov 17 12:06:04 Lab kernel: BTRFS error (device nvme0n1p1): parent transid verify failed on logical 808157184 mirror 1 wanted 423340 found 291850
Nov 17 12:06:04 Lab kernel: repair_io_failure: 3138 callbacks suppressed
Nov 17 12:06:04 Lab kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 808157184 (dev /dev/nvme0n1p1 sector 1578432)
Nov 17 12:06:04 Lab kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 808161280 (dev /dev/nvme0n1p1 sector 1578440)
Nov 17 12:06:04 Lab kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 808165376 (dev /dev/nvme0n1p1 sector 1578448)
Nov 17 12:06:04 Lab kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 808169472 (dev /dev/nvme0n1p1 sector 1578456)
Nov 17 12:06:05 Lab kernel: BTRFS error (device nvme0n1p1): parent transid verify failed on logical 594427904 mirror 1 wanted 812554 found 799910
Nov 17 12:06:05 Lab kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 594427904 (dev /dev/nvme0n1p1 sector 1160992)
Nov 17 12:06:05 Lab kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 594432000 (dev /dev/nvme0n1p1 sector 1161000)
Nov 17 12:06:05 Lab kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 594436096 (dev /dev/nvme0n1p1 sector 1161008)
Nov 17 12:06:05 Lab kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 594440192 (dev /dev/nvme0n1p1 sector 1161016)

run smart test and btrfs scrub

Both pool devices dropped offline in the past, run a correcting scrub and post the results.

Hello!

Can someone take a look at my file, I'm having the same issue with Docker Service failed to start.

unraid-tower-diagnostics-20251204-0034.zip

46 minutes ago, Nads said:

Can someone take a look at my file, I'm having the same issue with Docker Service failed to start.

Please don't make multiple posts about the same thing; already replied in your other thread.