Unfortunately the location where my unRAID server is located had a power outtage and my server is not attached to a UPS. Now after starting up the system again my docker page says "Docker Service failed to start.", and my VM with the uuid 11b7f5fd-88e3-2ad8-8f6f-b1c0734a078c is also acting off and not starting up properly. It is showing a "GNU GRUB Version 2.6" screen:

Apparently i have the CA Backup / Restore Appdata plugin installed but the automatic backup disabled, lol very helpful.

I read somewhere that the solution for the docker issue is to delete my docker.img file and recreate all containers. I don't even know which containers i had installed. Do i need to redo all configuration? I also cannot see my currently installed apps in the app store because the docker service is not working. I had around 30 or 40 containers as far as im aware so im not sure how to do this quickly and efficiently.

Can someone use my diagnostics to determine what is wrong and if this action is nessecary?

And how do i fix the VM?

Edit:
I do see this error in my log which im not sure what it means but it seems to have something to do with Docker:

May 20 21:38:27 Tower root: mount: /var/lib/docker: wrong fs type, bad option, bad superblock on /dev/loop3, missing codepage or helper program, or other error.

And i also see a couple lines like this:

May 20 21:38:27 Tower kernel: BTRFS error (device loop3): bad tree block start, mirror 1 want 16391684096 have 16866508800

tower-diagnostics-20250520-2122.zip

Edited May 20, 20251 yr by YourNightmar3
Attach screenshot.

Pool had issues with both devices in the past:

May 20 21:21:01 Tower kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 23646788, rd 227163, flush 2056662, corrupt 0, gen 0

May 20 21:21:01 Tower kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 168468466, rd 30680, flush 19078468, corrupt 0, gen 0

Scrub the pool and post the results.

Hi @jorge thanks for the help.

I ran a scrub on the cache pool with this result. I did not check "Repair corrupted blocks" because im not sure if i was supposed to.

UUID:             9accc309-fa94-42b6-9c0b-ca81d5254d87
Scrub started:    Wed May 21 09:18:13 2025
Status:           finished
Duration:         0:04:51
Total to scrub:   1.39TiB
Rate:             4.88GiB/s
Error summary:    verify=17220 csum=660592
  Corrected:      0
  Uncorrectable:  0
  Unverified:     0

New diags:

tower-diagnostics-20250521-0924.zip

1 hour ago, YourNightmar3 said:

I did not check "Repair corrupted blocks" because im not sure if i was supposed to.

Run it again with that checked.

Interesting... The last thing i saw was the scrub being at 98.9% but now my server is unreachable. Supposedly it crashed. I will be at the physical machine in a couple hours so i will be able to reboot it (my PiKVM broke), and i guess i will just try to run it again(?)

Edited May 21, 20251 yr by YourNightmar3

33 minutes ago, YourNightmar3 said:

and i guess i will just try to run it again(?)

Yep, and enable the syslog server, in case it crashes again.

I booted up the server again and did another scrub with this result:

UUID:             9accc309-fa94-42b6-9c0b-ca81d5254d87
Scrub started:    Wed May 21 17:22:23 2025
Status:           finished
Duration:         0:04:38
Total to scrub:   1.39TiB
Rate:             5.11GiB/s
Error summary:    csum=27371
  Corrected:      27371
  Uncorrectable:  0
  Unverified:     0

I then did another one just to see if there would be no errors anymore:

    UUID:             9accc309-fa94-42b6-9c0b-ca81d5254d87
    Scrub started:    Wed May 21 17:31:01 2025
    Status:           finished
    Duration:         0:03:56
    Total to scrub:   1.39TiB
    Rate:             6.02GiB/s
    Error summary:    no errors found

I then rebooted the server but the docker page still shows the same error.

Here is a new diag:

tower-diagnostics-20250521-1747.zip

All errors were correct, that's good, the docker image is corrupt and needs to be recreated:

https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file

Then:

https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications

Also see below if you have any custom docker networks:

https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks

Also recommend resetting the pool stats and keep monitoring for new errors:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/#findComment-700582

Ok all of that has fixed the docker error, thank you! But the VM still starts up to e GNU Grub screen. Is the VM image corrupt or something? Is there any way to fix that?

Edited May 22, 20251 yr by YourNightmar3

The vdisk may be corrupt due to the pool errors, the system shares have COW disabled, meaning also no checksums, you many need to restore the vdisk from a backup if available, but recommend enabling COW for those shares (I believe you would need to recreate them to change that)

Is having COW disabled a configuration error on my side or what would be the reason for this? And how do i recreate the system shares to enable COW? It does say "Set when adding new share only", i can't change it on the existing shares.

It was an old default, for a few years that it's been enabled by default.

8 minutes ago, YourNightmar3 said:

i can't change it on the existing shares.

I thought so, you would need to recreate the shares, backup and data there, disable the services, delete them, and Unraid would recreate it them when the services are re-enabled

Okay thanks i'll look into that.

I just had another crash while i really wasn't doing anything special. Syslog mirror to usb was still enabled so can you maybe see what caused this one? tower-diagnostics-20250523-1149.zip

Nothing obvious that I can see, I would also recommend updating Unraid to the latest stable, even from 6.12 series if you prefer to remain on that release.

Alright i have updated my unRAID version to 6.12.15 now. I do want to go to 7.0+ eventually but first i'd like to fix all of my issues before jumping to a major version update.

Since i had to redo my docker setup anyway, i switched from a docker img to a docker folder instead (Im not sure if this is related to the rest of this). Now i was just poking around on the commandline and i found this folder in /mnt/cache/docker/btrfs that is currently 427G. What is this folder and is it relatively normal for it to be this big? I do have 53 docker containers.

I tried checking the size of my containers with docker ps -a --size and i dont fully understand what the sized i see there means but the total sum of "vitual size" is around 52G.

the command docker system df gives this:

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          44        44        45.22GB   1.219GB (2%)
Containers      52        19        454.8MB   98.24MB (21%)
Local Volumes   4         4         328B      0B (0%)
Build Cache     0         0         0B        0B

I don't really have any idea what im doing with these commands, just trying to see if i can somehow figure out where that 427G comes from.

Can depend on the tool you are using, it can give a false result, not every tool reports the correct stats with btrfs, especially subvolumes, but I would recommend using a docker image, folders sometimes cause weird issues.