When I woke up this morning, my home assistant VM was being odd, clicking some menus in the UI would not open anything. I decided to reboot the unraid server to see if that would fix it.

While rebooting, it showed the message:

Array Stopping Retry unmounting disk share(s). I waited it out and eventually it was able to reboot after a minute or so.

When it came back online, the cache drive was showing a size of 0B and a file system problem.

I did get an alert after rebooting that an unclean shutdown was detected, and it starting running a parity check that seemed to stall out.

I stopped and started the array in maintenance mode to run a check on the drive.

It showed this:

The cache drive is NVME and formatted for btrfs.

Attached are the diagnostics.

Any advice on how to fix/recover the drive?

I tried following this but got lost pretty quickly:

thearchive-diagnostics-20250608-1334.zip

Edited June 8, 20251 yr by VulpesDormiens
Cleaing up messy post and clarifying info

Reading the documentation here: https://docs.unraid.net/legacy/FAQ/check-disk-filesystems/
It states:

1. If the file system is XFS or ReiserFS (but NOT BTRFS), then you must start the array in Maintenance mode, by clicking the Maintenance mode check box before clicking the Start button. This starts the Unraid driver but does not mount any of the drives.

2. If the file system is BTRFS, then make sure the array is started, and NOT in Maintenance mode.

However, I am only able to run the check in maintenance mode, although it shows that the drive is btrfs.

That documentation also says that the scrub command should be used for btrfs, but I can't as it shows it is only available when the filesystem is mounted, and not being able to mount it is my problem.

I'm given this warning about running the Check with --repair

But I can't find anything else that I can do. Is there any other step I can take before doing this?

Type btrfs rescue zero-log /dev/nvme0n1p1 then restart the array and post new diags.

14 hours ago, JorgeB said:

Type btrfs rescue zero-log /dev/nvme0n1p1 then restart the array and post new diags.

Thanks for the reply! I ran that command, and attached are the new diagnostics.

Out of curiosity, where in the diagnostics are you looking? I tried to go through them, but was lost.

thearchive-diagnostics-20250609-1802.zip

Edit:

I realized after starting the array again that the cache drive mounted! And it looks like I can see the files and everything that should be there. I'm not sure what changed from all the messing around I did earlier that it works after re-staring the array this time.

Maybe the diagnostic can reveal an issue with the drive I can address.

However, the dockers and my VM are still not appearing, so I need to figure out how repair that.

Edited June 9, 20251 yr by VulpesDormiens

10 hours ago, VulpesDormiens said:

I'm not sure what changed from all the messing around I did earlier that it works after re-staring the array this time.

Well, it was the command I asked you to run, since the log was the problem.

Besides the bad log, btrfs is also detecting some data corruption, scrub the pool and post the results from the GUI, it may also be a good idea to run memtest.

3 hours ago, JorgeB said:

Well, it was the command I asked you to run, since the log was the problem.

Besides the bad log, btrfs is also detecting some data corruption, scrub the pool and post the results from the GUI, it may also be a good idea to run memtest.

Ahh, that makes sense.

I ran the scrub and here are the results:

Attaching diagnostics just in case, because I'll be gone at work.

Running memtest while I'm gone at work, but I'm getting some errors within the first minute:

thearchive-diagnostics-20250610-0809.zip

Edited June 10, 20251 yr by VulpesDormiens

25 minutes ago, VulpesDormiens said:

but I'm getting some errors within the first minute:

First thing to do it to fix the RAM, once that's resolved, scrub the pool again.

3 hours ago, VulpesDormiens said:

Running memtest while I'm gone at work, but I'm getting some errors within the first minute:

Since even 1 error from memtest is too many that needs resolving before attempting anything else.

11 hours ago, JorgeB said:

First thing to do it to fix the RAM, once that's resolved, scrub the pool again.

Ran a few different tests to determine 1 of my 4 sticks was bad. Ran a final test on the 3 good ones (will switch to a more optimal setup later) to confirm they were good.

Ran the scrub again and looks the same:

What would the next step be?

11 hours ago, JorgeB said:

First thing to do it to fix the RAM, once that's resolved, scrub the pool again.

Look in the syslog for the list of corrupt file(s), they should have been logged during the scrub, those will need to be deleted/replaced from a backup, then run another scrub to confirm 0 errors.

Awesome! Looks like it was just my VM that had corrupted files, which fortunately I do have backups for.

Now I can just re-install and restore that VM.

That leaves me with the docker images now. The appdata is still there on the drive, but looks like unraid does not know they are installed. Do you know how I would go about repairing that?

You can just recreate it:

https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file

Then:

https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications

Also see below if you have any custom docker networks:

https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks

10 hours ago, JorgeB said:

You can just recreate it:

https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file

Then:

https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications

Also see below if you have any custom docker networks:

https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks

Thank you so much Jorge! You're a life saver.

I remade the docker image following those instructions, and didn't even need to do the second part. All of my apps (and my VM which I don't understand how) reappeared and all seem to be working as they were before.