Cache drive is not booting, how do I recover from my mirrored cache?

[EC28]

So I had a ton (3TB) of newgroups "article" downloading and woke up and my cache drive has possibly died.  Hopefully this is recoverable.  I do have a mirrored cache, but my the array isn't working of course due to an invalid config now and all of my docker and VM is gone.  Is it possible to recover?  Here is the diagnostics:

 

 

unraid-diagnostics-20231112-1030.zip

[EC28]

Anybody?  Don't want to make a move until I get some guidance.

[JorgeB]

Post the output of

btrfs fi show

 

[EC28]

1 hour ago, JorgeB said:

Post the output of

btrfs fi show

 

btrfs fi show

root@Unraid:~# btrfs fi show
warning, device 1 is missing
Label: none  uuid: f5db4bab-e8c2-4c8b-902d-aac6be164202
        Total devices 2 FS bytes used 646.66GiB
        devid    2 size 1.82TiB used 683.01GiB path /dev/nvme0n1p1
        *** Some devices missing

 

[JorgeB]

Try power/cycling the server to see if the missing device comes back. If the pool was mirrored (raid1) you can start with a device missing, it would still work, assuming the pool didn't have any other issues.

[EC28]

Tried that already but it's worth a shot doing it again.

 

Used the powerdown command to fully shut it off and I'll give it a few minutes to see if it'll help.

 

Thanks for jumping in to help.

[EC28]

Ok, somehow it found it now.  Docker didn't start though.  Going to try to start it manually and maybe reboot again.

[JorgeB]

Post new diags after array start if it doesn't work.

[EC28]

Yeah, docker won't start for some reason.  My VM is working fine.

unraid-diagnostics-20231113-0758.zip

[EC28]

 

Edited November 13, 2023 by EC28
No longer needed

[JorgeB]

Scrub the pool to bring the dropped device up to sync, then if there are no uncorrectable errors re-create the docker image, also see here for better pool monitoring, the device dropped a while ago.

[EC28]

Do I check repair corrupted blocks?

[JorgeB]

Yep.

[EC28]

Gotcha.  Looks like about a 6 hour run.  We'll see.  Thanks for all your help so far.

[EC28]

Starting to see some strange things going on with it.  Is this ok?

[EC28]

I think that fixed me up sir.  Thank you so much for your help.

 

I created the user script to run the scrub every week and check for errors.  Anything else I need to do?

[EC28]

One thing I've noticed is that when I run the script to check for errors, it's still showing all the errors from earlier.  Any thoughts on that?

[JorgeB]

All errors were corrected, that's good, and in the FAQ entry it mentions how to reset the stats.

[EC28]

That fixed it, thanks!

[EC28]

So....it happened again.  Guessing I might have an issue with that drive?  I was able to recover it the same as before, but this is popping up now.  And I'm not seeing that plugin in the nerdtools pack.

 

unraid-diagnostics-20231118-0842.zip

[JorgeB]

Assuming the stats were reset the device dropped offline again:

 

Nov 16 21:50:23 Unraid kernel: BTRFS info (device sdb1): bdev /dev/sdb1 errs: wr 92704976, rd 33512, flush 67810, corrupt 0, gen 0

 

If you haven't yet replace both cables, if it keeps dropping replace the device.

[EC28]

Kind of what I was figuring, thanks.