How to recover lost appdata/config docker data

[paulbram]

I'm not entirely sure what happened. I think I had my appdata setup to prefer cache, and after a power failure when things came back online I noticed my containers weren't working. Upon closer inspection, appdata data was all gone. I didn't notice right away however so as I was working to start my HomeAssistant docker container etc, it started filling out their folders with the base/default configs etc and now I'm in a situation where everything appears to be healthy, but I still don't have my old configs. Is there any hope of finding/recovering this data at this point?

 

I've attached my diagnostics file, but I'm really just not sure where to start at this point.

 

Thanks

tower-diagnostics-20230602-1256.zip

Edited June 2, 2023 by paulbram

[paulbram]

I should add one detail. I did remove my cache drive from the pool thinking I might be able to manually access the data. I'm not sure if it's safe to add it back or if I should leave it disconnected for now.

[JorgeB]

Cache is unassigned, assuming Appdata was there it's normal for it to be missing now, re-assign cache and post new diags after array start.

[paulbram]

Yes, I removed the cache drive because I was afraid of making things worse. Now I've disabled docker, re-added the cache drive but my HomeAssistant appdata folder does not appear to be on my cache drive anymore. Since starting docker with the cache removed it looks to have created a blank set of configs on one of my other drives. 

 

I've attached my latest diags after re-adding my cache.

 

Thanks!

tower-diagnostics-20230603-1622.zip

[trurl]

cache doesn't seem to have much data on it. You must have reformatted it.

[paulbram]

That's the part I don't understand. I didn't reformat anything. I think what may have happened, is the drive ran into some sort of error immediately after the power loss. Then as I was trying to figure out what was wrong, the docker images recreated new configs in appdata since the old ones were missing. I guess the part I don't understand is why parity didn't protect me in this case? Is there a scenario where "prefer cache" will enable unprotected files?

[itimpi]

Files on the cache are outside the array and thus not protected by parity which only protects the drives in the main array.

[paulbram]

5 hours ago, itimpi said:

Files on the cache are outside the array and thus not protected by parity which only protects the drives in the main array.

Makes sense, so is it fair to assume that "prefer cache" is 100% unprotected? Is there any way to "prefer" the cache, but still have protection?

[Hoopster]

1 hour ago, paulbram said:

Is there any way to "prefer" the cache, but still have protection?

In a word "no" with respect to parity protection.

 

By definition, prefer cache means to store appdata or whatever is in the share on a pool device outside of the parity-protected array.  This is normally done to avoid the parity write speed penalty and to avoid spinning up hard drives as these cache/pool devices are often SSDs.  Even setting a share to cache Yes means the data is not parity protected while it is on cache/pool location until the Mover runs and moves it to the array.

 

I have appdata as a cache preferred share and my protection for it is a scheduled run of  the appdata backup plugin a couple of times a week.  My appdata backup location is an unassigned device that is outside the array and pools.

[itimpi]

4 hours ago, paulbram said:

Is there any way to "prefer" the cache, but still have protection?

If you want this then you need a multi-device pool set up for redundancy.  The default for a multi-device pool if using BTRFS is RAID1.

 

There are plugins available to back up docker container data and VMs to the array at specified intervals.

[Hoopster]

6 hours ago, itimpi said:

If you want this then you need a multi-device pool set up for redundancy.  The default for a multi-device pool if using BTRFS is RAID1.

@paulbram This is another option (BTRFS RAID1) for pool devices, but there have been cases where power outages/spikes affected all devices in the pool.  Although this is not typical, it can happen.

 

Basically your options are:

• Redundancy within the pool
• Backup of pool data outside the pool
• Both of the above