Dockers missing, Appdata empty

[jdubbs23]

Server has been up for days/weeks/months not really sure, but haven't been on it in a few days. Logged in today to start a Docker container and got a Server Error pop up window. Tried a few other Dockers as well and same issue. VMs were all fine.

 

Did a restart and now all Dockers are gone along with all VMs. Appdata folder is empty.

tower-diagnostics-20211207-2139.zip

[jdubbs23]

OK I noticed my cache drive was in unassigned devices for some reason. I stopped the array and looked at the config and it wasn't assigning the cache drive to the cache slot. So I assigned it back in, started array and it says it can't mount it:

 

[trurl]

attach diagnostics to your NEXT post in this thread

[jdubbs23]

It was attached to my first post?

tower-diagnostics-20211207-2139.zip

Edited December 8, 2021 by jdubbs23

[trurl]

22 minutes ago, jdubbs23 said:

It was attached to my first post?

I was hoping for new diagnostics since you had changed something since your first post.

[jdubbs23]

Ah ok sorry, and thanks for your help with this. Just to clarify:

1. when I first went to the server and discovered I couldn't start the Dockers I did a restart with no diagnostics (I realize that was wrong).
2. Then upon the restart I just started the array and didn't take a close look at the drive assignments but captured the diagnostics then, that is what I attached to first post.
3. While the array was started I noticed the cache drive was in unassigned devices and there was effectively no cache drive. I stopped the array and then reassigned the cache drive to the cache slot. I didn't see any warnings about data loss, and it said "configuration is valid" so I started the array again.
4. Upon this array start with the cache drive in place I see the btrfs error from the screen cap.
5. I got nervous so I stopped the array again and here we are.

Attached is the latest diagnostics from where I am currently (step 5).

tower-diagnostics-20211208-1021.zip

[trurl]

3 minutes ago, jdubbs23 said:

I got nervous so I stopped the array again and here we are

Usually diagnostics with the array started would be better, but I can use those with the others.

 

Is your cache supposed to be using encryption too?

[jdubbs23]

Is it safe to start the array with the drive reassigned back to the cache slot? I wasn't sure, but can do that and restart the array.

 

I can't remember if I did encryption for the cache drive when I went through the whole process to encrypt the drives. I want to say that I remember not encrypting that one but I can't be sure. When I reassigned the drive to the cache slot it didn't give me any prompts about filesystem or encryption or anything. I had already passed the passphrase to unraid during the first array start after the reboot so I figured the cache drive would use that passphrase if the FS was encrypted.

[trurl]

Your diagnostics show that it isn't trying to use encryption for cache so that is why I asked if it was supposed to.

[jdubbs23]

Appreciate you looking into this for me, just wondering if you had any other suggestions. I don't want to lose the data on the flash drive, but I'm not sure how to go about recovering or if there is anything even wrong or if it just some config issue that got messed up.

[JorgeB]

Unassign the cache device, try to mount it with UD plugin and post new diags and the output of

fdisk -l /dev/sdX

if it fails to mount .

[jdubbs23]

OK Here is the diagnostics with the array started and the cache drive unassigned. I can't do anything with it in unassigned devices:

 

 

root@tower:~# fdisk -l /dev/sdm
Disk /dev/sdm: 298.09 GiB, 320072933376 bytes, 625142448 sectors
Disk model: SAMSUNG HD321KJ
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@tower:~#

 

 

tower-diagnostics-20211210-1012.zip

[JorgeB]

There's no partition on that device, so it was deleted or something else happened, you can try testdisk or other recovery tool to see if you can get it back.

[jdubbs23]

Any idea how this can possibly happen? A fully functional array that isn't even being used most of the time just randomly deletes a partition on the cache drive?

[JorgeB]

2 minutes ago, jdubbs23 said:

Any idea how this can possibly happen?

Other than it being deleted (on purpose or not), not really.

[jdubbs23]

1 minute ago, JorgeB said:

Other than it being deleted (on purpose or not), not really.

That's what I mean, I'm the only one that accesses the array and obviously didn't delete the partition, so I don't understand how this could happen.

[jdubbs23]

Well I did use the CA Backup and Restore plugin, so I can see I have a backup of Appdata from 11/27 along with flash and system. I guess the VMs needed to be backed up manually so I just see one of the VMs there from 1.5 years ago (must have did a manual one then).

 

So should I do a restore of the Appdata to at least get Dockers back? I guess the VMs are gone.

[trurl]

You should get cache installed before trying to get dockers/VMs back, since these should be on fast storage where they will perform better and not keep array disks spunup

[jdubbs23]

Yes I would format the current cache drive and add it back into the array and then do the restore. Just wanted to confirm this course of action.

 

Is there something about the drive I was using for cache that could be failing and why the partition is gone?

[trurl]

28 minutes ago, jdubbs23 said:

restore of the Appdata

After that you would use Previous Apps on the Apps page to reinstall your dockers exactly as they were

 

https://wiki.unraid.net/Manual/Docker_Management#Re-Create_the_Docker_image_file

[jdubbs23]

Well I feel like an idiot, but that was the wrong drive. What got me thinking about it was the size (not typical SSD size), and after running the fdisk utility the name (spinpoint) made me think of a spinning drive and I knew I had an SSD for the cache. 
 

So it seems that something happened with the PCIe adapter board that the SSD is connected to, because I opened the case and reseated it and then the drive showed up, I assigned it to cache, and all is fine. 
 

Thanks to @trurl and @JorgeB for all the help troubleshooting this with me, and sorry for the incorrect info and my mistakes/forgetfulness.