Lost appdata and cache data

[jameyho]

Good evening,

     I'm not sure when this happened, but this afternoon I went to my desk, and noticed that my server was actually off, which it never is. We had a power outage last night, and I think it lasted longer than my ups battery, so I had a dirty shutdown. I didn't think to check on it since in the past it's always made it thru outages. But upon reboot, docker would not launch. I went to the settings page, and my /mnt/user/system/docker/docker/docker-xfs.img and my /mnt/user/appdata/ paths are both missing.  I checked my shares, and the main data share is not listed either. I checked the drives, and all the data looks like it's still in place, but my cache drives seem to be missing the necessary files. I have a mirrored cache setup with 2x 4tb ssds. My assumption was that raid would allow me to recover from a failed cache disk, but I can't see the data when I open the cache directory. It shows 0files, 0folder, 0bytes, even though on the main page, it shows over 500gb in usage, which is where it normally sat with the docker.img files.

     I'm assuming one of the raid cache drives failed, i've been getting several notices about issues with one of them. Funny enough, I actually had my replacement already ordered, and was planning on doing maintenance tomorrow. All that aside, how can i recover from this? Do i just pull one of the cache drives, and see if unraid sees the data on the other drive? Am I just screwed and start over a new cache drive?

  

     I also had appdata backup/restore installed, but upon checking that, it apparently hasn't been running since June of last  year. So i need to look into that as well. Does anyone need logs, or is this an easy enough diagnosis/recovery?

 

Thank you!

Jamey

[trurl]

Attach diagnostics to your NEXT post in this thread.

[jameyho]

Diagnostics attached.

[JorgeB]

Where?

[jameyho]

I'll try again.

media-diagnostics-20240810-2138.zip

[JorgeB]

Looks like one of the cache devices is unassigned, stop the array, unassign cache2, start array. stop array, assign both cache devices, start array, and the pool should import.

[jameyho]

The cache is still showing 0files, 0folders, 0bytes, even though on the main page it shows 546gb used. Fix common problems says that the cache disk is either out of space or mounted as read only. It also stated it was going to erase the cache disk when I started the array after assigning both cache devices. Nothing has imported, but I still see the files on the array.

[JorgeB]

Post new diags.

[jameyho]

Attached.

 

My /mnt/user/system/ folder is missing as well, and I can not recreate the /mnt/user/appdata/ folder.

 

I had assumed that the cache drive in raid format, would allow for the failure of one drive, and have the ability to rebuild. It seems that's not the case, is there a chance at saving the cache drive from the second cache disk, or is that the disk that is failing?

media-diagnostics-20240812-0714.zip

[JorgeB]

You are having issues with both pool devices:

 

Aug 11 18:59:07 Media kernel: ata6.00: supports DRM functions and may not be fully accessible
Aug 11 18:59:07 Media kernel: ata10.00: supports DRM functions and may not be fully accessible
Aug 11 18:59:07 Media kernel: ata10.00: failed to enable AA (error_mask=0x1)
Aug 11 18:59:07 Media kernel: ata6.00: failed to enable AA (error_mask=0x1)
Aug 11 18:59:07 Media kernel: ata6.00: supports DRM functions and may not be fully accessible
Aug 11 18:59:07 Media kernel: ata6.00: failed to enable AA (error_mask=0x1)
Aug 11 18:59:07 Media kernel: ata10.00: supports DRM functions and may not be fully accessible
Aug 11 18:59:07 Media kernel: ata10.00: failed to enable AA (error_mask=0x1)
Aug 11 18:59:07 Media kernel: ata6.00: configured for UDMA/133 (device error ignored)
Aug 11 18:59:07 Media kernel: ata10.00: configured for UDMA/133 (device error ignored)

 

Check/replace cables, power and SATA, and post new diags after array start

[jameyho]

Reset all connections, checked all drives, not just the ssds. Everything seems connected as should be.

 

 

media-diagnostics-20240812-1914.zip

[JorgeB]

Both still showing errors, try replacing the cables.

[jameyho]

So that was fun, ran new cables and power cables for all 4 ssds, still no go.

media-diagnostics-20240813-2307.zip

[JorgeB]

They are both still showing the same errors, so possibly they failed.