Disk Errors, Docker Failures, help me save what I can

[Agathon813]

Hi all,

I've recently had all kinds of errors pop up in my system. I have 14 disks in Unraid and my disk #8 had failed not long ago. I haven't been able to replace it yet and it's contents are being emulated. Unfortunately, now I'm having issues with two other disks I think (looks like #6 and #7) and I'm seeing weird Docker errors I've never run across before. If I reboot the system, everything will be fine for a while and eventually I will see this (completely random):

 

I've attached the diagnostics file here. 

 

If anyone can help me save whatever is possible and get my machine back in a stable condition, I would greatly appreciate it

asgard-diagnostics-20221025-0831.zip

[JorgeB]

Only issue I'm seeing for now is the cache device dropping offline and reconnecting again, reboot and post new diags after array start.

[Agathon813]

asgard-diagnostics-20221025-0950.zip

 

Thank you for getting back to me @JorgeB

I rebooted and have attached the new diag

[JorgeB]

Everything looks good so far, are the dockers containers working?

[Agathon813]

They are working fine and stay working after a reboot -- but it will not take very long and they will crash again like that screenshot I initially posted

[JorgeB]

If the cache device drops again they will crash, but that's unrelated to the array, check/replace cables for the cache device.

[Agathon813]

I spoke too soon -- they crashed.

[Agathon813]

1 minute ago, JorgeB said:

If the cache device drops again they will crash, but that's unrelated to the array, check/replace cables for the cache device.

Ok, I'll take a look at this. Am I able to just shutdown and move the cache drive into a different bay?

[JorgeB]

Should be.

[Agathon813]

4 hours ago, JorgeB said:

Should be.

Alright, so earlier I did this and moved the cache drive to another bay. Unfortunately, my issues are not resolved. I haven't seen the docker service error I had previously posted pop up again yet but the docker containers are intermittently unreachable. Every minute or two I will get SABnzbd saying it lost connection, my Plex server (different machine) will stop playing and freeze because it can't read the content off of the unraid box anymore - then after a few minutes it will resolve.

 

Do you have any other tips on what I should do? Here's another diag from just now, one of these intermittent blackouts just happened a minute ago.  I truly appreciate your assistance. Thank you

asgard-diagnostics-20221025-1530.zip

Edited October 25, 2022 by Agathon813

[JorgeB]

Nothing in the logs, you can try recreating the docker image and/or that specif docker container.

[Agathon813]

I know nothing shows in the logs. I'm still having these problems though. I am getting errors on 3 different disks now and I'm wondering if they could be causing the problem? Maybe I should just get rid of them. If I'm OK with losing the data on these three disks, which method of shrinking the array would you recommend? (They're pretty low usage, ~1.5TB total. I can restore the data by other means quicker than I can scatter the data with unBalance)

[Agathon813]

Follow-Up:

I went ahead and followed the "Remove Drives Then Rebuild Parity" Method.  Parity-Sync is in-progress now.  I'll report back after it's all done and see if I'm still getting the same errors I had before, without these 3 drives installed.  It's a 12TB Parity Drive so it's estimating about a full 24hours.

[JorgeB]

10 hours ago, Agathon813 said:

I am getting errors on 3 different disks now

Did you save the diags, or didn't reboot? Those errors must be logged. 

[Agathon813]

On 10/27/2022 at 2:53 AM, JorgeB said:

Did you save the diags, or didn't reboot? Those errors must be logged. 

Things have escalated, my parity sync is frozen and I'm having a *lot* more errors now.  Dashboard is also showing some funky numbers... like this

asgard-diagnostics-20221028-0857.zip

[JorgeB]

Cache device dropped offline, check/replace cables and try again.