Docker issues after upgrading to 6.9.2

[cypres0099]

I upgraded unraid last week and have been running into issues with a few dockers not starting and not upgrading and then this happened. Can't get the docker service to start anymore. What can I do?

 

[cypres0099]

Fix common problems showing these issues.

 

[cypres0099]

Looking at it now, it seems my cache drive is completely full. 

[ChatNoir]

6 hours ago, cypres0099 said:

Looking at it now, it seems my cache drive is completely full. 

You should move data from your cache.

 

Maybe your issue has to do with your video share ? Is it possible that videos are downloaded on cache and moved to "video" ? Since the mover does not consider files with cache set to NO, they might pile up.

 

In anycase, your diagnostics might help provide better advices. Go to Tools / Diagnostics and attach the full zip to your next post.

[ChatNoir]

The System Notification should have alerted you of the cache pool getting past certain thresholds.

Is it not set up to send you emails or other type of messages ?

[cypres0099]

11 hours ago, ChatNoir said:

Maybe your issue has to do with your video share ? Is it possible that videos are downloaded on cache and moved to "video" ? Since the mover does not consider files with cache set to NO, they might pile up.

 

Thanks for chiming in @ChatNoir

 

That folder doesn't really have anything in it. I think it was setup just for pre-roll in plex for some reason. 

 

I've had this cache setup for like 6 years without issues, but all of a sudden after upgrading to 6.9.2 it's stopped moving files every night.

 

Now there's another new issue popping up, it says my disk 2 is disabled and stuck in read only or something.

I've attached the zip diagnostic

 

mongo-diagnostics-20210725-1448.zip

[cypres0099]

I'm doing a disk rebuild on that disabled disk, so I guess if that works fine, it's not that big of an issue.  Mostly just the mover not running now.

[cypres0099]

Ugh, now it's saying that Disk1 can't be written too. 

 

Disk 2 is currently being rebuilt. 

[cypres0099]

Disk 1 started throwing millions of errors during the disk 2 rebuild. 

Am I going to lose 20+ TB of data?

 

[cypres0099]

Most recent diagnostic 

mongo-diagnostics-20210725-1700.zip

[JorgeB]

Constant ATA errors on disks 1 and 3:

 

Jul 25 15:30:57 Mongo kernel: ata8: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Jul 25 15:30:57 Mongo kernel: ata8.00: configured for UDMA/33
Jul 25 15:30:57 Mongo kernel: ata8: EH complete
Jul 25 15:30:58 Mongo kernel: ata8.00: exception Emask 0x50 SAct 0x0 SErr 0x30802 action 0xe frozen
Jul 25 15:30:58 Mongo kernel: ata8.00: irq_stat 0x00400000, PHY RDY changed
Jul 25 15:30:58 Mongo kernel: ata8: SError: { RecovComm HostInt PHYRdyChg PHYInt }
Jul 25 15:30:58 Mongo kernel: ata8.00: failed command: READ DMA EXT
Jul 25 15:30:58 Mongo kernel: ata8.00: cmd 25/00:40:90:0a:b4/00:05:19:00:00/e0 tag 19 dma 688128 in
Jul 25 15:30:58 Mongo kernel:         res 50/00:00:8f:0a:b4/00:00:19:00:00/40 Emask 0x50 (ATA bus error)
Jul 25 15:30:58 Mongo kernel: ata8.00: status: { DRDY }
Jul 25 15:30:58 Mongo kernel: ata8: hard resetting link
Jul 25 15:30:58 Mongo kernel: ata6.00: exception Emask 0x50 SAct 0x0 SErr 0x30802 action 0xe frozen
Jul 25 15:30:58 Mongo kernel: ata6.00: irq_stat 0x00400000, PHY RDY changed
Jul 25 15:30:58 Mongo kernel: ata6: SError: { RecovComm HostInt PHYRdyChg PHYInt }
Jul 25 15:30:58 Mongo kernel: ata6.00: failed command: READ DMA EXT
Jul 25 15:30:58 Mongo kernel: ata6.00: cmd 25/00:00:50:15:b4/00:05:19:00:00/e0 tag 16 dma 655360 in
Jul 25 15:30:58 Mongo kernel:         res 50/00:00:4f:15:b4/00:00:19:00:00/40 Emask 0x50 (ATA bus error)
Jul 25 15:30:58 Mongo kernel: ata6.00: status: { DRDY }
Jul 25 15:30:58 Mongo kernel: ata6: hard resetting link

 

Cancel rebuild, replace cables on those disks and try again, monitor syslog to see if the errors are gone.

[ChatNoir]

I see that your drives are still using ReiserFS. This is not recommended anymore (especially for drives over a few TB); once your issues are fixed you should consider changing that.

 

The procedure is available there.

[cypres0099]

On 7/26/2021 at 3:24 AM, JorgeB said:

Constant ATA errors on disks 1 and 3:

 

Cancel rebuild, replace cables on those disks and try again, monitor syslog to see if the errors are gone.

 

@JorgeB the rebuild of Disk 2 did finish (not 100% sure if finished correctly). I hope I didn't screw myself by doing that

 

I only had one spare SATA cable so I replaced it on Disk 1 (which was the one throwing the errors). I rebooted and Disk 1 is showing disabled. If it was a bad cable that did it, how do I get the array to pull Disk 1 back in?

 

mongo-diagnostics-20210727-1755.zip

[cypres0099]

On 7/26/2021 at 11:57 AM, ChatNoir said:

I see that your drives are still using ReiserFS. This is not recommended anymore (especially for drives over a few TB); once your issues are fixed you should consider changing that.

 

The procedure is available there.

 

Will have to do that after I hopefully get it sorted!

[trurl]

4 minutes ago, cypres0099 said:

how do I get the array to pull Disk 1 back in?

rebuild it

[cypres0099]

1 minute ago, trurl said:

rebuild it

 

So I just want to be sure about something. 

I had Disk 2 go down last week, so I replaced Disk 2 and started a rebuild. However, during the rebuild Disk 1 started throwing errors and I couldn't access files on that disk. But the rebuild supposedly completed on Disk 2 (says lots of errors). 

 

If I now try a rebuild on Disk 1 and the Disk 2 data is corrupted, will I lose the data on both drives?

 

[trurl]

Just now, cypres0099 said:

replaced Disk 2

Do you mean you replaced it with another disk?

[cypres0099]

1 minute ago, trurl said:

Do you mean you replaced it with another disk?

Oh boy, I think I'm losing track of what disk went bad when.

A disk (pretty sure it was disk 2) went offline last week and I replaced it with a spare disk I had in case of this situation. While it was rebuilding, Disk 1 started throwing errors. 

[trurl]

Do you still have the original disk?

[cypres0099]

2 minutes ago, trurl said:

Do you still have the original disk?

 

Yes

[trurl]

What made you think the disk needed to be replaced? Connection problems are much more common than disk problems. Probably nothing at all wrong with the disk.

Can you mount it in Unassigned Devices?

[cypres0099]

6 minutes ago, trurl said:

What made you think the disk needed to be replaced? Connection problems are much more common than disk problems. Probably nothing at all wrong with the disk.

Can you mount it in Unassigned Devices?

 

I just didn't think through the problem very well. Saw a failed dis and thought "I have another disk to pop in there" so that's what I did :-/

I just hooked up the drive that I thought had failed to an external sata enclosure. It's now showing in unassigned devices.

 

[cypres0099]

This is what my current situation looks like. The unassigned drive (that i thought had failed) is accessible in my windows network folders and it's about the same size as Drive 1. So maybe I got confused about which drive was having the errors and replaced the wrong drive? So the unassigned drive is actually a still good version of drive 1 and I can just put it back in place of the red x Disk 1?

 

 

 

[trurl]

Do the contents of disk2 look OK after the rebuild?

 

Do you have backups of anything important and irreplaceable?

[trurl]

Any disk you assign as disk1 at this point is going to result in a rebuild to that disk.

 

Post new diagnostics since you mounted that original disk with Unassigned Devices and wait for further instructions.

 

Would have been better if you had asked for advice before doing anything.