System freeze up

[shadoh]

Hey all,  I've been trying to resolve this for a couple weeks now.

 

System runs fine, roughly two days in the dockers begin to fail, can't stop them, can't restart them, cant turn docker off. Eventually the unraid web UI goes south. Hadn't been able to export logs during these situations,I reboot with IPMI, and it's fine for two days. After a few times of this, my usb stick failed to be bootable. I reimaged it and booted in, accidentally overwrote one of my drives when I added it as parity... I've pulled down 16TB of data to recover since and haven't had any issues. Until now, roughly two days later my dockers locked up, but I was able to snag logs and view some errors. Can't stop docker, can't stop array, can't gracefully shut down. 

 

Docker error: file system read only (lost the actual message)

BTRFS critical (device loop2): corrupt leaf

BTRFS: error (device sdc1) 

screen shot attached.

 

Drive layout:

9xHDD (1 parity and 8 data) XFS

1xSSD (cache) BTRFS

 

I blew away the docker.img and rebuilt it to try and fix the corrupt leaf error  a few days ago. Seems it didn't help. I've also done scrub on the BTRFS drive.

 

Device sdc1 is my SSD and set to be my cache drive holding appdata/downloads/transcode.

 

I'm guessing I need to replace the SSD, but happy to be wrong.. any guidance of the best way to go about it?

 

 

tower-diagnostics-20200522-1618.zip

Edited May 22, 2020 by shadoh

[shadoh]

Reboot of the server couldn't find any Dockers.

 

Removing cache, formatting, and swapping it to XFS to see if there is any improvement.

[JorgeB]

Constant crashing ton the logs, if you haven't yet run memtest.