I recently moved my server into a proper rackmount enclosure, and upon doing so, I am unable to start my dockers, or write to either the cache or the array.  

 

I ran a Parity scan and it went through after 4 or so hours.  

 

Where in these diagnostics should I be looking?  

tower-diagnostics-20170912-1910.zip

Lots of errors on cache6 SSD, like this:

 

Sep 11 21:32:13 Tower kernel: blk_update_request: I/O error, dev sdj, sector 255078976
Sep 11 21:32:13 Tower kernel: BTRFS error (device sdf1): bdev /dev/sdj1 errs: wr 1521, rd 3, flush 60, corrupt 0, gen 0

 

These are hardware errors, probably from a bad cable/connection, check it and then run a correcting scrub making sure all errors were corrected.

Thank you for that.  I tried swapping the disks in the hot swap bays, and the errors moved to another drive, so it's for sure that cable.  I swapped the SAS cable, and ran a BTRFS scurb, and it's been going well so far.  There were some "uncorrectable errors" but it hasn't locked the filesystem to readonly.  

 

Only issue I have now is a couple Docker containers that won't update.  I had tried to update these while the system was not working well, and got the initial errors about readonly files system.  I guess I could just recreate the dockers, only Plex would be a annoying as I'd lose the metadata.  

 

Here's my error message:  

Error: failed to register layer: rename /var/lib/docker/image/btrfs/layerdb/tmp/layer-597596418 /var/lib/docker/image/btrfs/layerdb/sha256/18f9b4e2e1bcd5abe381a557c44cba379884c88f6049564f58fd8c10ab5733df: directory not empty

Delete and recreate the docker image, you can keep you dockers config: