checking the logs, it looks like it happened right around the time i was trying to restart a docker, not sure if that means anything. please see attached syslog.

 

after this, appears system went readonly and i had to restart via the power button. i was unable to telnet into the box. upon reboot, it happen again almost immediately, although ssh was ok. see syslog2

syslog.zip

it appears this guy might be having a similar issue as me: https://lime-technology.com/forum/index.php?topic=47177.0

 

going through some of the responses in his thread...

 

- the only btrfs drive i have is my cache drive. i did suspect possibly an issue with my docker, and my docker img is infact on my cache drive. SMART report on that is all good as far as i can tell (see attached).

- stopped array and started in maintenance mode and as far as i can tell it's working ok? i've never used maintenance mode before and i'm not sure what i'm looking for exactly. see attached syslog3 that picks up where syslog2 left off.

 

for now i will disable docker and see how it goes...

 

EDIT: disabled docker and rebooted. appears to be OK. i'll delete my docker image and start over, possibly got corrupted somehow.

tower-smart-20160316-0050.zip

syslog3.txt

I think you are correct.  Something went wrong with the Btrfs system on the docker.img, which is located in the btrfs system on the Cache drive.  As near as I can tell, it's the docker.img btrfs that's corrupted, not the Cache drive btrfs, so deleting docker.img and recreating it looks like the right solution.

 

You might want to run the btrfs scrub any way on the Cache drive, just to make sure it's clean.  After the unclean shutdown, transactions were replayed as normal, but there were extra lines associated with many of the replays.  I would recommend checking all of the file systems on your data drives, see Check Disk File systems.  Hopefully you won't find anything amiss.