Help, all my dockers are missing

[Tabraxis]

Hello All,

 

So i woke up this morning and when to sign in to one of my VM's (to check some stats) and it was taking a long while, so i figure i'd login though the VNC directly. After logging into my site (and to my horror) all my dockers were not shown. I went directly to the docker page and found that nothing was there... literally nothing, no offline dockers, no add or remove, just a blank page. I've done a reboot on the server and still nothing, i've gone to the docker page and disabled docker, then enabled it, nothing. i've also disabled it, deleted the storage location (via the webpage), then enabled it again, still nothing.

 

Has this happened to anyone before? if so, is there something else i can try. 

 

I just found this in my logs, but i don't know what it means (other than BTBFS has/had an error of some kind). Could anyone explain further, or have a fix suggestion?

 

Mar 19 09:23:19 Gibson kernel: BTRFS info (device sdc1): read error corrected: ino 1 off 71995260928 (dev /dev/sdd1 sector 39913536)
Mar 19 09:23:19 Gibson kernel: BTRFS info (device sdc1): read error corrected: ino 1 off 71995265024 (dev /dev/sdd1 sector 39913544)
Mar 19 09:23:19 Gibson kernel: BTRFS info (device sdc1): read error corrected: ino 1 off 71995269120 (dev /dev/sdd1 sector 39913552)
Mar 19 09:23:19 Gibson kernel: BTRFS info (device sdc1): read error corrected: ino 1 off 71995273216 (dev /dev/sdd1 sector 39913560)
Mar 19 09:23:19 Gibson kernel: BTRFS error (device sdc1): parent transid verify failed on 71995277312 wanted 1364297 found 1351260
Mar 19 09:23:19 Gibson kernel: BTRFS info (device sdc1): read error corrected: ino 1 off 71995277312 (dev /dev/sdd1 sector 39913568)
Mar 19 09:23:19 Gibson kernel: BTRFS info (device sdc1): read error corrected: ino 1 off 71995281408 (dev /dev/sdd1 sector 39913576)
Mar 19 09:23:19 Gibson kernel: BTRFS info (device sdc1): read error corrected: ino 1 off 71995285504 (dev /dev/sdd1 sector 39913584)
Mar 19 09:23:19 Gibson kernel: BTRFS info (device sdc1): read error corrected: ino 1 off 71995289600 (dev /dev/sdd1 sector 39913592)
Mar 19 09:23:19 Gibson kernel: BTRFS error (device sdc1): parent transid verify failed on 71995310080 wanted 1364297 found 1342776
Mar 19 09:23:19 Gibson kernel: BTRFS info (device sdc1): read error corrected: ino 1 off 71995310080 (dev /dev/sdd1 sector 39913632)
Mar 19 09:23:19 Gibson kernel: BTRFS info (device sdc1): read error corrected: ino 1 off 71995314176 (dev /dev/sdd1 sector 39913640)
Mar 19 09:23:19 Gibson kernel: BTRFS error (device sdc1): parent transid verify failed on 73533718528 wanted 1381390 found 1346780
Mar 19 09:23:20 Gibson kernel: BTRFS error (device sdc1): parent transid verify failed on 73933422592 wanted 1381253 found 1357623
Mar 19 09:23:20 Gibson kernel: BTRFS error (device sdc1): parent transid verify failed on 73783640064 wanted 1367864 found 1357608
Mar 19 09:23:20 Gibson kernel: BTRFS error (device sdc1): parent transid verify failed on 73329033216 wanted 1380278 found 1357795
Mar 19 09:23:20 Gibson kernel: BTRFS error (device sdc1): parent transid verify failed on 73385132032 wanted 1380850 found 1357801
Mar 19 09:23:20 Gibson kernel: BTRFS error (device sdc1): parent transid verify failed on 73393733632 wanted 1380850 found 1357802

[JorgeB]

Tools -> Diagnostics

[Tabraxis]

Hello Johnnie,

 

Please see attached diagnostics file. 

 

Thanks,

gibson-diagnostics-20170319-0957.zip

[JorgeB]

You're having problems with one of your cache devices (cache4), check cables and run an extended SMART test, if it passes run a scrub.

 

Then you'll need to recreate your docker image.

 

PS: having a mixed HDD/SSD pool is not ideal for best performance.

 

 

[Tabraxis]

Hello Johnnie,

 

Thank you for your time and suggestions. 

 

So after the extended self-test for cache drive 4, if it proves to be bad i'll replace it, will this affect my docker settings? ie, will i have to reconfigure each docker from fresh, or will the drop down with the saved configuration file still be available? 

 

Lastly, if i remove the two SSD drives and just use the HDD drives (or add another 1-2 of the same spec), will i have any issues where there is data on the SSD's that the system will be looking for. Or would a parity check resolve this maybe?

[JorgeB]

19 minutes ago, Tabraxis said:

So after the extended self-test for cache drive 4, if it proves to be bad i'll replace it, will this affect my docker settings? ie, will i have to reconfigure each docker from fresh, or will the drop down with the saved configuration file still be available? 

 

Disk should be OK but the errors suggest an issue prior to the current syslog, most likely a cable problem since SMART looks OK.

To restore your docker see:

 

 

20 minutes ago, Tabraxis said:

Lastly, if i remove the two SSD drives and just use the HDD drives (or add another 1-2 of the same spec), will i have any issues where there is data on the SSD's that the system will be looking for. Or would a parity check resolve this maybe?

 

Parity is for data disks only, you can remove devices from the cache pool following this guide:

 

 

[trurl]

54 minutes ago, Tabraxis said:

Or would a parity check resolve this maybe?

And in general a parity check won't do anything to recover missing data, it just checks parity. It is possible to rebuild a missing disk from parity, but this is not at all the same thing as a parity check. And as johnnie said, parity doesn't cover cache. That is sort of the point of cache, to avoid the performance hit of writing parity.