Unable to write to Docker Image file

[sgt_spike]

Need help figuring out what's going on with docker.  I had updates to some container and just to update them.  I got an error, can't remember what, so I ran common fix which stated "Unable to Write to Docker Images file".  Searching for this error in the forum told me to delete my image file and reinstall the apps again.  I deleted the docker image file created a new one and started the reinstall.  I received the same error as before, so I ran common fix again and got the same "Unable to Write to Docker Images file" error again.  At this point I'm not sure what to do.

 

Diagnostics attached

 

Thanks in advance

tower-diagnostics-20180920-1639.zip

[JorgeB]

There are a million errors like these on your cache device:

 

Sep 19 18:45:58 Tower kernel: ata5.00: status: { DRDY }
Sep 19 18:45:58 Tower kernel: ata5.00: failed command: WRITE FPDMA QUEUED
Sep 19 18:45:58 Tower kernel: ata5.00: cmd 61/00:20:80:1d:14/0a:00:02:00:00/40 tag 4 ncq dma 1310720 ou
Sep 19 18:45:58 Tower kernel:         res 40/00:2c:80:27:14/00:00:02:00:00/40 Emask 0x10 (ATA bus error)
Sep 19 18:45:58 Tower kernel: ata5.00: status: { DRDY }
Sep 19 18:45:58 Tower kernel: ata5.00: failed command: WRITE FPDMA QUEUED
Sep 19 18:45:58 Tower kernel: ata5.00: cmd 61/c0:28:80:27:14/05:00:02:00:00/40 tag 5 ncq dma 753664 out
Sep 19 18:45:58 Tower kernel:         res 40/00:2c:80:27:14/00:00:02:00:00/40 Emask 0x10 (ATA bus error)
Sep 19 18:45:58 Tower kernel: ata5.00: status: { DRDY }
Sep 19 18:45:58 Tower kernel: ata5.00: failed command: WRITE FPDMA QUEUED
Sep 19 18:45:58 Tower kernel: ata5.00: cmd 61/00:30:20:30:14/0a:00:02:00:00/40 tag 6 ncq dma 1310720 ou
Sep 19 18:45:58 Tower kernel:         res 40/00:2c:80:27:14/00:00:02:00:00/40 Emask 0x10 (ATA bus error)
Sep 19 18:45:58 Tower kernel: ata5.00: status: { DRDY }
Sep 19 18:45:58 Tower kernel: ata5.00: failed command: WRITE FPDMA QUEUED
Sep 19 18:45:58 Tower kernel: ata5.00: cmd 61/00:38:20:3a:14/06:00:02:00:00/40 tag 7 ncq dma 786432 out
Sep 19 18:45:58 Tower kernel:         res 40/00:2c:80:27:14/00:00:02:00:00/40 Emask 0x10 (ATA bus error)
Sep 19 18:45:58 Tower kernel: ata5.00: status: { DRDY }

Replace cables and check if they go way.

[sgt_spike]

13 hours ago, johnnie.black said:

There are a million errors like these on your cache device:

 

Sep 19 18:45:58 Tower kernel: ata5.00: status: { DRDY }
Sep 19 18:45:58 Tower kernel: ata5.00: failed command: WRITE FPDMA QUEUED
Sep 19 18:45:58 Tower kernel: ata5.00: cmd 61/00:20:80:1d:14/0a:00:02:00:00/40 tag 4 ncq dma 1310720 ou
Sep 19 18:45:58 Tower kernel:         res 40/00:2c:80:27:14/00:00:02:00:00/40 Emask 0x10 (ATA bus error)
Sep 19 18:45:58 Tower kernel: ata5.00: status: { DRDY }
Sep 19 18:45:58 Tower kernel: ata5.00: failed command: WRITE FPDMA QUEUED
Sep 19 18:45:58 Tower kernel: ata5.00: cmd 61/c0:28:80:27:14/05:00:02:00:00/40 tag 5 ncq dma 753664 out
Sep 19 18:45:58 Tower kernel:         res 40/00:2c:80:27:14/00:00:02:00:00/40 Emask 0x10 (ATA bus error)
Sep 19 18:45:58 Tower kernel: ata5.00: status: { DRDY }
Sep 19 18:45:58 Tower kernel: ata5.00: failed command: WRITE FPDMA QUEUED
Sep 19 18:45:58 Tower kernel: ata5.00: cmd 61/00:30:20:30:14/0a:00:02:00:00/40 tag 6 ncq dma 1310720 ou
Sep 19 18:45:58 Tower kernel:         res 40/00:2c:80:27:14/00:00:02:00:00/40 Emask 0x10 (ATA bus error)
Sep 19 18:45:58 Tower kernel: ata5.00: status: { DRDY }
Sep 19 18:45:58 Tower kernel: ata5.00: failed command: WRITE FPDMA QUEUED
Sep 19 18:45:58 Tower kernel: ata5.00: cmd 61/00:38:20:3a:14/06:00:02:00:00/40 tag 7 ncq dma 786432 out
Sep 19 18:45:58 Tower kernel:         res 40/00:2c:80:27:14/00:00:02:00:00/40 Emask 0x10 (ATA bus error)
Sep 19 18:45:58 Tower kernel: ata5.00: status: { DRDY }

Replace cables and check if they go way.

Replaced the SATA cable on the cache drive. 

 

Still having various issues with the cache drive.  Rebooting the server took quite a long time.  It started the array but hung on "starting services".  Not sure if that was because of the read/write issues I am having with the cache drive.  My docker containers and vms files all reside on my cache drive.  To get array completely up and running I had to change my cache settings in Global Share Settings start the array then change it back.  So far everything seems to be working.

 

My concern is the cache drive.  

 

I've attached the smart report along with the diag after the cable was replaced.

tower-diagnostics-20180921-0847.zip

tower-smart-20180921-0823.zip

[JorgeB]

All seems fine now, no errors so far.

[sgt_spike]

1 minute ago, johnnie.black said:

All seems fine now, no errors so far.

That's great, but not ready to stop worrying just yet.

 

What do I need to look for if things start to go haywire again.  What in the logs shows you all is fine?  What should I be looking for?

 

Also on the cache drive page, there is a warning about udma crc error count.  Will this eventually correct itself?

[JorgeB]

19 minutes ago, sgt_spike said:

Will this eventually correct itself?

No, those can't be reset, but as long and they don't increase you're fine, as to what to look out for periodically check the syslog for ATA errors, similar to the ones I post above.

[sgt_spike]

Seems to be doing it again and the crc error count is increasing

 

Sep 21 10:39:25 Tower kernel: ata5.00: exception Emask 0x10 SAct 0xc SErr 0xc00100 action 0x6 frozen
Sep 21 10:39:25 Tower kernel: ata5.00: irq_stat 0x08000000, interface fatal error
Sep 21 10:39:25 Tower kernel: ata5: SError: { UnrecovData Handshk LinkSeq }
Sep 21 10:39:25 Tower kernel: ata5.00: failed command: WRITE FPDMA QUEUED
Sep 21 10:39:25 Tower kernel: ata5.00: cmd 61/00:10:f8:e0:31/0a:00:02:00:00/40 tag 2 ncq dma 1310720 ou
Sep 21 10:39:25 Tower kernel: res 40/00:14:f8:e0:31/00:00:02:00:00/40 Emask 0x10 (ATA bus error)
Sep 21 10:39:25 Tower kernel: ata5.00: status: { DRDY }
Sep 21 10:39:25 Tower kernel: ata5.00: failed command: WRITE FPDMA QUEUED
Sep 21 10:39:25 Tower kernel: ata5.00: cmd 61/00:18:f8:ea:31/06:00:02:00:00/40 tag 3 ncq dma 786432 out
Sep 21 10:39:25 Tower kernel: res 40/00:14:f8:e0:31/00:00:02:00:00/40 Emask 0x10 (ATA bus error)
Sep 21 10:39:25 Tower kernel: ata5.00: status: { DRDY }
Sep 21 10:39:25 Tower kernel: ata5: hard resetting link
Sep 21 10:39:26 Tower kernel: ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Sep 21 10:39:26 Tower kernel: ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT4._GTF.DSSP], AE_NOT_FOUND (20180531/psargs-330)
Sep 21 10:39:26 Tower kernel: ACPI Error: Method parse/execution failed \_SB.PCI0.SAT0.SPT4._GTF, AE_NOT_FOUND (20180531/psparse-516)
Sep 21 10:39:26 Tower kernel: ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT4._GTF.DSSP], AE_NOT_FOUND (20180531/psargs-330)
Sep 21 10:39:26 Tower kernel: ACPI Error: Method parse/execution failed \_SB.PCI0.SAT0.SPT4._GTF, AE_NOT_FOUND (20180531/psparse-516)
Sep 21 10:39:26 Tower kernel: ata5.00: configured for UDMA/133
Sep 21 10:39:26 Tower kernel: ata5: EH complete
Sep 21 10:40:01 Tower crond[1803]: exit status 127 from user root /usr/lib/sa/sa1 1 1 &> /dev/null

[JorgeB]

Then there's still a problem, usually the SATA cable, did you replace it?

[JorgeB]

It can also be a problem with the power cable, or the SATA port, in this order, very unlikely it's a problem with the SSD itself.

[sgt_spike]

Just now, johnnie.black said:

Then there's still a problem, usually the SATA cable, did you replace it?

sure did. 

 

The drive is attached to a PCIe m.2 sata adapter.  could it be the PCIe card or maybe the PCIe slot?

[JorgeB]

4 minutes ago, sgt_spike said:

could it be the PCIe card or maybe the PCIe slot?

Not the PCIe slot, it's a SATA device so all communication is done by the SATA cable, but it could be a problem with the adapter, the link from power/SATA plugs to the M.2 device.