(SOLVED) Docker is unresponsive / BTFRS error


Recommended Posts

My Docker applications stopped responding and gave errors, so I decided to restart the Docker service. Now it will not start again.

The docker.log file has some weird errors

time="2021-05-15T01:37:31.824854002+02:00" level=error msg="failed to shut down container" container=5737833477bf295c968a386f3a006e88fc5ab3a6d7a4d4ba2916f54f17afcbe5 error="Failed to stop container 5737833477bf295c968a386f3a006e88fc5ab3a6d7a4d4ba2916f54f17afcbe5 with error: Cannot kill container 5737833477bf295c968a386f3a006e88fc5ab3a6d7a4d4ba2916f54f17afcbe5: connection error: desc = \"transport: Error while dialing dial unix:///var/run/docker/containerd/containerd.sock: timeout\": unavailable"
time="2021-05-15T01:37:31.824854022+02:00" level=error msg="failed to shut down container" container=b648c97c1da2ae6af99a2786be6237cdae8251192de30d8a8dc963d174aa73bb error="Failed to stop container b648c97c1da2ae6af99a2786be6237cdae8251192de30d8a8dc963d174aa73bb with error: Cannot kill container b648c97c1da2ae6af99a2786be6237cdae8251192de30d8a8dc963d174aa73bb: connection error: desc = \"transport: Error while dialing dial unix:///var/run/docker/containerd/containerd.sock: timeout\": unavailable"
time="2021-05-15T01:37:31.824774012+02:00" level=error msg="failed to shut down container" container=0660c886da7b6383a8e626d3a1437132d4db999180362f294314ba7baa08457a error="Failed to stop container 0660c886da7b6383a8e626d3a1437132d4db999180362f294314ba7baa08457a with error: Cannot kill container 0660c886da7b6383a8e626d3a1437132d4db999180362f294314ba7baa08457a: connection error: desc = \"transport: Error while dialing dial unix:///var/run/docker/containerd/containerd.sock: timeout\": unavailable"
time="2021-05-15T01:37:31.824856887+02:00" level=error msg="failed to shut down container" container=41a3ba0c7b84d3c46e3538386cd88d26c8d0e5e964e7d6cc3e1e38002f851a28 error="Failed to stop container 41a3ba0c7b84d3c46e3538386cd88d26c8d0e5e964e7d6cc3e1e38002f851a28 with error: Cannot kill container 41a3ba0c7b84d3c46e3538386cd88d26c8d0e5e964e7d6cc3e1e38002f851a28: connection error: desc = \"transport: Error while dialing dial unix:///var/run/docker/containerd/containerd.sock: timeout\": unavailable"
time="2021-05-15T01:37:31.824816832+02:00" level=error msg="failed to shut down container" container=f97bc8fe5c742958c6134cd10b4c4c9567a75444c87f4af9a050e5739da25b53 error="Failed to stop container f97bc8fe5c742958c6134cd10b4c4c9567a75444c87f4af9a050e5739da25b53 with error: Cannot kill container f97bc8fe5c742958c6134cd10b4c4c9567a75444c87f4af9a050e5739da25b53: connection error: desc = \"transport: Error while dialing dial unix:///var/run/docker/containerd/containerd.sock: timeout\": unavailable"
time="2021-05-15T01:37:31.825494052+02:00" level=error msg="Cannot get orphan layers: open /var/lib/docker/image/btrfs/layerdb/sha256: input/output error"
time="2021-05-15T01:37:33.299838481+02:00" level=error msg="Error replicating health state for container d15743aa54acaf4815433823bb12e36f9de9927a29fe4c3242f0fee9b2a633c8: open /var/lib/docker/containers/d15743aa54acaf4815433823bb12e36f9de9927a29fe4c3242f0fee9b2a633c8/.tmp-config.v2.json507327996: read-only file system"
time="2021-05-15T01:37:33.299840415+02:00" level=error msg="Error replicating health state for container d41e3ee8e950758c4409c6122512e249f8bc23e0a2b2f15ce68f29d22836c468: open /var/lib/docker/containers/d41e3ee8e950758c4409c6122512e249f8bc23e0a2b2f15ce68f29d22836c468/.tmp-config.v2.json883396817: read-only file system"
time="2021-05-15T01:37:33.299838601+02:00" level=error msg="Error replicating health state for container f97bc8fe5c742958c6134cd10b4c4c9567a75444c87f4af9a050e5739da25b53: open /var/lib/docker/containers/f97bc8fe5c742958c6134cd10b4c4c9567a75444c87f4af9a050e5739da25b53/.tmp-config.v2.json500534970: read-only file system"

Also errors in syslog reporting BTRFS error

May 15 01:43:56 whale kernel: blk_update_request: I/O error, dev loop2, sector 75840 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
May 15 01:43:56 whale kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 33, flush 0, corrupt 0, gen 0
May 15 01:43:56 whale kernel: blk_update_request: I/O error, dev loop2, sector 180672 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
May 15 01:43:56 whale kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 34, flush 0, corrupt 0, gen 0
May 15 01:45:00 whale kernel: blk_update_request: I/O error, dev loop3, sector 2199952 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
May 15 01:45:00 whale kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 23, rd 6488, flush 1, corrupt 0, gen 0
May 15 01:45:00 whale kernel: blk_update_request: I/O error, dev loop3, sector 2199952 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
May 15 01:45:00 whale kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 23, rd 6489, flush 1, corrupt 0, gen 0

I have also attached system diagnostics.

 

My appdata share is not showing under SHARES. And listing appdata in the CLI
 

root@whale:~# ls /mnt/user/appdata
/bin/ls: cannot access '/mnt/user/appdata': Input/output error

whale-diagnostics-20210515-0143.zip

Edited by Celsiuss
Link to comment

Cache device problems:

 

May 15 00:00:47 whale kernel: sd 1:0:0:0: [sdc] tag#14 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x06 cmd_age=37s
May 15 00:00:47 whale kernel: sd 1:0:0:0: [sdc] tag#14 CDB: opcode=0x2a 2a 00 0a f8 92 b8 00 00 10 00
May 15 00:00:47 whale kernel: blk_update_request: I/O error, dev sdc, sector 184062648 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
May 15 00:00:47 whale kernel: sd 1:0:0:0: [sdc] tag#13 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x06 cmd_age=37s
May 15 00:00:47 whale kernel: sd 1:0:0:0: [sdc] tag#13 CDB: opcode=0x2a 2a 00 0a f8 8f 70 00 00 08 00
May 15 00:00:47 whale kernel: blk_update_request: I/O error, dev sdc, sector 184061808 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
May 15 00:00:47 whale kernel: sd 1:0:0:0: [sdc] tag#7 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x06 cmd_age=37s
May 15 00:00:47 whale kernel: sd 1:0:0:0: [sdc] tag#7 CDB: opcode=0x28 28 00 00 52 c4 70 00 00 08 00
May 15 00:00:47 whale kernel: blk_update_request: I/O error, dev sdc, sector 5424240 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
May 15 00:00:47 whale kernel: sd 1:0:0:0: [sdc] tag#8 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x06 cmd_age=37s
May 15 00:00:47 whale kernel: sd 1:0:0:0: [sdc] tag#8 CDB: opcode=0x28 28 00 07 8f 48 80 00 00 20 00
May 15 00:00:47 whale kernel: blk_update_request: I/O error, dev sdc, sector 126830720 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
May 15 00:00:47 whale kernel: dm-1: writeback error on inode 209549217, offset 4096, sector 184028976
May 15 00:00:47 whale kernel: dm-1: writeback error on inode 209549217, offset 8192, sector 184029816
May 15 00:00:47 whale kernel: XFS (dm-1): metadata I/O error in "xfs_imap_to_bp+0x5c/0xa2 [xfs]" at daddr 0x78ec840 len 32 error 5
May 15 00:00:47 whale kernel: XFS (dm-1): xfs_do_force_shutdown(0x1) called from line 296 of file fs/xfs/xfs_trans_buf.c. Return address = 00000000b84208d3
May 15 00:00:47 whale kernel: XFS (dm-1): I/O Error Detected. Shutting down filesystem
May 15 00:00:47 whale kernel: XFS (dm-1): Please unmount the filesystem and rectify the problem(s)

 

Start by replacing cables to see if it helps.

  • Thanks 1
Link to comment
6 hours ago, JorgeB said:

Cache device problems:

 


May 15 00:00:47 whale kernel: sd 1:0:0:0: [sdc] tag#14 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x06 cmd_age=37s
May 15 00:00:47 whale kernel: sd 1:0:0:0: [sdc] tag#14 CDB: opcode=0x2a 2a 00 0a f8 92 b8 00 00 10 00
May 15 00:00:47 whale kernel: blk_update_request: I/O error, dev sdc, sector 184062648 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
May 15 00:00:47 whale kernel: sd 1:0:0:0: [sdc] tag#13 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x06 cmd_age=37s
May 15 00:00:47 whale kernel: sd 1:0:0:0: [sdc] tag#13 CDB: opcode=0x2a 2a 00 0a f8 8f 70 00 00 08 00
May 15 00:00:47 whale kernel: blk_update_request: I/O error, dev sdc, sector 184061808 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
May 15 00:00:47 whale kernel: sd 1:0:0:0: [sdc] tag#7 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x06 cmd_age=37s
May 15 00:00:47 whale kernel: sd 1:0:0:0: [sdc] tag#7 CDB: opcode=0x28 28 00 00 52 c4 70 00 00 08 00
May 15 00:00:47 whale kernel: blk_update_request: I/O error, dev sdc, sector 5424240 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
May 15 00:00:47 whale kernel: sd 1:0:0:0: [sdc] tag#8 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x06 cmd_age=37s
May 15 00:00:47 whale kernel: sd 1:0:0:0: [sdc] tag#8 CDB: opcode=0x28 28 00 07 8f 48 80 00 00 20 00
May 15 00:00:47 whale kernel: blk_update_request: I/O error, dev sdc, sector 126830720 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
May 15 00:00:47 whale kernel: dm-1: writeback error on inode 209549217, offset 4096, sector 184028976
May 15 00:00:47 whale kernel: dm-1: writeback error on inode 209549217, offset 8192, sector 184029816
May 15 00:00:47 whale kernel: XFS (dm-1): metadata I/O error in "xfs_imap_to_bp+0x5c/0xa2 [xfs]" at daddr 0x78ec840 len 32 error 5
May 15 00:00:47 whale kernel: XFS (dm-1): xfs_do_force_shutdown(0x1) called from line 296 of file fs/xfs/xfs_trans_buf.c. Return address = 00000000b84208d3
May 15 00:00:47 whale kernel: XFS (dm-1): I/O Error Detected. Shutting down filesystem
May 15 00:00:47 whale kernel: XFS (dm-1): Please unmount the filesystem and rectify the problem(s)

 

Start by replacing cables to see if it helps.

Thank you, I replaced the cable for the cache drive and it seems to be working fine

Link to comment
  • Celsiuss changed the title to (SOLVED) Docker is unresponsive / BTFRS error

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.