[Solved] Docker corrupt?

[bally12345]

All of a sudden got plex down notification. So logged on to my dashboard and noticed all the dockers are stopping. 

 

Checked error log and saw this

 

Feb 16 22:00:53 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:00:53 Server kernel: btrfs_dev_stat_print_on_error: 1 callbacks suppressed Feb 16 22:00:53 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2464, flush 0, corrupt 0, gen 0 Feb 16 22:00:53 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:00:53 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2465, flush 0, corrupt 0, gen 0 Feb 16 22:00:54 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:00:54 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2466, flush 0, corrupt 0, gen 0 Feb 16 22:00:54 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:00:54 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2467, flush 0, corrupt 0, gen 0 Feb 16 22:00:55 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:00:55 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2468, flush 0, corrupt 0, gen 0 Feb 16 22:00:55 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:00:55 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2469, flush 0, corrupt 0, gen 0 Feb 16 22:00:56 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:00:56 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2470, flush 0, corrupt 0, gen 0 Feb 16 22:00:56 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:00:56 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2471, flush 0, corrupt 0, gen 0 Feb 16 22:00:57 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:00:57 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2472, flush 0, corrupt 0, gen 0 Feb 16 22:00:57 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:00:57 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2473, flush 0, corrupt 0, gen 0 Feb 16 22:00:58 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:00:58 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2474, flush 0, corrupt 0, gen 0 Feb 16 22:00:58 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:00:58 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2475, flush 0, corrupt 0, gen 0 Feb 16 22:00:58 Server kernel: blk_update_request: I/O error, dev loop2, sector 2667536 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:00:58 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2476, flush 0, corrupt 0, gen 0 Feb 16 22:00:59 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:00:59 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2477, flush 0, corrupt 0, gen 0 Feb 16 22:00:59 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:00:59 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2478, flush 0, corrupt 0, gen 0 Feb 16 22:01:00 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:01:00 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2479, flush 0, corrupt 0, gen 0 Feb 16 22:01:00 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:01:00 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2480, flush 0, corrupt 0, gen 0 Feb 16 22:01:01 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:01:01 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2481, flush 0, corrupt 0, gen 0 Feb 16 22:01:01 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:01:01 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2482, flush 0, corrupt 0, gen 0 Feb 16 22:01:02 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:01:02 Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 2483, flush 0, corrupt 0, gen 0 Feb 16 22:01:03 Server kernel: print_req_error: 1 callbacks suppressed Feb 16 22:01:03 Server kernel: blk_update_request: I/O error, dev loop2, sector 8085008 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Feb 16 22:01:03 Server kernel: btrfs_dev_stat_print_on_error: 1 callbacks suppressed

 

I have attached diagnostics. Is the docker.img the issue all my drives are XFS so assuming it's has to be the docker?

server-diagnostics-20210216-2203.zip

Edited February 17, 2021 by bally12345
Solved

[bally12345]

Just noticed on the docker tab
One or more paths do not exist (view)

So checked the cache SSD and that's online and accessible

So I'm guessing the next step would be to create a new docker.img with a new name?

If course would like to get to root cause too.

Sent from my SM-G973F using Tapatalk

[bally12345]

I think the SSD is dead! But it's showing online but nothing seems to be available in there and appdata share is missing.

Before I go into full panic mode. I am pretty certain I have appdata backed up atleast weekly to my array.

But still not sure what is going on as I'm on my mobile and have to vpn on to server.

Sent from my SM-G973F using Tapatalk

[bally12345]

Appdata share seems to be missing. If SSD is corrupt surely it should still be visible under shares tab?

Sent from my SM-G973F using Tapatalk

[Squid]

Looks like it just dropped offline.  Reseat the cabling.

[bally12345]

Looks like it just dropped offline.  Reseat the cabling.

Shut down, unplugged and replugged SSD.

Unraid showed it as unmounted, after remount I started docker.

All sorter

Cable wise everything looks fine, might just order another SATA cable just in case.

Sent from my SM-G973F using Tapatalk

[bally12345]

Just got the following alert

Event: Unraid Cache disk SMART health [199]
Subject: Warning [sERVER] - udma crc error count is 1
Description: CT1000MX500SSD1_1910E1EE9FE0 (sdb)
Importance: warning

Pretty sure this related to bad sata connection. I have a replacement coming from amazon tomorrow.

It's just my luck when I need a spare cable I can't find it

Sent from my SM-G973F using Tapatalk