Failing SSD cache

[Baskedk]

I noticed that a service i was running via docker was unresponsive and logged in to the unRAID instance to see that the CPU was all over the place and was running quite hot due to the CPU stressing.
I then opened the log to see that my SSD was failing big time.
Was getting the following:

Jun 10 15:16:10 Jinx kernel: print_req_error: I/O error, dev loop2, sector 4475240
Jun 10 15:16:10 Jinx kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 180, rd 0, flush 0, corrupt 0, gen 0
Jun 10 15:16:40 Jinx kernel: loop: Write error at byte offset 2291322880, length 4096.
Jun 10 15:16:40 Jinx kernel: print_req_error: I/O error, dev loop2, sector 4475240
Jun 10 15:16:40 Jinx kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 181, rd 0, flush 0, corrupt 0, gen 0
Jun 10 15:17:11 Jinx kernel: loop: Write error at byte offset 2291322880, length 4096.
Jun 10 15:17:11 Jinx kernel: print_req_error: I/O error, dev loop2, sector 4475240
Jun 10 15:17:11 Jinx kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 182, rd 0, flush 0, corrupt 0, gen 0
Jun 10 15:17:42 Jinx kernel: loop: Write error at byte offset 2291322880, length 4096.
Jun 10 15:17:42 Jinx kernel: print_req_error: I/O error, dev loop2, sector 4475240
Jun 10 15:17:42 Jinx kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 183, rd 0, flush 0, corrupt 0, gen 0
Jun 10 15:18:12 Jinx kernel: loop: Write error at byte offset 2291322880, length 4096.
Jun 10 15:18:12 Jinx kernel: print_req_error: I/O error, dev loop2, sector 4475240
Jun 10 15:18:12 Jinx kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 184, rd 0, flush 0, corrupt 0, gen 0
Jun 10 15:18:43 Jinx kernel: loop: Write error at byte offset 2291322880, length 4096.
Jun 10 15:18:43 Jinx kernel: print_req_error: I/O error, dev loop2, sector 4475240
Jun 10 15:18:43 Jinx kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 185, rd 0, flush 0, corrupt 0, gen 0
Jun 10 15:19:14 Jinx kernel: loop: Write error at byte offset 2291322880, length 4096.
Jun 10 15:19:14 Jinx kernel: print_req_error: I/O error, dev loop2, sector 4475240
Jun 10 15:19:14 Jinx kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 186, rd 0, flush 0, corrupt 0, gen 0
Jun 10 15:19:45 Jinx kernel: loop: Write error at byte offset 2291322880, length 4096.
Jun 10 15:19:45 Jinx kernel: print_req_error: I/O error, dev loop2, sector 4475240
Jun 10 15:19:45 Jinx kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 187, rd 0, flush 0, corrupt 0, gen 0
Jun 10 15:20:15 Jinx kernel: loop: Write error at byte offset 2291322880, length 4096.
Jun 10 15:20:15 Jinx kernel: print_req_error: I/O error, dev loop2, sector 4475240
Jun 10 15:20:15 Jinx kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 188, rd 0, flush 0, corrupt 0, gen 0
Jun 10 15:20:46 Jinx kernel: loop: Write error at byte offset 2291322880, length 4096.
Jun 10 15:20:46 Jinx kernel: print_req_error: I/O error, dev loop2, sector 4475240
Jun 10 15:20:46 Jinx kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 189, rd 0, flush 0, corrupt 0, gen 0
Jun 10 15:21:17 Jinx kernel: loop: Write error at byte offset 2291322880, length 4096.
Jun 10 15:21:17 Jinx kernel: print_req_error: I/O error, dev loop2, sector 4475240
Jun 10 15:21:17 Jinx kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 190, rd 0, flush 0, corrupt 0, gen 0
Jun 10 15:21:48 Jinx kernel: loop: Write error at byte offset 2291322880, length 4096.
Jun 10 15:21:48 Jinx kernel: print_req_error: I/O error, dev loop2, sector 4475240
Jun 10 15:21:48 Jinx kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 191, rd 0, flush 0, corrupt 0, gen 0
Jun 10 15:22:18 Jinx kernel: loop: Write error at byte offset 2291322880, length 4096.
Jun 10 15:22:18 Jinx kernel: print_req_error: I/O error, dev loop2, sector 4475240
Jun 10 15:22:18 Jinx kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 192, rd 0, flush 0, corrupt 0, gen 0
Jun 10 15:22:49 Jinx kernel: loop: Write error at byte offset 2291322880, length 4096.
Jun 10 15:22:49 Jinx kernel: print_req_error: I/O error, dev loop2, sector 4475240
Jun 10 15:22:49 Jinx kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 193, rd 0, flush 0, corrupt 0, gen 0

What can i do to combat this. Do I have to replace the disk? Is it only software related, so that a format could get me running again? And if i turn off docker and virtualization and reboot the device, could I maybe get the data off the drive ..... 

 

Thx in advance....

Edited June 10, 2020 by Baskedk

[JorgeB]

loop2 is usually the docker image, please post the diagnostics: Tools -> Diagnostics

[Baskedk]

Alright. Here you are. 

I actually resized the docker image about a week ago. Is that not a thing you should do? 

jinx-diagnostics-20200610-1847.zip

[JorgeB]

Cache filesystem is corrupt and going read-only, hence then the docker image errors, backup cache, re-format and restore the data.

[Baskedk]

That did the trick, thanks 👍