BTRFS error (device loop2) causing docker issues

[Gus Brock]

I posted about this before but I was able to narrow this down to the these errors in the log. I have stopped, deleted and recreated the docker.img multiple times this past week. It works for maybe a day before all containers start erroring out.

 

I have also stopped the array and run all the disk (extended) SMART tests but there were no errors.

 

What are my options?

Oct 11 13:00:57 Unraid kernel: loop: Write error at byte offset 6330511360, length 4096.
Oct 11 13:00:57 Unraid kernel: print_req_error: I/O error, dev loop2, sector 12364280
Oct 11 13:00:57 Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Oct 11 13:00:59 Unraid kernel: usb 3-5: USB disconnect, device number 11
Oct 11 13:01:02 Unraid kernel: usb 3-5: new high-speed USB device number 12 using xhci_hcd
Oct 11 13:01:04 Unraid kernel: usb 3-5: USB disconnect, device number 12
Oct 11 13:01:35 Unraid kernel: loop: Write error at byte offset 6337925120, length 4096.
Oct 11 13:01:35 Unraid kernel: print_req_error: I/O error, dev loop2, sector 12378760
Oct 11 13:01:35 Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
Oct 11 13:01:36 Unraid kernel: loop: Write error at byte offset 6372851712, length 4096.
Oct 11 13:01:36 Unraid kernel: print_req_error: I/O error, dev loop2, sector 12446640
Oct 11 13:01:36 Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
Oct 11 13:01:36 Unraid kernel: loop: Write error at byte offset 6372974592, length 4096.
Oct 11 13:01:36 Unraid kernel: print_req_error: I/O error, dev loop2, sector 12447152
Oct 11 13:01:36 Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
Oct 11 13:01:36 Unraid kernel: loop: Write error at byte offset 6337925120, length 4096.
Oct 11 13:01:36 Unraid kernel: print_req_error: I/O error, dev loop2, sector 12378760
Oct 11 13:01:36 Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
Oct 11 13:01:36 Unraid kernel: loop: Write error at byte offset 6334545920, length 4096.
Oct 11 13:01:36 Unraid kernel: print_req_error: I/O error, dev loop2, sector 12372160
Oct 11 13:01:36 Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
Oct 11 13:01:36 Unraid kernel: loop: Write error at byte offset 737681408, length 4096.
Oct 11 13:01:36 Unraid kernel: print_req_error: I/O error, dev loop2, sector 1440768
Oct 11 13:01:36 Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
Oct 11 13:01:37 Unraid kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2267: errno=-5 IO failure (Error while writing out transaction)
Oct 11 13:01:37 Unraid kernel: BTRFS info (device loop2): forced readonly
Oct 11 13:01:37 Unraid kernel: BTRFS warning (device loop2): Skipping commit of aborted transaction.
Oct 11 13:01:37 Unraid kernel: BTRFS: error (device loop2) in cleanup_transaction:1860: errno=-5 IO failure

 

tower-diagnostics-20201011-1349.zip

Edited October 11, 2020 by Gus Brock

[JorgeB]

Cache pool is completely full, errors are caused by the docker image running out of usable space, free up some space.

[Gus Brock]

I was looking at it but the UI showed about 400GB free but looks like this was false. I followed your instructions here and was able to get things going again. Thank you for the help!

[JorgeB]

1 minute ago, Gus Brock said:

I was looking at it but the UI showed about 400GB free but looks like this was false

No, it's true, my mistake, the problem is that the filesystem is fully allocated, you need to run balance, see here for more details:

https://forums.unraid.net/topic/62230-out-of-space-errors-on-cache-drive/?do=findComment&comment=610551