upgraded to 6.8.1, now cache pool is read-only

[RoBear]

I upgraded to 6.8.1 yesterday morning, and roughly 12 hours later my server became unresponsive, couldn't access via web GUI or SSH, docker containers stopped working.  After rebooting the server, dockers started running again, but the syslog is filling with the following:

Jan 16 14:57:39 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Jan 16 14:57:39 Tower kernel: print_req_error: I/O error, dev loop2, sector 1162312
Jan 16 14:57:39 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
Jan 16 14:57:44 Tower kernel: loop: Write error at byte offset 576221184, length 4096.
Jan 16 14:57:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 1125432
Jan 16 14:57:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
Jan 16 14:57:44 Tower kernel: loop: Write error at byte offset 1467031552, length 4096.
Jan 16 14:57:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 2865296
Jan 16 14:57:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
Jan 16 14:57:44 Tower kernel: loop: Write error at byte offset 1464651776, length 4096.
Jan 16 14:57:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 2860648
Jan 16 14:57:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
Jan 16 14:57:44 Tower kernel: loop: Write error at byte offset 576221184, length 4096.
Jan 16 14:57:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 1125432
Jan 16 14:57:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
Jan 16 14:57:44 Tower kernel: loop: Write error at byte offset 1467031552, length 4096.
Jan 16 14:57:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 2865296
Jan 16 14:57:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
Jan 16 14:57:44 Tower kernel: loop: Write error at byte offset 576221184, length 4096.
Jan 16 14:57:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 1125432
Jan 16 14:57:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
Jan 16 14:57:44 Tower kernel: loop: Write error at byte offset 1464651776, length 4096.
Jan 16 14:57:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 2860648
Jan 16 14:57:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
Jan 16 14:57:44 Tower kernel: loop: Write error at byte offset 40386560, length 4096.
Jan 16 14:57:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 78880
Jan 16 14:57:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
Jan 16 14:57:44 Tower kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2236: errno=-5 IO failure (Error while writing out transaction)
Jan 16 14:57:44 Tower kernel: BTRFS info (device loop2): forced readonly
Jan 16 14:57:44 Tower kernel: BTRFS warning (device loop2): Skipping commit of aborted transaction.
Jan 16 14:57:44 Tower kernel: BTRFS: error (device loop2) in cleanup_transaction:1846: errno=-5 IO failure
Jan 16 14:57:44 Tower kernel: BTRFS info (device loop2): delayed_refs has NO entry
Jan 16 14:58:10 Tower kernel: lo_write_bvec: 487 callbacks suppressed
Jan 16 14:58:10 Tower kernel: loop: Write error at byte offset 595128320, length 4096.
Jan 16 14:58:10 Tower kernel: print_req_error: 487 callbacks suppressed
Jan 16 14:58:10 Tower kernel: print_req_error: I/O error, dev loop2, sector 1162360
Jan 16 14:58:10 Tower kernel: btrfs_dev_stat_print_on_error: 487 callbacks suppressed
Jan 16 14:58:10 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 498, rd 0, flush 0, corrupt 0, gen 0
Jan 16 14:58:41 Tower kernel: loop: Write error at byte offset 595144704, length 4096.
Jan 16 14:58:41 Tower kernel: loop: Write error at byte offset 595795968, length 4096.
Jan 16 14:58:41 Tower kernel: print_req_error: I/O error, dev loop2, sector 1162392

 

I tried running the check repair utility on the cache and came out with this:

Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/nvme0n1p1
UUID: fe89e9aa-3823-42e5-890c-d4cf99342837
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
incorrect offsets 12835 12663
incorrect offsets 12835 12663
incorrect offsets 12835 12663
Shifting item nr 260 by 172 bytes in block 5292032
Shifting item nr 261 by 172 bytes in block 5292032
Shifting item nr 262 by 172 bytes in block 5292032
Shifting item nr 263 by 172 bytes in block 5292032
Shifting item nr 264 by 172 bytes in block 5292032
Shifting item nr 265 by 172 bytes in block 5292032
Shifting item nr 266 by 172 bytes in block 5292032
Shifting item nr 267 by 172 bytes in block 5292032
Shifting item nr 268 by 172 bytes in block 5292032
Shifting item nr 269 by 172 bytes in block 5292032
Shifting item nr 270 by 172 bytes in block 5292032
Shifting item nr 271 by 172 bytes in block 5292032
items overlap, can't fix
check/main.c:4333: fix_item_offset: BUG_ON `ret` triggered, value -5
btrfs[0x42f1fd]
btrfs[0x438339]
btrfs[0x4387f0]
btrfs[0x43937c]
btrfs[0x43d255]
btrfs(main+0x90)[0x40ecc0]
/lib64/libc.so.6(__libc_start_main+0xeb)[0x15403cae0e5b]
btrfs(_start+0x2a)[0x40ef4a]
Aborted

 

I downgraded back to 6.7.2 and again my dockers have started up, but the syslog is still filling with BTRFS errors.  Machine was rebuilt 2 weeks ago with new hardware, had no major issues after upgrading my hardware until I did the upgrade to 6.8.2

 

I could use some guidance on what I need to do here.  Diagnostic file attached.

 

tower-diagnostics-20200116-2123.zip

[JorgeB]

There's filesystem corruption on cache, with btrfs best bet is to backup data, re-format and star over.

 

Also note that because of this bug your pool isn't redundant.