RoBear

I upgraded to 6.8.1 yesterday morning, and roughly 12 hours later my server became unresponsive, couldn't access via web GUI or SSH, docker containers stopped working. After rebooting the server, dockers started running again, but the syslog is filling with the following: Jan 16 14:57:39 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0 Jan 16 14:57:39 Tower kernel: print_req_error: I/O error, dev loop2, sector 1162312 Jan 16 14:57:39 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0 Jan 16 14:57:44 Tower kernel: loop: Write error at byte offset 576221184, length 4096. Jan 16 14:57:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 1125432 Jan 16 14:57:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0 Jan 16 14:57:44 Tower kernel: loop: Write error at byte offset 1467031552, length 4096. Jan 16 14:57:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 2865296 Jan 16 14:57:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0 Jan 16 14:57:44 Tower kernel: loop: Write error at byte offset 1464651776, length 4096. Jan 16 14:57:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 2860648 Jan 16 14:57:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0 Jan 16 14:57:44 Tower kernel: loop: Write error at byte offset 576221184, length 4096. Jan 16 14:57:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 1125432 Jan 16 14:57:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0 Jan 16 14:57:44 Tower kernel: loop: Write error at byte offset 1467031552, length 4096. Jan 16 14:57:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 2865296 Jan 16 14:57:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0 Jan 16 14:57:44 Tower kernel: loop: Write error at byte offset 576221184, length 4096. Jan 16 14:57:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 1125432 Jan 16 14:57:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0 Jan 16 14:57:44 Tower kernel: loop: Write error at byte offset 1464651776, length 4096. Jan 16 14:57:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 2860648 Jan 16 14:57:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0 Jan 16 14:57:44 Tower kernel: loop: Write error at byte offset 40386560, length 4096. Jan 16 14:57:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 78880 Jan 16 14:57:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0 Jan 16 14:57:44 Tower kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2236: errno=-5 IO failure (Error while writing out transaction) Jan 16 14:57:44 Tower kernel: BTRFS info (device loop2): forced readonly Jan 16 14:57:44 Tower kernel: BTRFS warning (device loop2): Skipping commit of aborted transaction. Jan 16 14:57:44 Tower kernel: BTRFS: error (device loop2) in cleanup_transaction:1846: errno=-5 IO failure Jan 16 14:57:44 Tower kernel: BTRFS info (device loop2): delayed_refs has NO entry Jan 16 14:58:10 Tower kernel: lo_write_bvec: 487 callbacks suppressed Jan 16 14:58:10 Tower kernel: loop: Write error at byte offset 595128320, length 4096. Jan 16 14:58:10 Tower kernel: print_req_error: 487 callbacks suppressed Jan 16 14:58:10 Tower kernel: print_req_error: I/O error, dev loop2, sector 1162360 Jan 16 14:58:10 Tower kernel: btrfs_dev_stat_print_on_error: 487 callbacks suppressed Jan 16 14:58:10 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 498, rd 0, flush 0, corrupt 0, gen 0 Jan 16 14:58:41 Tower kernel: loop: Write error at byte offset 595144704, length 4096. Jan 16 14:58:41 Tower kernel: loop: Write error at byte offset 595795968, length 4096. Jan 16 14:58:41 Tower kernel: print_req_error: I/O error, dev loop2, sector 1162392 I tried running the check repair utility on the cache and came out with this: Starting repair. Opening filesystem to check... Checking filesystem on /dev/nvme0n1p1 UUID: fe89e9aa-3823-42e5-890c-d4cf99342837 [1/7] checking root items Fixed 0 roots. [2/7] checking extents incorrect offsets 12835 12663 incorrect offsets 12835 12663 incorrect offsets 12835 12663 Shifting item nr 260 by 172 bytes in block 5292032 Shifting item nr 261 by 172 bytes in block 5292032 Shifting item nr 262 by 172 bytes in block 5292032 Shifting item nr 263 by 172 bytes in block 5292032 Shifting item nr 264 by 172 bytes in block 5292032 Shifting item nr 265 by 172 bytes in block 5292032 Shifting item nr 266 by 172 bytes in block 5292032 Shifting item nr 267 by 172 bytes in block 5292032 Shifting item nr 268 by 172 bytes in block 5292032 Shifting item nr 269 by 172 bytes in block 5292032 Shifting item nr 270 by 172 bytes in block 5292032 Shifting item nr 271 by 172 bytes in block 5292032 items overlap, can't fix check/main.c:4333: fix_item_offset: BUG_ON `ret` triggered, value -5 btrfs[0x42f1fd] btrfs[0x438339] btrfs[0x4387f0] btrfs[0x43937c] btrfs[0x43d255] btrfs(main+0x90)[0x40ecc0] /lib64/libc.so.6(__libc_start_main+0xeb)[0x15403cae0e5b] btrfs(_start+0x2a)[0x40ef4a] Aborted I downgraded back to 6.7.2 and again my dockers have started up, but the syslog is still filling with BTRFS errors. Machine was rebuilt 2 weeks ago with new hardware, had no major issues after upgrading my hardware until I did the upgrade to 6.8.2 I could use some guidance on what I need to do here. Diagnostic file attached. tower-diagnostics-20200116-2123.zip