I have a pair of 1TB NVME that are 6 months old as cache drives..
My cache drives are not showing any errors...

For the 2nd time in a month my system has basically chewed itself up.. the cache drive goes readonly docker stops, VM die and then I can't shutdown..

The log shows this

Jun 16 13:56:08 BWSi7 emhttpd: spinning down /dev/sdc

Jun 16 16:00:36 BWSi7 kernel: page: refcount:4 mapcount:0 mapping:00000000882dc6c2 index:0x9724 pfn:0xfd6738

Jun 16 16:00:36 BWSi7 kernel: memcg:ffff8881002cb800

Jun 16 16:00:36 BWSi7 kernel: aops:btree_aops ino:1

Jun 16 16:00:36 BWSi7 kernel: flags: 0x17fffc00000432e(referenced|uptodate|lru|active|workingset|private|writeback|node=0|zone=2|lastcpupid=0x1ffff)

Jun 16 16:00:36 BWSi7 kernel: raw: 017fffc00000432e ffffea003f421008 ffffea003f59ce48 ffff8881506882f0

Jun 16 16:00:36 BWSi7 kernel: raw: 0000000000009724 ffff88860e34db30 00000004ffffffff ffff8881002cb800

Jun 16 16:00:36 BWSi7 kernel: page dumped because: eb page dump

Jun 16 16:00:36 BWSi7 kernel: BTRFS critical (device nvme0n1p1): corrupt leaf: root=5 block=158482432 slot=48 ino=368804, name hash mismatch with key, have 0x00000000548318b9 expect 0x0000000080f7096a

Jun 16 16:00:36 BWSi7 kernel: BTRFS info (device nvme0n1p1): leaf 158482432 gen 42106 total ptrs 126 free space 2812 owner 5

shows a whack of key data and then this

Jun 16 16:00:36 BWSi7 kernel: BTRFS info (device nvme0n1p1 state E): forced readonly

Jun 16 16:00:36 BWSi7 kernel: BTRFS warning (device nvme0n1p1 state E): Skipping commit of aborted transaction.

Jun 16 16:00:36 BWSi7 kernel: BTRFS error (device nvme0n1p1 state EA): Transaction aborted (error -5)

Jun 16 16:00:36 BWSi7 kernel: BTRFS: error (device nvme0n1p1 state EA) in cleanup_transaction:2025: errno=-5 IO failure

Jun 16 16:00:40 BWSi7 kernel: loop: Write error at byte offset 519290880, length 4096.

Jun 16 16:00:40 BWSi7 kernel: loop: Write error at byte offset 1593032704, length 4096.

Jun 16 16:00:40 BWSi7 kernel: I/O error, dev loop2, sector 1014240 op 0x1:(WRITE) flags 0x101000 phys_seg 4 prio class 0

Jun 16 16:00:40 BWSi7 kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0

And then things get messy as it complains about loop2 for a zillion pages

Docker image is not full either

3 minutes ago, BWS001 said:

Docker image is not full either

Is the cache pool full though?

If not, filesystem corruption is usually caused by bad RAM/overall system instability. Run a memtest for a good 24h unless errors are found earlier.

cache was using about 360Gb of the TB.. I did up my one of my VM's from 20 to 24 Gb but free was telling me I had 16gb available when the system was fully loaded.. the VM was just into swap space so I gave it the small bump... Free was still telling me I had 14Gb so the VM didn't consume the entire bump (or other stuff had loosened up)

11 hours ago, BWS001 said:

Jun 16 16:00:36 BWSi7 kernel: BTRFS critical (device nvme0n1p1): corrupt leaf: root=5 block=158482432 slot=48 ino=368804, name hash mismatch with key, have 0x00000000548318b9 expect 0x0000000080f7096a

There's filesystem corruption, with btrfs recommend backing up and reformatting