[6.9.2] cache disk full when at 20% usage

[Struck]

Hi

I have the past two days got an error that indicated my cache drive is out of space. This apparently causes the system to stop functioning correctly. The docker service turns offline, and some of the cpu cores are pinned at 100%.

 

The sys logs says this:

Nov  4 07:59:24 ChiaTower kernel: BTRFS info (device sdo1): leaf 139039850496 gen 76585 total ptrs 0 free space 16283 owner 18446744073709551610
Nov  4 07:59:24 ChiaTower kernel: BTRFS critical (device sdo1): not enough freespace need 48992 have 16283
Nov  4 07:59:24 ChiaTower kernel: ------------[ cut here ]------------
Nov  4 07:59:24 ChiaTower kernel: kernel BUG at fs/btrfs/ctree.c:4814!
Nov  4 07:59:24 ChiaTower kernel: invalid opcode: 0000 [#1] SMP PTI
Nov  4 07:59:24 ChiaTower kernel: CPU: 11 PID: 28875 Comm: shfs Not tainted 5.10.28-Unraid #1
Nov  4 07:59:24 ChiaTower kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X99M Extreme4, BIOS P3.10 06/13/2016
Nov  4 07:59:24 ChiaTower kernel: RIP: 0010:setup_items_for_insert+0xe1/0x294
Nov  4 07:59:24 ChiaTower kernel: Code: 39 f0 73 28 48 89 ef e8 f9 8c 00 00 48 89 ef e8 83 d9 ff ff 48 8b 7c 24 08 44 89 f2 48 c7 c6 de f3 d8 81 89 c1 e8 e3 82 49 00 <0f> 0b 48 8d 7c 24 40 48 89 ee e8 6b 97 ff ff 44 3b 64 24 20 0f 84
Nov  4 07:59:24 ChiaTower kernel: RSP: 0018:ffffc90002b03b10 EFLAGS: 00010296
Nov  4 07:59:24 ChiaTower kernel: RAX: 0000000000000000 RBX: 0000000000003f9b RCX: 0000000000000027
Nov  4 07:59:24 ChiaTower kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88848fcd8920
Nov  4 07:59:24 ChiaTower kernel: RBP: ffff88810cd498c0 R08: 0000000000000000 R09: 00000000ffffefff
Nov  4 07:59:24 ChiaTower kernel: R10: ffffc90002b038c0 R11: ffffc90002b038b8 R12: 0000000000000000
Nov  4 07:59:24 ChiaTower kernel: R13: 000000000000ab10 R14: 000000000000bf60 R15: ffff8881037e0000
Nov  4 07:59:24 ChiaTower kernel: FS:  000014ece30f0700(0000) GS:ffff88848fcc0000(0000) knlGS:0000000000000000
Nov  4 07:59:24 ChiaTower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov  4 07:59:24 ChiaTower kernel: CR2: 00001527d203ca24 CR3: 000000014a2e4005 CR4: 00000000001706e0
Nov  4 07:59:24 ChiaTower kernel: Call Trace:
Nov  4 07:59:24 ChiaTower kernel: btrfs_insert_empty_items+0x65/0x77
Nov  4 07:59:24 ChiaTower kernel: copy_items.isra.0+0xe2/0x358
Nov  4 07:59:24 ChiaTower kernel: ? read_extent_buffer+0x1d/0x91
Nov  4 07:59:24 ChiaTower kernel: ? btrfs_set_path_blocking+0x1f/0x3e
Nov  4 07:59:24 ChiaTower kernel: ? btrfs_search_forward+0x23d/0x274
Nov  4 07:59:24 ChiaTower kernel: btrfs_log_inode+0x597/0xab6
Nov  4 07:59:24 ChiaTower kernel: btrfs_log_inode_parent+0x25b/0x901
Nov  4 07:59:24 ChiaTower kernel: ? slab_post_alloc_hook+0x1e/0x14a
Nov  4 07:59:24 ChiaTower kernel: ? _cond_resched+0x1b/0x1e
Nov  4 07:59:24 ChiaTower kernel: ? wait_current_trans+0xbc/0xda
Nov  4 07:59:24 ChiaTower kernel: ? kmem_cache_alloc+0x108/0x130
Nov  4 07:59:24 ChiaTower kernel: ? join_transaction+0x9d/0x3a3
Nov  4 07:59:24 ChiaTower kernel: btrfs_log_dentry_safe+0x36/0x4a
Nov  4 07:59:24 ChiaTower kernel: btrfs_sync_file+0x250/0x340
Nov  4 07:59:24 ChiaTower kernel: do_fsync+0x2a/0x44
Nov  4 07:59:24 ChiaTower kernel: __x64_sys_fsync+0xb/0xe
Nov  4 07:59:24 ChiaTower kernel: do_syscall_64+0x5d/0x6a
Nov  4 07:59:24 ChiaTower kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Nov  4 07:59:24 ChiaTower kernel: RIP: 0033:0x14ecf13f5a8b
Nov  4 07:59:24 ChiaTower kernel: Code: 0f 05 48 3d 00 f0 ff ff 77 45 c3 0f 1f 40 00 48 83 ec 18 89 7c 24 0c e8 43 f7 ff ff 8b 7c 24 0c 41 89 c0 b8 4a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2f 44 89 c7 89 44 24 0c e8 81 f7 ff ff 8b 44
Nov  4 07:59:24 ChiaTower kernel: RSP: 002b:000014ece30efc70 EFLAGS: 00000297 ORIG_RAX: 000000000000004a
Nov  4 07:59:24 ChiaTower kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 000014ecf13f5a8b
Nov  4 07:59:24 ChiaTower kernel: RDX: 000014ece30efd60 RSI: 0000000000000001 RDI: 000000000000001e
Nov  4 07:59:24 ChiaTower kernel: RBP: 000014ece30efcc0 R08: 0000000000000001 R09: 000014ecc001c724
Nov  4 07:59:24 ChiaTower kernel: R10: 000000000046ab58 R11: 0000000000000297 R12: 000014ecc0010600
Nov  4 07:59:24 ChiaTower kernel: R13: 0000000000000001 R14: 00000000000000ac R15: 000014ece30efd60
Nov  4 07:59:24 ChiaTower kernel: Modules linked in: veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs md_mod nct6775 hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables e1000e alx mdio intel_wmi_thunderbolt mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper rapl intel_cstate intel_uncore i2c_i801 input_leds i2c_smbus i2c_core ahci led_class aacraid libahci wmi button [last unloaded: e1000e]
Nov  4 07:59:24 ChiaTower kernel: ---[ end trace d8547447692a8044 ]---
Nov  4 07:59:24 ChiaTower kernel: RIP: 0010:setup_items_for_insert+0xe1/0x294
Nov  4 07:59:24 ChiaTower kernel: Code: 39 f0 73 28 48 89 ef e8 f9 8c 00 00 48 89 ef e8 83 d9 ff ff 48 8b 7c 24 08 44 89 f2 48 c7 c6 de f3 d8 81 89 c1 e8 e3 82 49 00 <0f> 0b 48 8d 7c 24 40 48 89 ee e8 6b 97 ff ff 44 3b 64 24 20 0f 84
Nov  4 07:59:24 ChiaTower kernel: RSP: 0018:ffffc90002b03b10 EFLAGS: 00010296
Nov  4 07:59:24 ChiaTower kernel: RAX: 0000000000000000 RBX: 0000000000003f9b RCX: 0000000000000027
Nov  4 07:59:24 ChiaTower kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88848fcd8920
Nov  4 07:59:24 ChiaTower kernel: RBP: ffff88810cd498c0 R08: 0000000000000000 R09: 00000000ffffefff
Nov  4 07:59:24 ChiaTower kernel: R10: ffffc90002b038c0 R11: ffffc90002b038b8 R12: 0000000000000000
Nov  4 07:59:24 ChiaTower kernel: R13: 000000000000ab10 R14: 000000000000bf60 R15: ffff8881037e0000
Nov  4 07:59:24 ChiaTower kernel: FS:  000014ece30f0700(0000) GS:ffff88848fcc0000(0000) knlGS:0000000000000000
Nov  4 07:59:24 ChiaTower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov  4 07:59:24 ChiaTower kernel: CR2: 00001527d203ca24 CR3: 000000014a2e4005 CR4: 00000000001706e0
Nov  4 08:59:23 ChiaTower emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog 

Where we in the first lines see this:

Nov  4 07:59:24 ChiaTower kernel: BTRFS critical (device sdo1): not enough freespace need 48992 have 16283

Which means my drive sdo does not have enough space. That drive is mapped to my cache drive.

An Intel 480GB SSD, which ONLY holds my docker and app data for 3 dockers.

Two of these dockers have a high disk usage, but nowhere near 480GB, they will also write to the disk all the time, but not at full speed. The dockers are: krusader, Machinaris, and Machinaris-flax. Which are used as chia farming and flax farming

Unraid says i have used 102GiB of space on the disk.

 

Is the disk bad or am i experiencing another problem?
I have lots of spare disks of the same type and size, so i can swap it, if needed.

 

Currently i have 2 ssds as UD, one of which i have used previously as my cache disk, but it appeared to have some problems, where btrfs whould be corrupted a couple of times. That issue can be found here: 

But i have not experienced that issue with the replaced cache disk.

 

chiatower-diagnostics-20211104-0902.zip

[JorgeB]

You should backup and re-format the filesystem.

[trurl]

In that other thread you linked there was some question about whether you had run memtest long enough. You shouldn't even consider running a computer unless you are confident in its memory. Everything goes through memory, your applications, the OS code, your data, everything.

[Struck]

I realize i never replied to this topic.

I tried reformattig the disk, and it seems to be working.

But i later changed the filesystem to xfs, and ever since i haven't had a problem with corrupt data or full disk.