Struck Posted November 4, 2021 Share Posted November 4, 2021 Hi I have the past two days got an error that indicated my cache drive is out of space. This apparently causes the system to stop functioning correctly. The docker service turns offline, and some of the cpu cores are pinned at 100%. The sys logs says this: Nov 4 07:59:24 ChiaTower kernel: BTRFS info (device sdo1): leaf 139039850496 gen 76585 total ptrs 0 free space 16283 owner 18446744073709551610 Nov 4 07:59:24 ChiaTower kernel: BTRFS critical (device sdo1): not enough freespace need 48992 have 16283 Nov 4 07:59:24 ChiaTower kernel: ------------[ cut here ]------------ Nov 4 07:59:24 ChiaTower kernel: kernel BUG at fs/btrfs/ctree.c:4814! Nov 4 07:59:24 ChiaTower kernel: invalid opcode: 0000 [#1] SMP PTI Nov 4 07:59:24 ChiaTower kernel: CPU: 11 PID: 28875 Comm: shfs Not tainted 5.10.28-Unraid #1 Nov 4 07:59:24 ChiaTower kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X99M Extreme4, BIOS P3.10 06/13/2016 Nov 4 07:59:24 ChiaTower kernel: RIP: 0010:setup_items_for_insert+0xe1/0x294 Nov 4 07:59:24 ChiaTower kernel: Code: 39 f0 73 28 48 89 ef e8 f9 8c 00 00 48 89 ef e8 83 d9 ff ff 48 8b 7c 24 08 44 89 f2 48 c7 c6 de f3 d8 81 89 c1 e8 e3 82 49 00 <0f> 0b 48 8d 7c 24 40 48 89 ee e8 6b 97 ff ff 44 3b 64 24 20 0f 84 Nov 4 07:59:24 ChiaTower kernel: RSP: 0018:ffffc90002b03b10 EFLAGS: 00010296 Nov 4 07:59:24 ChiaTower kernel: RAX: 0000000000000000 RBX: 0000000000003f9b RCX: 0000000000000027 Nov 4 07:59:24 ChiaTower kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88848fcd8920 Nov 4 07:59:24 ChiaTower kernel: RBP: ffff88810cd498c0 R08: 0000000000000000 R09: 00000000ffffefff Nov 4 07:59:24 ChiaTower kernel: R10: ffffc90002b038c0 R11: ffffc90002b038b8 R12: 0000000000000000 Nov 4 07:59:24 ChiaTower kernel: R13: 000000000000ab10 R14: 000000000000bf60 R15: ffff8881037e0000 Nov 4 07:59:24 ChiaTower kernel: FS: 000014ece30f0700(0000) GS:ffff88848fcc0000(0000) knlGS:0000000000000000 Nov 4 07:59:24 ChiaTower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 4 07:59:24 ChiaTower kernel: CR2: 00001527d203ca24 CR3: 000000014a2e4005 CR4: 00000000001706e0 Nov 4 07:59:24 ChiaTower kernel: Call Trace: Nov 4 07:59:24 ChiaTower kernel: btrfs_insert_empty_items+0x65/0x77 Nov 4 07:59:24 ChiaTower kernel: copy_items.isra.0+0xe2/0x358 Nov 4 07:59:24 ChiaTower kernel: ? read_extent_buffer+0x1d/0x91 Nov 4 07:59:24 ChiaTower kernel: ? btrfs_set_path_blocking+0x1f/0x3e Nov 4 07:59:24 ChiaTower kernel: ? btrfs_search_forward+0x23d/0x274 Nov 4 07:59:24 ChiaTower kernel: btrfs_log_inode+0x597/0xab6 Nov 4 07:59:24 ChiaTower kernel: btrfs_log_inode_parent+0x25b/0x901 Nov 4 07:59:24 ChiaTower kernel: ? slab_post_alloc_hook+0x1e/0x14a Nov 4 07:59:24 ChiaTower kernel: ? _cond_resched+0x1b/0x1e Nov 4 07:59:24 ChiaTower kernel: ? wait_current_trans+0xbc/0xda Nov 4 07:59:24 ChiaTower kernel: ? kmem_cache_alloc+0x108/0x130 Nov 4 07:59:24 ChiaTower kernel: ? join_transaction+0x9d/0x3a3 Nov 4 07:59:24 ChiaTower kernel: btrfs_log_dentry_safe+0x36/0x4a Nov 4 07:59:24 ChiaTower kernel: btrfs_sync_file+0x250/0x340 Nov 4 07:59:24 ChiaTower kernel: do_fsync+0x2a/0x44 Nov 4 07:59:24 ChiaTower kernel: __x64_sys_fsync+0xb/0xe Nov 4 07:59:24 ChiaTower kernel: do_syscall_64+0x5d/0x6a Nov 4 07:59:24 ChiaTower kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Nov 4 07:59:24 ChiaTower kernel: RIP: 0033:0x14ecf13f5a8b Nov 4 07:59:24 ChiaTower kernel: Code: 0f 05 48 3d 00 f0 ff ff 77 45 c3 0f 1f 40 00 48 83 ec 18 89 7c 24 0c e8 43 f7 ff ff 8b 7c 24 0c 41 89 c0 b8 4a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2f 44 89 c7 89 44 24 0c e8 81 f7 ff ff 8b 44 Nov 4 07:59:24 ChiaTower kernel: RSP: 002b:000014ece30efc70 EFLAGS: 00000297 ORIG_RAX: 000000000000004a Nov 4 07:59:24 ChiaTower kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 000014ecf13f5a8b Nov 4 07:59:24 ChiaTower kernel: RDX: 000014ece30efd60 RSI: 0000000000000001 RDI: 000000000000001e Nov 4 07:59:24 ChiaTower kernel: RBP: 000014ece30efcc0 R08: 0000000000000001 R09: 000014ecc001c724 Nov 4 07:59:24 ChiaTower kernel: R10: 000000000046ab58 R11: 0000000000000297 R12: 000014ecc0010600 Nov 4 07:59:24 ChiaTower kernel: R13: 0000000000000001 R14: 00000000000000ac R15: 000014ece30efd60 Nov 4 07:59:24 ChiaTower kernel: Modules linked in: veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs md_mod nct6775 hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables e1000e alx mdio intel_wmi_thunderbolt mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper rapl intel_cstate intel_uncore i2c_i801 input_leds i2c_smbus i2c_core ahci led_class aacraid libahci wmi button [last unloaded: e1000e] Nov 4 07:59:24 ChiaTower kernel: ---[ end trace d8547447692a8044 ]--- Nov 4 07:59:24 ChiaTower kernel: RIP: 0010:setup_items_for_insert+0xe1/0x294 Nov 4 07:59:24 ChiaTower kernel: Code: 39 f0 73 28 48 89 ef e8 f9 8c 00 00 48 89 ef e8 83 d9 ff ff 48 8b 7c 24 08 44 89 f2 48 c7 c6 de f3 d8 81 89 c1 e8 e3 82 49 00 <0f> 0b 48 8d 7c 24 40 48 89 ee e8 6b 97 ff ff 44 3b 64 24 20 0f 84 Nov 4 07:59:24 ChiaTower kernel: RSP: 0018:ffffc90002b03b10 EFLAGS: 00010296 Nov 4 07:59:24 ChiaTower kernel: RAX: 0000000000000000 RBX: 0000000000003f9b RCX: 0000000000000027 Nov 4 07:59:24 ChiaTower kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88848fcd8920 Nov 4 07:59:24 ChiaTower kernel: RBP: ffff88810cd498c0 R08: 0000000000000000 R09: 00000000ffffefff Nov 4 07:59:24 ChiaTower kernel: R10: ffffc90002b038c0 R11: ffffc90002b038b8 R12: 0000000000000000 Nov 4 07:59:24 ChiaTower kernel: R13: 000000000000ab10 R14: 000000000000bf60 R15: ffff8881037e0000 Nov 4 07:59:24 ChiaTower kernel: FS: 000014ece30f0700(0000) GS:ffff88848fcc0000(0000) knlGS:0000000000000000 Nov 4 07:59:24 ChiaTower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 4 07:59:24 ChiaTower kernel: CR2: 00001527d203ca24 CR3: 000000014a2e4005 CR4: 00000000001706e0 Nov 4 08:59:23 ChiaTower emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog Where we in the first lines see this: Nov 4 07:59:24 ChiaTower kernel: BTRFS critical (device sdo1): not enough freespace need 48992 have 16283 Which means my drive sdo does not have enough space. That drive is mapped to my cache drive. An Intel 480GB SSD, which ONLY holds my docker and app data for 3 dockers. Two of these dockers have a high disk usage, but nowhere near 480GB, they will also write to the disk all the time, but not at full speed. The dockers are: krusader, Machinaris, and Machinaris-flax. Which are used as chia farming and flax farming Unraid says i have used 102GiB of space on the disk. Is the disk bad or am i experiencing another problem? I have lots of spare disks of the same type and size, so i can swap it, if needed. Currently i have 2 ssds as UD, one of which i have used previously as my cache disk, but it appeared to have some problems, where btrfs whould be corrupted a couple of times. That issue can be found here: But i have not experienced that issue with the replaced cache disk. chiatower-diagnostics-20211104-0902.zip Quote Link to comment
JorgeB Posted November 4, 2021 Share Posted November 4, 2021 You should backup and re-format the filesystem. Quote Link to comment
trurl Posted November 4, 2021 Share Posted November 4, 2021 In that other thread you linked there was some question about whether you had run memtest long enough. You shouldn't even consider running a computer unless you are confident in its memory. Everything goes through memory, your applications, the OS code, your data, everything. Quote Link to comment
Solution Struck Posted January 18, 2022 Author Solution Share Posted January 18, 2022 I realize i never replied to this topic. I tried reformattig the disk, and it seems to be working. But i later changed the filesystem to xfs, and ever since i haven't had a problem with corrupt data or full disk. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.