My disk 5 which runs BTRFS on my array has gone into read only and comes up with the errors BTRFS error (device md5p1: state EA): parent transid verify failed on logical.

 

 

I didn't notice until later how low the space was getting on my array and disk 5 only has 5GB of space left so I'm not too sure if this is what has caused it.

 

I also noticed a lot of my files on it is missing when I look at the disk5 share. This has also caused some folder on my unraid FUSE share to just have all the files missing and not show up because of this.

 

I did have a recent power outage so it may have contributed to this as well but I regret so much going BTRFS.

 

I've attached the diagnostic logs

 

What could I do to fix this?

 

Quote

Sep 30 05:00:01 Unraid Recycle Bin: Scheduled: Files older than 7 days have been removed

Sep 30 18:48:01 Unraid sSMTP[7630]: Creating SSL connection to host

Sep 30 18:48:01 Unraid sSMTP[7630]: SSL connection using TLS_AES_256_GCM_SHA384

Sep 30 18:48:06 Unraid sSMTP[7630]: Authorization failed (535 5.7.139 Authentication unsuccessful, basic authentication is disabled. [LO3P265CA0023.GBRP265.PROD.OUTLOOK.COM 2024-09-30T17:48:06.933Z 08DCE1182ED59559])

Sep 30 19:06:29 Unraid flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update

Oct  1 01:51:35 Unraid kernel: ------------[ cut here ]------------

Oct  1 01:51:35 Unraid kernel: BTRFS: Transaction aborted (error -28)

Oct  1 01:51:35 Unraid kernel: WARNING: CPU: 11 PID: 18978 at fs/btrfs/free-space-tree.c:859 remove_from_free_space_tree+0x2d8/0x321

Oct  1 01:51:35 Unraid kernel: Modules linked in: udp_diag nvidia_uvm(PO) veth xt_nat xt_tcpudp ipvlan xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype br_netfilter nfsd auth_rpcgss oid_registry lockd grace sunrpc

Oct  1 01:51:35 Unraid kernel: BTRFS: error (device md5p1: state A) in remove_from_free_space_tree:859: errno=-28 No space left

Oct  1 01:51:35 Unraid kernel: md_mod zfs(PO) zunicode(PO) zzstd(O)

Oct  1 01:51:35 Unraid kernel: BTRFS info (device md5p1: state EA): forced readonly

Oct  1 01:51:35 Unraid kernel: zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard

Oct  1 01:51:35 Unraid kernel: BTRFS error (device md5p1: state EA): failed to run delayed ref for logical 3168065204224 num_bytes 12288 type 178 action 1 ref_mod 1: -28

Oct  1 01:51:35 Unraid kernel: curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64

Oct  1 01:51:35 Unraid kernel: BTRFS: error (device md5p1: state EA) in btrfs_run_delayed_refs:2149: errno=-28 No space left

Oct  1 01:51:35 Unraid kernel: poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge stp llc ixgbe xfrm_algo mdio e1000e nvidia_drm(PO) nvidia_modeset(PO) intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel nvidia(PO) kvm video drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 drm aesni_intel crypto_simd cryptd backlight isci i2c_i801 rapl mei_pxp mei_hdcp wmi_bmof syscopyarea i2c_smbus intel_cstate mpt3sas firewire_ohci mei_me libsas sysfillrect ahci intel_uncore input_leds raid_class sysimgblt i2c_core joydev led_class firewire_core libahci mei fb_sys_fops scsi_transport_sas wmi button unix [last unloaded: xfrm_algo]

Oct  1 01:51:35 Unraid kernel: CPU: 11 PID: 18978 Comm: kworker/u32:3 Tainted: P           O       6.1.64-Unraid #1

Oct  1 01:51:35 Unraid kernel: Hardware name: Hewlett-Packard HP Z420 Workstation/1589, BIOS J61 v01.14 07/17/2012

Oct  1 01:51:35 Unraid kernel: Workqueue: events_unbound btrfs_async_reclaim_metadata_space

Oct  1 01:51:35 Unraid kernel: RIP: 0010:remove_from_free_space_tree+0x2d8/0x321

Oct  1 01:51:35 Unraid kernel: Code: 48 05 28 0a 00 00 f0 48 0f ba 28 03 41 b0 00 72 1e 89 df e8 a7 90 4b 00 84 c0 74 10 89 de 48 c7 c7 e7 31 0f 82 e8 76 39 cc ff <0f> 0b 41 b0 01 41 83 e0 01 89 d9 ba 5b 03 00 00 48 c7 c6 a0 40 e4

Oct  1 01:51:35 Unraid kernel: RSP: 0018:ffffc90005ef7bd8 EFLAGS: 00010282

Oct  1 01:51:35 Unraid kernel: RAX: 0000000000000000 RBX: 00000000ffffffe4 RCX: 0000000000000027

Oct  1 01:51:35 Unraid kernel: RDX: 0000000000000002 RSI: ffffffff820d7e01 RDI: 00000000ffffffff

Oct  1 01:51:35 Unraid kernel: RBP: ffff8886c69cd620 R08: 0000000000000000 R09: ffffffff82245f10

Oct  1 01:51:35 Unraid kernel: R10: 00007fffffffffff R11: 00000000322d2072 R12: ffff888215bc5800

Oct  1 01:51:35 Unraid kernel: R13: ffff88821cd10750 R14: 000002e19f5f3000 R15: ffff88822900c2a0

Oct  1 01:51:35 Unraid kernel: FS:  0000000000000000(0000) GS:ffff888fefcc0000(0000) knlGS:0000000000000000

Oct  1 01:51:35 Unraid kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033

Oct  1 01:51:35 Unraid kernel: CR2: 000000c0004a3000 CR3: 000000000c20a006 CR4: 00000000000606e0

Oct  1 01:51:35 Unraid kernel: Call Trace:

Oct  1 01:51:35 Unraid kernel: <TASK>

Oct  1 01:51:35 Unraid kernel: ? __warn+0xab/0x122

Oct  1 01:51:35 Unraid kernel: ? report_bug+0x109/0x17e

Oct  1 01:51:35 Unraid kernel: ? remove_from_free_space_tree+0x2d8/0x321

Oct  1 01:51:35 Unraid kernel: ? handle_bug+0x41/0x6f

Oct  1 01:51:35 Unraid kernel: ? exc_invalid_op+0x13/0x60

Oct  1 01:51:35 Unraid kernel: ? asm_exc_invalid_op+0x16/0x20

Oct  1 01:51:35 Unraid kernel: ? remove_from_free_space_tree+0x2d8/0x321

Oct  1 01:51:35 Unraid kernel: ? remove_from_free_space_tree+0x2d8/0x321

Oct  1 01:51:35 Unraid kernel: ? alloc_reserved_file_extent+0x1a7/0x1dd

Oct  1 01:51:35 Unraid kernel: alloc_reserved_extent+0x1f/0xab

Oct  1 01:51:35 Unraid kernel: __btrfs_run_delayed_refs+0x656/0xbe2

Oct  1 01:51:35 Unraid kernel: btrfs_run_delayed_refs+0x65/0x146

Oct  1 01:51:35 Unraid kernel: flush_space+0x37e/0x491

Oct  1 01:51:35 Unraid kernel: ? _raw_spin_lock+0x13/0x1c

Oct  1 01:51:35 Unraid kernel: ? _raw_spin_unlock+0x14/0x29

Oct  1 01:51:35 Unraid kernel: ? btrfs_get_alloc_profile+0xdb/0x17c

Oct  1 01:51:35 Unraid kernel: ? btrfs_bg_type_to_factor+0xa/0x1c

Oct  1 01:51:35 Unraid kernel: ? calc_available_free_space.isra.0+0x33/0x59

Oct  1 01:51:35 Unraid kernel: btrfs_async_reclaim_metadata_space+0x7d/0x135

Oct  1 01:51:35 Unraid kernel: process_one_work+0x1ab/0x295

Oct  1 01:51:35 Unraid kernel: worker_thread+0x18b/0x244

Oct  1 01:51:35 Unraid kernel: ? rescuer_thread+0x281/0x281

Oct  1 01:51:35 Unraid kernel: kthread+0xe7/0xef

Oct  1 01:51:35 Unraid kernel: ? kthread_complete_and_exit+0x1b/0x1b

Oct  1 01:51:35 Unraid kernel: ret_from_fork+0x22/0x30

Oct  1 01:51:35 Unraid kernel: </TASK>

Oct  1 01:51:35 Unraid kernel: ---[ end trace 0000000000000000 ]---

Oct  1 01:51:35 Unraid kernel: BTRFS info (device md5p1: state EA): dumping space info:

Oct  1 01:51:35 Unraid kernel: BTRFS info (device md5p1: state EA): space_info DATA has 2816659456 free, is full

Oct  1 01:51:35 Unraid kernel: BTRFS info (device md5p1: state EA): space_info total=17918585667584, used=17913175400448, pinned=12288, reserved=2593484800, may_use=45056, readonly=65536 zone_unusable=0

Oct  1 01:51:35 Unraid kernel: BTRFS info (device md5p1: state EA): space_info METADATA has 540672 free, is not full

Oct  1 01:51:35 Unraid kernel: BTRFS info (device md5p1: state EA): space_info total=40802189312, used=36196302848, pinned=869482496, reserved=357285888, may_use=3378511872, readonly=65536 zone_unusable=0

Oct  1 01:51:35 Unraid kernel: BTRFS info (device md5p1: state EA): space_info SYSTEM has 6258688 free, is not full

Oct  1 01:51:35 Unraid kernel: BTRFS info (device md5p1: state EA): space_info total=8388608, used=2129920, pinned=0, reserved=0, may_use=0, readonly=0 zone_unusable=0

Oct  1 01:51:35 Unraid kernel: BTRFS info (device md5p1: state EA): global_block_rsv: size 536870912 reserved 0

Oct  1 01:51:35 Unraid kernel: BTRFS info (device md5p1: state EA): trans_block_rsv: size 0 reserved 0

Oct  1 01:51:35 Unraid kernel: BTRFS info (device md5p1: state EA): chunk_block_rsv: size 0 reserved 0

Oct  1 01:51:35 Unraid kernel: BTRFS info (device md5p1: state EA): delayed_block_rsv: size 131072 reserved 131072

Oct  1 01:51:35 Unraid kernel: BTRFS info (device md5p1: state EA): delayed_refs_rsv: size 71543291904 reserved 0

Oct  1 01:51:35 Unraid kernel: BTRFS: error (device md5p1: state EA) in remove_from_free_space_tree:859: errno=-28 No space left

Oct  1 01:51:35 Unraid kernel: BTRFS error (device md5p1: state EA): failed to run delayed ref for logical 3168064712704 num_bytes 12288 type 178 action 1 ref_mod 1: -28

Oct  1 01:51:35 Unraid kernel: BTRFS: error (device md5p1: state EA) in btrfs_run_delayed_refs:2149: errno=-28 No space left

Oct  1 04:40:02 Unraid root: Fix Common Problems Version 2024.07.18

Oct  1 04:40:02 Unraid root: Fix Common Problems: Warning: Docker Application Authelia has an update available for it

Oct  1 04:40:02 Unraid root: Fix Common Problems: Warning: Docker Application Czkawka has an update available for it

Oct  1 04:40:06 Unraid root: Failed to write /mnt/disk5/985608852.tmp

Oct  1 04:40:06 Unraid root: Fix Common Problems: Error: Unable to write to disk5

Oct  1 04:40:14 Unraid root: Fix Common Problems: Warning: Jumbo Frames detected on eth1 ** Ignored

Oct  1 04:40:15 Unraid sSMTP[8410]: Creating SSL connection to host

Oct  1 04:40:15 Unraid sSMTP[8410]: SSL connection using TLS_AES_256_GCM_SHA384

Oct  1 04:40:21 Unraid sSMTP[8410]: Authorization failed (535 5.7.139 Authentication unsuccessful, basic authentication is disabled. [LO2P265CA0481.GBRP265.PROD.OUTLOOK.COM 2024-10-01T03:40:21.217Z 08DCE1B07914442B])

Oct  1 05:00:01 Unraid Recycle Bin: Scheduled: Files older than 7 days have been removed

Oct  1 05:02:29 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23544708595712 mirror 1 wanted 353256 found 333667

Oct  1 05:02:29 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23544708595712 mirror 2 wanted 353256 found 333667

Oct  1 06:17:55 Unraid monitor: Stop running nchan processes

Oct  1 13:16:58 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 1 wanted 353256 found 349110

Oct  1 13:16:58 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 2 wanted 353256 found 349110

Oct  1 13:16:58 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 1 wanted 353256 found 349110

Oct  1 13:16:58 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 2 wanted 353256 found 349110

Oct  1 13:16:58 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 1 wanted 353256 found 349110

Oct  1 13:16:58 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 2 wanted 353256 found 349110

Oct  1 13:16:58 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 1 wanted 353256 found 349110

Oct  1 13:16:58 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 2 wanted 353256 found 349110

Oct  1 13:16:58 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 1 wanted 353256 found 349110

Oct  1 13:16:58 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 2 wanted 353256 found 349110

Oct  1 13:17:04 Unraid kernel: verify_parent_transid: 156 callbacks suppressed

Oct  1 13:17:04 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 1 wanted 353256 found 349110

Oct  1 13:17:04 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 2 wanted 353256 found 349110

Oct  1 13:17:04 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 1 wanted 353256 found 349110

Oct  1 13:17:04 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 2 wanted 353256 found 349110

Oct  1 13:17:04 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 1 wanted 353256 found 349110

Oct  1 13:17:04 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 2 wanted 353256 found 349110

Oct  1 13:17:04 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 1 wanted 353256 found 349110

Oct  1 13:17:04 Unraid kernel: BTRFS error (device md5p1: state EA): parent transid verify failed on logical 23761478664192 mirror 2 wanted 353256 found 349110

 

 

unraid-diagnostics-20241002-2117.zip

Edited October 2, 20241 yr by Hellomynameisleo

10 hours ago, Hellomynameisleo said:

on my array and disk 5 only has 5GB of space left so I'm not too sure if this is what has caused it.

Looks like that was the problem, since the fs ran out of space, reboot, start the array, don't try to write anything to disk5, and dicne appdata exists on disk5, make sure you disable the docker service, then post new diags and the output from

btrfs fi usage -T /mnt/disk5

 

10 hours ago, JorgeB said:

Looks like that was the problem, since the fs ran out of space, reboot, start the array, don't try to write anything to disk5, and dicne appdata exists on disk5, make sure you disable the docker service, then post new diags and the output from

btrfs fi usage -T /mnt/disk5

 

Yep got it working again. I've restarted the array like you said and it came online fine. Then deleted a bunch of stuff to free space up. Thanks, had a panic attack.

Quote

Overall:
    Device size:                  16.37TiB
    Device allocated:             16.09TiB
    Device unallocated:          287.00GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                         15.95TiB
    Free (estimated):            416.71GiB      (min: 273.21GiB)
    Free (statfs, df):           416.71GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

              Data     Metadata System                             
Id Path       single   DUP      DUP      Unallocated Total    Slack
-- ---------- -------- -------- -------- ----------- -------- -----
 1 /dev/md5p1 16.02TiB 76.00GiB 16.00MiB   287.00GiB 16.37TiB     -
-- ---------- -------- -------- -------- ----------- -------- -----
   Total      16.02TiB 38.00GiB  8.00MiB   287.00GiB 16.37TiB 0.00B
   Used       15.89TiB 32.83GiB  2.00MiB  

 

Looks good now, avoid completely filling up a COW filesystem like btrfs or zfs.