Running the latest beta (v7 b3) but I don't think this is a beta issue.

 

I checked in on my server and noticed that the webui said docker wasn't running. Checked and all my container processes are still up and plex is playing media but something crashed docker. UnRAID managed to force a reboot even though it couldn't get the disk shares unmounted and things seem ok now.

 

I checked the logs and saw this:

Oct 24 06:00:25 VOID kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Oct 24 06:00:25 VOID kernel: SQUASHFS error: Failed to read block 0x28348c8: -5
Oct 24 06:00:25 VOID kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Oct 24 06:00:25 VOID kernel: SQUASHFS error: Failed to read block 0x28348c8: -5
Oct 24 06:00:25 VOID kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Oct 24 06:00:25 VOID kernel: SQUASHFS error: Failed to read block 0x28348c8: -5
Oct 24 06:00:25 VOID kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Oct 24 06:00:25 VOID kernel: SQUASHFS error: Failed to read block 0x28348c8: -5
Oct 24 06:00:25 VOID kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Oct 24 06:00:25 VOID kernel: SQUASHFS error: Failed to read block 0x28348c8: -5
Oct 24 06:00:25 VOID kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Oct 24 06:00:25 VOID kernel: SQUASHFS error: Failed to read block 0x28348c8: -5

It sounds like this is a flash issue. Should I even try to troubleshoot or just buy a new stick and replace?

 

I see sandisk are no longer recommended (which is what I've got now) so what are safe name brands to pick up? I'm actually going to best buy here in a few hours to pickup a cache nvme so I can just grab something while I'm there.

 

EDIT: This is just about the only thing local that isn't sandisk, is USB type A, and isn't ridiculously huge storage wise (unraid doesnt need a 256GB flash drive) https://www.bestbuy.com/site/pny-elite-turbo-attache-4-32gb-usb-3-2-flash-drive-black/1498843.p?skuId=1498843

void-diagnostics-20241024-0911.zip

Edited October 24, 20241 yr by weirdcrap
forgot the diagnostics

This is usually a flash drive issue.

10 minutes ago, JorgeB said:

This is usually a flash drive issue.

a formatting issue or a hardware issue? If i just need to reformat great. If I need to replace it that's fine to.

 

If I have to replace it I just do the key replacement request and then restore my USB backup to the new drive and run the make bootable script?

Looks like it's recommended to use the new USB tool now https://docs.unraid.net/unraid-os/manual/changing-the-flash-device/

Edited October 24, 20241 yr by weirdcrap

You can try re-formatting it first, if the issue persists, replace it.

Well it locked up real good just a bit ago and had to be hard shutdown. It's been rock solid stable up until that squashfs error showed up yesterday.

 

This was the last thing recorded in the log:

Oct 24 11:10:33 VOID kernel: general protection fault, probably for non-canonical address 0x8000000000010: 0000 [#1] PREEMPT SMP NOPTI
Oct 24 11:10:33 VOID kernel: CPU: 10 PID: 1344233 Comm: fuser Tainted: P           O       6.6.52-Unraid #2
Oct 24 11:10:33 VOID kernel: Hardware name: Micro-Star International Co., Ltd. MS-7D31/MPG Z690 EDGE WIFI DDR4 (MS-7D31), BIOS 1.90 11/08/2022
Oct 24 11:10:33 VOID kernel: RIP: 0010:mem_cgroup_from_slab_obj+0x88/0x9a
Oct 24 11:10:33 VOID kernel: Code: 0c 48 01 c7 48 29 fb 89 d8 48 0f af c1 0f b6 4e 24 48 c1 e8 20 29 c3 d3 eb 0f b6 4e 25 01 c3 d3 eb 48 8b 04 da 48 85 c0 74 8a <48> 8b 40 10 eb 06 5b e9 1a 90 ff ff 5b c3 cc cc cc cc 90 90 90 90
Oct 24 11:10:33 VOID kernel: RSP: 0018:ffffc9000b07bd50 EFLAGS: 00010206
Oct 24 11:10:33 VOID kernel: RAX: 0008000000000000 RBX: 000000000000000e RCX: 0000000000000007
Oct 24 11:10:33 VOID kernel: RDX: ffff88801387e400 RSI: ffff8881001dec00 RDI: ffff8880282f6000
Oct 24 11:10:33 VOID kernel: RBP: 0000000000000000 R08: 00000000008a40a4 R09: 000000000000000a
Oct 24 11:10:33 VOID kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff888109699460
Oct 24 11:10:33 VOID kernel: R13: ffff888107152800 R14: ffff8880282f6b00 R15: 0000000000000000
Oct 24 11:10:33 VOID kernel: FS:  0000146877134740(0000) GS:ffff888890480000(0000) knlGS:0000000000000000
Oct 24 11:10:33 VOID kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 24 11:10:33 VOID kernel: CR2: 000014e3ac739860 CR3: 0000000116a26006 CR4: 0000000000770ee0
Oct 24 11:10:33 VOID kernel: PKRU: 55555554
Oct 24 11:10:33 VOID kernel: Call Trace:
Oct 24 11:10:33 VOID kernel: <TASK>
Oct 24 11:10:33 VOID kernel: ? __die_body+0x1a/0x5c
Oct 24 11:10:33 VOID kernel: ? die_addr+0x38/0x51
Oct 24 11:10:33 VOID kernel: ? exc_general_protection+0x2f7/0x331
Oct 24 11:10:33 VOID kernel: ? asm_exc_general_protection+0x22/0x30
Oct 24 11:10:33 VOID kernel: ? mem_cgroup_from_slab_obj+0x88/0x9a
Oct 24 11:10:33 VOID kernel: list_lru_add+0x84/0xfa
Oct 24 11:10:33 VOID kernel: ? __pfx_proc_fd_instantiate+0x10/0x10
Oct 24 11:10:33 VOID kernel: retain_dentry+0x8a/0xa5
Oct 24 11:10:33 VOID kernel: dput+0x41/0x13a
Oct 24 11:10:33 VOID kernel: proc_fill_cache+0x111/0x157
Oct 24 11:10:33 VOID kernel: ? __pfx_filldir64+0x10/0x10
Oct 24 11:10:33 VOID kernel: proc_readfd_common+0x178/0x1c9
Oct 24 11:10:33 VOID kernel: ? __pfx_proc_fd_instantiate+0x10/0x10
Oct 24 11:10:33 VOID kernel: iterate_dir+0x75/0x12f
Oct 24 11:10:33 VOID kernel: __do_sys_getdents64+0x6b/0xd8
Oct 24 11:10:33 VOID kernel: ? __pfx_filldir64+0x10/0x10
Oct 24 11:10:33 VOID kernel: do_syscall_64+0x57/0x7b
Oct 24 11:10:33 VOID kernel: entry_SYSCALL_64_after_hwframe+0x78/0xe2
Oct 24 11:10:33 VOID kernel: RIP: 0033:0x14687721e053
Oct 24 11:10:33 VOID kernel: Code: 00 16 00 00 00 31 c0 eb c6 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 b8 ff ff ff 7f 48 39 c2 48 0f 47 d0 b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 81 6d 11 00 f7 d8
Oct 24 11:10:33 VOID kernel: RSP: 002b:00007ffd6d62ed08 EFLAGS: 00000293 ORIG_RAX: 00000000000000d9
Oct 24 11:10:33 VOID kernel: RAX: ffffffffffffffda RBX: 0000000000415bd0 RCX: 000014687721e053
Oct 24 11:10:33 VOID kernel: RDX: 0000000000008000 RSI: 0000000000415c00 RDI: 0000000000000008
Oct 24 11:10:33 VOID kernel: RBP: 0000000000415bd4 R08: 0000000000000001 R09: 0000000000000000
Oct 24 11:10:33 VOID kernel: R10: 0000000000000007 R11: 0000000000000293 R12: 0000000000415c00
Oct 24 11:10:33 VOID kernel: R13: ffffffffffffff88 R14: 0000000000000002 R15: 0000000000000000
Oct 24 11:10:33 VOID kernel: </TASK>
Oct 24 11:10:33 VOID kernel: Modules linked in: xt_connmark xt_comment iptable_raw wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha xt_mark af_packet veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo ip6table_nat iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter md_mod zfs(PO) spl(O) iptable_mangle ntfs3 tcp_diag inet_diag nct6683 ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge stp llc intel_rapl_common x86_pkg_temp_thermal intel_powerclamp i915 coretemp kvm_intel kvm iosf_mbi drm_buddy crct10dif_pclmul crc32_pclmul ttm crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 i2c_algo_bit sha1_ssse3 drm_display_helper aesni_intel crypto_simd cryptd drm_kms_helper btusb btrtl rapl btbcm intel_cstate mei_pxp mei_hdcp wmi_bmof mxm_wmi btintel drm bluetooth intel_gtt mpt3sas intel_uncore i2c_i801 mei_me i2c_smbus igc agpgart ahci mei
Oct 24 11:10:33 VOID kernel: input_leds raid_class ecdh_generic libahci led_class joydev i2c_core ecc scsi_transport_sas tpm_crb tpm_tis tpm_tis_core video tpm backlight wmi fan thermal acpi_tad acpi_pad button
Oct 24 11:10:33 VOID kernel: ---[ end trace 0000000000000000 ]---
Oct 24 11:10:33 VOID kernel: RIP: 0010:mem_cgroup_from_slab_obj+0x88/0x9a
Oct 24 11:10:33 VOID kernel: Code: 0c 48 01 c7 48 29 fb 89 d8 48 0f af c1 0f b6 4e 24 48 c1 e8 20 29 c3 d3 eb 0f b6 4e 25 01 c3 d3 eb 48 8b 04 da 48 85 c0 74 8a <48> 8b 40 10 eb 06 5b e9 1a 90 ff ff 5b c3 cc cc cc cc 90 90 90 90
Oct 24 11:10:33 VOID kernel: RSP: 0018:ffffc9000b07bd50 EFLAGS: 00010206
Oct 24 11:10:33 VOID kernel: RAX: 0008000000000000 RBX: 000000000000000e RCX: 0000000000000007
Oct 24 11:10:33 VOID kernel: RDX: ffff88801387e400 RSI: ffff8881001dec00 RDI: ffff8880282f6000
Oct 24 11:10:33 VOID kernel: RBP: 0000000000000000 R08: 00000000008a40a4 R09: 000000000000000a
Oct 24 11:10:33 VOID kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff888109699460
Oct 24 11:10:33 VOID kernel: R13: ffff888107152800 R14: ffff8880282f6b00 R15: 0000000000000000
Oct 24 11:10:33 VOID kernel: FS:  0000146877134740(0000) GS:ffff888890480000(0000) knlGS:0000000000000000
Oct 24 11:10:33 VOID kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 24 11:10:33 VOID kernel: CR2: 000014e3ac739860 CR3: 0000000116a26006 CR4: 0000000000770ee0
Oct 24 11:10:33 VOID kernel: PKRU: 55555554

 

I had mover running taking everything off of my cache drive in preparation for its replacement. I went ahead and replaced the flash out of caution and the fact that I got a 3 pack for ~$15.

Could the squshfs errors also have been caused by RAM? I noticed Kometa crashed this morning with python general protection faults so I ran memtest and apparently my RAM is toast (4000+ errors in under 10 minutes). It's to late at this point I replaced the USB already but I'm curious if I may have jumped the gun on the USB without checking the RAM (because its almost always the RAM).

 

The RAM being bad explains several things over the past month or so that I wrote off as isolated incidents (my plex db corrupting twice, kometa being very crashy, squashfs errors from the flash).

 

Interestingly there were no parity errors on the last few checks. Usually when my RAM is bad the first indicator is randomly shifting parity errors.

Edited October 25, 20241 yr by weirdcrap

34 minutes ago, weirdcrap said:

Could the squshfs errors also have been caused by RAM?

They can.

52 minutes ago, JorgeB said:

They can.

I need to just make a habit of testing the RAM 1st whenever i have issues. Just like in networking it's always DNS in hardware it seems like it's always the RAM.