LOF_HWalker

Embarrassed that I didn't think of this, it failed so I'll get some new RAM in there ASAP. They are scheduled, but I was wrong, I did already have them set to non-correcting so I will leave them that way. Thanks for the help!

We are running Unraid 6.12.9. Every time a parity check is run, it has a high error count (anywhere from 20-100+). The parity checks are set to correct errors, so I think these are new errors every time. None of our drives are showing any errors. How can I narrow down the source of these errors? I've also attached the diagnostics file. Thanks all! lof1-nas-diagnostics-20240531-0808.zip

I forgot to update this thread, but a couple of days after I posted this (on Thanksgiving, naturally) the problem progressed and the system wouldn't come back up after a reboot (it was throwing a bzimage checksum error). I ended up setting up a new install on a new flash drive and copying the config over and things have been fine since. Thank you!

We have been having an issue with one of our Unraid NAS's where SMB seems to get into a loop of crashing and restarting, and all of the shares are inaccessible. This same basically identical block appears over and over in the syslog when we are having these issues: Nov 21 10:08:28 LOF1-NAS kernel: SQUASHFS error: xz decompression failed, data probably corrupt Nov 21 10:08:28 LOF1-NAS kernel: SQUASHFS error: Failed to read block 0x92b88c0: -5 Nov 21 10:08:28 LOF1-NAS smbd[32695]: [2023/11/21 10:08:28.598924, 0] ../../lib/util/fault.c:173(smb_panic_log) Nov 21 10:08:28 LOF1-NAS smbd[32695]: =============================================================== Nov 21 10:08:28 LOF1-NAS smbd[32695]: [2023/11/21 10:08:28.598973, 0] ../../lib/util/fault.c:174(smb_panic_log) Nov 21 10:08:28 LOF1-NAS smbd[32695]: INTERNAL ERROR: Signal 7: Bus error in pid 32695 (4.17.10) Nov 21 10:08:28 LOF1-NAS smbd[32695]: [2023/11/21 10:08:28.598987, 0] ../../lib/util/fault.c:178(smb_panic_log) Nov 21 10:08:28 LOF1-NAS smbd[32695]: If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting Nov 21 10:08:28 LOF1-NAS smbd[32695]: [2023/11/21 10:08:28.599012, 0] ../../lib/util/fault.c:183(smb_panic_log) Nov 21 10:08:28 LOF1-NAS smbd[32695]: =============================================================== Nov 21 10:08:28 LOF1-NAS smbd[32695]: [2023/11/21 10:08:28.599024, 0] ../../lib/util/fault.c:184(smb_panic_log) Nov 21 10:08:28 LOF1-NAS smbd[32695]: PANIC (pid 32695): Signal 7: Bus error in 4.17.10 Nov 21 10:08:28 LOF1-NAS kernel: SQUASHFS error: xz decompression failed, data probably corrupt Nov 21 10:08:28 LOF1-NAS kernel: SQUASHFS error: Failed to read block 0x92b88c0: -5 Nov 21 10:08:28 LOF1-NAS smbd[32695]: [2023/11/21 10:08:28.599591, 0] ../../lib/util/fault.c:292(log_stack_trace) Nov 21 10:08:28 LOF1-NAS smbd[32695]: BACKTRACE: 30 stack frames: Nov 21 10:08:28 LOF1-NAS smbd[32695]: #0 /usr/lib64/libgenrand-samba4.so(log_stack_trace+0x2e) [0x153cd843d64e] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #1 /usr/lib64/libgenrand-samba4.so(smb_panic+0x9) [0x153cd843d8a9] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #2 /usr/lib64/libgenrand-samba4.so(+0x2937) [0x153cd843d937] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #3 /lib64/libc.so.6(+0x3ae20) [0x153cd821ee20] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #4 /usr/lib64/libsmbd-base-samba4.so(+0x77f50) [0x153cd8849f50] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #5 /usr/lib64/libsmbd-base-samba4.so(smbd_become_root+0x12) [0x153cd884adb2] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #6 /usr/lib64/libsmbconf.so.0(remote_arch_cache_update+0xd5) [0x153cd8715d05] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #7 /usr/lib64/libsmbd-base-samba4.so(smbd_smb2_request_process_negprot+0xe28) [0x153cd888a7d8] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #8 /usr/lib64/libsmbd-base-samba4.so(smbd_smb2_request_dispatch+0xe48) [0x153cd8886a98] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #9 /usr/lib64/libsmbd-base-samba4.so(smbd_smb2_process_negprot+0x21d) [0x153cd8888f9d] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #10 /usr/lib64/libsmbd-base-samba4.so(process_smb+0x1f3) [0x153cd8873b13] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #11 /usr/lib64/libsmbd-base-samba4.so(smbd_smb1_server_connection_read_handler+0xe0) [0x153cd88eec50] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #12 /usr/lib64/libtevent.so.0(tevent_common_invoke_fd_handler+0x91) [0x153cd83ff8c1] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #13 /usr/lib64/libtevent.so.0(+0xee07) [0x153cd8405e07] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #14 /usr/lib64/libtevent.so.0(+0xcef7) [0x153cd8403ef7] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #15 /usr/lib64/libtevent.so.0(_tevent_loop_once+0x91) [0x153cd83feba1] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #16 /usr/lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x153cd83fee7b] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #17 /usr/lib64/libtevent.so.0(+0xce97) [0x153cd8403e97] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #18 /usr/lib64/libsmbd-base-samba4.so(smbd_process+0x817) [0x153cd8875a37] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #19 /usr/sbin/smbd(+0xb090) [0x55bc3e2f0090] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #20 /usr/lib64/libtevent.so.0(tevent_common_invoke_fd_handler+0x91) [0x153cd83ff8c1] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #21 /usr/lib64/libtevent.so.0(+0xee07) [0x153cd8405e07] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #22 /usr/lib64/libtevent.so.0(+0xcef7) [0x153cd8403ef7] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #23 /usr/lib64/libtevent.so.0(_tevent_loop_once+0x91) [0x153cd83feba1] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #24 /usr/lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x153cd83fee7b] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #25 /usr/lib64/libtevent.so.0(+0xce97) [0x153cd8403e97] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #26 /usr/sbin/smbd(main+0x1489) [0x55bc3e2ed259] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #27 /lib64/libc.so.6(+0x236b7) [0x153cd82076b7] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #28 /lib64/libc.so.6(__libc_start_main+0x85) [0x153cd8207775] Nov 21 10:08:28 LOF1-NAS smbd[32695]: #29 /usr/sbin/s I've attached our diagnostics zip as well. It seems like SMB is hitting some corrupt data and getting tripped up, but it isn't clear to me what data it is trying to access when this happens. lof1-nas-diagnostics-20231121-1202.zip

Yeah when we originally specced this system it was going to run Windows so they used a RAID card. We've been using this for over a year without issue, so hopefully this is just a one-time issue, but I'll make sure to grab diags before rebooting if we run into anything else.

It is a 9361-16i.

Ahhhhhh ok that makes sense. The parity sync/data rebuild is running now, so we'll see if I have any further issues from there. Do you have any resources/information on how I might go about doing that? Thanks for the help!!

After booting my server after installing some new fans, two of my drives (one parity and one data) are disabled. I tried following this guide to check and repair the data drive, but after restarting the array, the data drive is still offline. I've included the results of the most recent check/repair below: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 1 - agno = 0 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done I've also attached my diagnostics export. I have a spare drive that I can install if needed, but the timing and the fact that two of them went out at the same time to me suggests that it isn't a hardware issue. I did try reseating all of the drives as well. Any insight or ideas are much appreciated! lof1-nas-diagnostics-20220428-0059.zip