bucketphobia

Am I correct in understandign that ata3 maps to drive sdn? unique_id3 -> host5 host 5 -> sys/block/sdn

Hi! I've been having an issue over the past day with a SATA error - drive first showed up as disabled and when checking syslog I see a spam error of "SATA link down" Diagnostics attached, would appreciate any help! omar-diagnostics-20250505-1123.zip

Yeah sorry, that’s what I meant - Stop array, readd disk, let it do its thing?

Booted up array normally (but without adding the drive back in, leaving it emulated) - drive is fine with all content intact it seems! (no lost+found folder either) I assume next step is to just verify that everything is intact, then re-add the drive and rebuild parity? Thanks a ton for all the help btw, super appreciated!!

Phase 1 - find and verify superblock... sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128 resetting superblock root inode pointer to 128 sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129 resetting superblock realtime bitmap inode pointer to 129 sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130 resetting superblock realtime summary inode pointer to 130 Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata sb_icount 0, counted 25472 sb_ifree 0, counted 14336 sb_fdblocks 3417569701, counted 2424849617 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 7 - agno = 10 - agno = 1 - agno = 12 - agno = 8 - agno = 9 - agno = 6 - agno = 3 - agno = 4 - agno = 11 - agno = 5 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (1:399930) is ahead of log (1:2). Format log to cycle 4. done Ran check afterwards, says "no filesystem corruption detected"

Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... .found candidate secondary superblock... verified secondary superblock... writing modified primary superblock sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128 resetting superblock root inode pointer to 128 sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129 resetting superblock realtime bitmap inode pointer to 129 sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130 resetting superblock realtime summary inode pointer to 130 Phase 2 - using internal log - zero log... ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.

Started in maintenance mode, went to the disk and checked filesystem status, this is what pops up - should I continue with Fix? Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... .found candidate secondary superblock... verified secondary superblock... would write modified primary superblock Primary superblock would have been modified. Cannot proceed further in no_modify mode. Exiting now.

Attached, thank you! omar-diagnostics-20250313-0121.zip

having an issue with a drive showing up as unmountable I removed it from array and started it - hoping it would get emulated, but unfortunately shows up as "Not installed" and data is still not available. I tried to use Unassigned devices to mount it, but it gets an error saying Mar 13 00:15:41 Omar root: Cannot use device /dev/sdc1 which is in use (already mapped or mounted). Mar 13 00:15:41 Omar root: Warning: keyslot operation could fail as it requires more than available memory. Mar 13 00:15:41 Omar emhttpd: shcmd (2030): exit status: 5 Mar 13 00:15:41 Omar emhttpd: read SMART /dev/sdc Mar 13 00:15:41 Omar unassigned.devices: luksOpen result: Error: Passphrase or Key File not found. I tried to add a keyfile to unassigned devices, but can only seem to insert a passphrase, not a keyfile Would appreciate any help, thank you!

Yeah, exactly. I had /mnt/*pool*/ for certain dockers like PLEX, but not all of them.

For anyone who may stumble on the post in the future, I have confirmed that the fix for this specific error is to enable "Exclusive Shares" option in Unraid 6.12 and to verify that it is "Yes" on your appdata share. I have brought back the dockers I had disabled/changed, and everything runs as expected. If that option is turned off, the same iowait issue comes back nearly instantaneously.

If it says no - you likely need to (1): disable secondary storage; (2): make sure that nothing on the share is on other drives

Hey @DrSpaldo, I was having a very similar issue, but slightly different - the cause of mine was a dockerd log process. The solution to mine was to ensure that "Permit exclusive shares" is on, and that "Exclusive Access" is indeed active on my cache pool. Hope this helps.

So as an update, I did three things as of the last post - and it appears one of them has worked, as I have ~84 hours of uptime so far with no signs of the issue happening again so far. Fingers crossed one of these is the long-term solution: 1) Updated BIOS to the latest version (in my case, I have a i7-4790K and was running on quite an old BIOS compared to the latest available 2) Enabled the "Exclusive Shares" option introduced in Unraid 6.12.4 (I had it manually done for certain heavy dockers previously like Plex, but it seems like it has helped.) 3) Disabled the KASM container (I am honestly not so sure how much this could potentially affect the outcome, but this time around I have had it stopped. I will attempt to use KASM again in the future and see if the issue is replicated) Thanks for all the help Jorge, appreciated.

I've been doing some additional research, and found a few threads that I believe may be the same issue I've been experiencing. I updated BIOS as suggested there, and enabled exclusive shares. Will see how it affects the problem.

Replaced the flash drive, but the same symptoms still exist unfortunately. Server was slow for a few hours, then eventually died a few minutes ago. Any other ideas of things to try out? Thinking of rolling back to 6.11 to see if that resolves it?

Thanks. I'll replace the flash drive tonight and get back to you.

Happened again, but some new logs in syslog right before the crash: Oct 10 06:06:22 Omar winbindd[15096]: [2023/10/10 06:06:22.578378, 0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn) Oct 10 06:06:22 Omar winbindd[15096]: open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_IO_TIMEOUT ... Oct 10 06:30:15 Omar usbhid-ups[5571]: nut_libusb_get_report: Input/Output Error Oct 10 06:30:17 Omar usbhid-ups[5571]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 10 06:30:17 Omar usbhid-ups[5571]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 10 06:30:17 Omar upsd[5575]: Data for UPS [ups] is stale - check driver Oct 10 06:30:17 Omar upsmon[5579]: Poll UPS [[email protected]] failed - Data stale Oct 10 06:30:17 Omar upsmon[5579]: Communications with UPS [email protected] lost Oct 10 06:30:19 Omar usbhid-ups[5571]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 10 06:30:19 Omar usbhid-ups[5571]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 10 06:30:21 Omar usbhid-ups[5571]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 10 06:30:21 Omar usbhid-ups[5571]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 10 06:30:22 Omar upsmon[5579]: Poll UPS [[email protected]] failed - Data stale ... (some more of the UPS errors) ... Oct 10 06:30:43 Omar usbhid-ups[5571]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 10 06:30:43 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:43 Omar kernel: SQUASHFS error: Failed to read block 0x2e0d6c4: -5 Oct 10 06:30:43 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:43 Omar kernel: SQUASHFS error: Failed to read block 0x2e4d66c: -5 Oct 10 06:30:43 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:43 Omar kernel: SQUASHFS error: Failed to read block 0x10722d14: -5 Oct 10 06:30:43 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:43 Omar kernel: SQUASHFS error: Failed to read block 0x2e4d66c: -5 Oct 10 06:30:43 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:43 Omar kernel: SQUASHFS error: Failed to read block 0x10732670: -5 Oct 10 06:30:43 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:43 Omar kernel: SQUASHFS error: Failed to read block 0x2e4d66c: -5 Oct 10 06:30:44 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:44 Omar kernel: SQUASHFS error: Failed to read block 0x2e4d66c: -5 Oct 10 06:30:44 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:44 Omar kernel: SQUASHFS error: Failed to read block 0x10722d14: -5 Oct 10 06:30:44 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:44 Omar kernel: SQUASHFS error: Failed to read block 0x2e4d66c: -5 Oct 10 06:30:44 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:44 Omar kernel: SQUASHFS error: Failed to read block 0x2e4d66c: -5 Oct 10 06:30:44 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:44 Omar kernel: SQUASHFS error: Failed to read block 0x10722d14: -5 Oct 10 06:30:44 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:44 Omar kernel: SQUASHFS error: Failed to read block 0x2e4d66c: -5

It does not, RAM usage seems to be pretty stable.

To rule out any issue with docker containers - ran the server with all containers stopped, and issue still persisted with a server crash. I guess it's an issue with the docker service itself. Any idea on potential fixes? Netdata: After the spike in iowait, server goes down and is unresponsive unless manually hard restarted. Stats across the board also drop (i.e., disk io, network, RAM, etc....) Syslog right before this latest crash: Oct 8 15:01:50 Omar usbhid-ups[5576]: nut_libusb_get_report: Input/Output Error Oct 8 15:01:52 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:01:52 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:01:52 Omar upsd[5580]: Data for UPS [ups] is stale - check driver Oct 8 15:01:54 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:01:56 Omar upsmon[5584]: Poll UPS [[email protected]] failed - Data stale Oct 8 15:01:56 Omar upsmon[5584]: Communications with UPS [email protected] lost Oct 8 15:01:58 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:01:58 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:01:59 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:00 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:01 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:01 Omar upsmon[5584]: Poll UPS [[email protected]] failed - Data stale Oct 8 15:02:02 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:03 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:04 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:05 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:06 Omar upsmon[5584]: Poll UPS [[email protected]] failed - Data stale Oct 8 15:02:06 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:07 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:08 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:09 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:10 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:11 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:11 Omar upsmon[5584]: Poll UPS [[email protected]] failed - Data stale Oct 8 15:02:12 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:13 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:15 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:15 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:16 Omar upsmon[5584]: Poll UPS [[email protected]] failed - Data stale Oct 8 15:02:17 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:17 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:19 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:19 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:21 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:21 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:21 Omar upsmon[5584]: Poll UPS [[email protected]] failed - Data stale Oct 8 15:02:23 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:23 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:25 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround

Tried that, as well as tried docker image instead of folders. Still persists issue.

Another thing I noticed - is if the dashboard is open on a tab, system load and iowait goes up, but seems to go down to normal-ish after it is closed. Scratch that, went back to very high CPU usage a short while. (Initial Graph) (10-ish mins later)

Just happened again - noticed these lines in syslog shortly before server froze and needed to be hard reset - any ideas if related? Oct 6 00:31:39 Omar usbhid-ups[4724]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 6 00:31:39 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:31:40 Omar kernel: usb 3-3: USB disconnect, device number 4 Oct 6 00:31:40 Omar usb_manager: Info: rc.usb_manager usb_remove American_Power_Conversion_Back-UPS_BX750MI_FW:8_T©____-302202G /dev/bus/usb/003/004 003 004 Oct 6 00:31:40 Omar usb_manager: Info: rc.usb_manager Device Match 003/004 vm: 003 004 Oct 6 00:31:40 Omar usb_manager: Info: rc.usb_manager Removed 003/004 vm: nostate 003 004 Oct 6 00:31:40 Omar kernel: usb 3-3: new full-speed USB device number 5 using xhci_hcd Oct 6 00:31:40 Omar kernel: hid-generic 0003:051D:0002.0003: hiddev96,hidraw0: USB HID v1.10 Device [American Power Conversion Back-UPS BX750MI FW:219A3541-302202G ] on usb-0000:00:14.0-3/input0 Oct 6 00:31:40 Omar usb_manager: Info: rc.usb_manager usb_add American_Power_Conversion_Back-UPS_BX750MI_FW:219A3541-302202G /dev/bus/usb/003/005 003 005 Oct 6 00:31:41 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:31:41 Omar usb_manager: Info: rc.usb_manager Autoconnect No Mapping found American_Power_Conversion_Back-UPS_BX750MI_FW:219A3541-302202G /dev/bus/usb/003/005 003 005 port 3-3 Oct 6 00:31:41 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything... Oct 6 00:37:11 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:13 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:15 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:16 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:17 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:19 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:21 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:21 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:23 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:25 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:26 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:27 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:29 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:31 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:31 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:33 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:35 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:36 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:37 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:39 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:41 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:41 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:43 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:45 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:46 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:47 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:49 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:51 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:51 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:53 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:55 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:56 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:57 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:59 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything I also installed netdata to see if that would help, and there's a large spike in iowait right before things go down. Also, in an effort to narrow down the issue, I've also uninstalled the unassigned device plugin family, and switched to ipvlan as well. Still no dice.

Let me give that a shot. I currently use docker folders instead of image. Is there a reason to switch over to docker image?

Disabling docker service does indeed seem to resolve the issue, but I cannot seem to trace it to any docker containers. If I stop all containers, the issue still persists. I did two things and it seems to have helped a bit (server only died about ~3d in this time, vs 2-5 hours), but not entirely sure of how related they are: implemented the macvlan fix, and used binhex's libtorrentv1 qbit image. Interesting. Can you please share more details, would be great to see if I could isolate it to the same issue as you!