bucketphobia

Members
  • Posts

    26
  • Joined

  • Last visited

bucketphobia's Achievements

Noob

Noob (1/14)

2

Reputation

  1. Yeah, exactly. I had /mnt/*pool*/ for certain dockers like PLEX, but not all of them.
  2. For anyone who may stumble on the post in the future, I have confirmed that the fix for this specific error is to enable "Exclusive Shares" option in Unraid 6.12 and to verify that it is "Yes" on your appdata share. I have brought back the dockers I had disabled/changed, and everything runs as expected. If that option is turned off, the same iowait issue comes back nearly instantaneously.
  3. If it says no - you likely need to (1): disable secondary storage; (2): make sure that nothing on the share is on other drives
  4. Hey @DrSpaldo, I was having a very similar issue, but slightly different - the cause of mine was a dockerd log process. The solution to mine was to ensure that "Permit exclusive shares" is on, and that "Exclusive Access" is indeed active on my cache pool. Hope this helps.
  5. So as an update, I did three things as of the last post - and it appears one of them has worked, as I have ~84 hours of uptime so far with no signs of the issue happening again so far. Fingers crossed one of these is the long-term solution: 1) Updated BIOS to the latest version (in my case, I have a i7-4790K and was running on quite an old BIOS compared to the latest available 2) Enabled the "Exclusive Shares" option introduced in Unraid 6.12.4 (I had it manually done for certain heavy dockers previously like Plex, but it seems like it has helped.) 3) Disabled the KASM container (I am honestly not so sure how much this could potentially affect the outcome, but this time around I have had it stopped. I will attempt to use KASM again in the future and see if the issue is replicated) Thanks for all the help Jorge, appreciated.
  6. I've been doing some additional research, and found a few threads that I believe may be the same issue I've been experiencing. I updated BIOS as suggested there, and enabled exclusive shares. Will see how it affects the problem.
  7. Replaced the flash drive, but the same symptoms still exist unfortunately. Server was slow for a few hours, then eventually died a few minutes ago. Any other ideas of things to try out? Thinking of rolling back to 6.11 to see if that resolves it?
  8. Thanks. I'll replace the flash drive tonight and get back to you.
  9. Happened again, but some new logs in syslog right before the crash: Oct 10 06:06:22 Omar winbindd[15096]: [2023/10/10 06:06:22.578378, 0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn) Oct 10 06:06:22 Omar winbindd[15096]: open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_IO_TIMEOUT ... Oct 10 06:30:15 Omar usbhid-ups[5571]: nut_libusb_get_report: Input/Output Error Oct 10 06:30:17 Omar usbhid-ups[5571]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 10 06:30:17 Omar usbhid-ups[5571]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 10 06:30:17 Omar upsd[5575]: Data for UPS [ups] is stale - check driver Oct 10 06:30:17 Omar upsmon[5579]: Poll UPS [[email protected]] failed - Data stale Oct 10 06:30:17 Omar upsmon[5579]: Communications with UPS [email protected] lost Oct 10 06:30:19 Omar usbhid-ups[5571]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 10 06:30:19 Omar usbhid-ups[5571]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 10 06:30:21 Omar usbhid-ups[5571]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 10 06:30:21 Omar usbhid-ups[5571]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 10 06:30:22 Omar upsmon[5579]: Poll UPS [[email protected]] failed - Data stale ... (some more of the UPS errors) ... Oct 10 06:30:43 Omar usbhid-ups[5571]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 10 06:30:43 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:43 Omar kernel: SQUASHFS error: Failed to read block 0x2e0d6c4: -5 Oct 10 06:30:43 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:43 Omar kernel: SQUASHFS error: Failed to read block 0x2e4d66c: -5 Oct 10 06:30:43 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:43 Omar kernel: SQUASHFS error: Failed to read block 0x10722d14: -5 Oct 10 06:30:43 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:43 Omar kernel: SQUASHFS error: Failed to read block 0x2e4d66c: -5 Oct 10 06:30:43 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:43 Omar kernel: SQUASHFS error: Failed to read block 0x10732670: -5 Oct 10 06:30:43 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:43 Omar kernel: SQUASHFS error: Failed to read block 0x2e4d66c: -5 Oct 10 06:30:44 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:44 Omar kernel: SQUASHFS error: Failed to read block 0x2e4d66c: -5 Oct 10 06:30:44 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:44 Omar kernel: SQUASHFS error: Failed to read block 0x10722d14: -5 Oct 10 06:30:44 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:44 Omar kernel: SQUASHFS error: Failed to read block 0x2e4d66c: -5 Oct 10 06:30:44 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:44 Omar kernel: SQUASHFS error: Failed to read block 0x2e4d66c: -5 Oct 10 06:30:44 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:44 Omar kernel: SQUASHFS error: Failed to read block 0x10722d14: -5 Oct 10 06:30:44 Omar kernel: SQUASHFS error: xz decompression failed, data probably corrupt Oct 10 06:30:44 Omar kernel: SQUASHFS error: Failed to read block 0x2e4d66c: -5
  10. It does not, RAM usage seems to be pretty stable.
  11. To rule out any issue with docker containers - ran the server with all containers stopped, and issue still persisted with a server crash. I guess it's an issue with the docker service itself. Any idea on potential fixes? Netdata: After the spike in iowait, server goes down and is unresponsive unless manually hard restarted. Stats across the board also drop (i.e., disk io, network, RAM, etc....) Syslog right before this latest crash: Oct 8 15:01:50 Omar usbhid-ups[5576]: nut_libusb_get_report: Input/Output Error Oct 8 15:01:52 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:01:52 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:01:52 Omar upsd[5580]: Data for UPS [ups] is stale - check driver Oct 8 15:01:54 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:01:56 Omar upsmon[5584]: Poll UPS [[email protected]] failed - Data stale Oct 8 15:01:56 Omar upsmon[5584]: Communications with UPS [email protected] lost Oct 8 15:01:58 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:01:58 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:01:59 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:00 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:01 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:01 Omar upsmon[5584]: Poll UPS [[email protected]] failed - Data stale Oct 8 15:02:02 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:03 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:04 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:05 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:06 Omar upsmon[5584]: Poll UPS [[email protected]] failed - Data stale Oct 8 15:02:06 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:07 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:08 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:09 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:10 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:11 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:11 Omar upsmon[5584]: Poll UPS [[email protected]] failed - Data stale Oct 8 15:02:12 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:13 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:15 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:15 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:16 Omar upsmon[5584]: Poll UPS [[email protected]] failed - Data stale Oct 8 15:02:17 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:17 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:19 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:19 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:21 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:21 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:21 Omar upsmon[5584]: Poll UPS [[email protected]] failed - Data stale Oct 8 15:02:23 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 8 15:02:23 Omar usbhid-ups[5576]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 8 15:02:25 Omar usbhid-ups[5576]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround
  12. Tried that, as well as tried docker image instead of folders. Still persists issue.
  13. Another thing I noticed - is if the dashboard is open on a tab, system load and iowait goes up, but seems to go down to normal-ish after it is closed. Scratch that, went back to very high CPU usage a short while. (Initial Graph) (10-ish mins later)
  14. Just happened again - noticed these lines in syslog shortly before server froze and needed to be hard reset - any ideas if related? Oct 6 00:31:39 Omar usbhid-ups[4724]: device->Product is NULL so it is not possible to determine whether to activate max_report_size workaround Oct 6 00:31:39 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:31:40 Omar kernel: usb 3-3: USB disconnect, device number 4 Oct 6 00:31:40 Omar usb_manager: Info: rc.usb_manager usb_remove American_Power_Conversion_Back-UPS_BX750MI_FW:8_T©____-302202G /dev/bus/usb/003/004 003 004 Oct 6 00:31:40 Omar usb_manager: Info: rc.usb_manager Device Match 003/004 vm: 003 004 Oct 6 00:31:40 Omar usb_manager: Info: rc.usb_manager Removed 003/004 vm: nostate 003 004 Oct 6 00:31:40 Omar kernel: usb 3-3: new full-speed USB device number 5 using xhci_hcd Oct 6 00:31:40 Omar kernel: hid-generic 0003:051D:0002.0003: hiddev96,hidraw0: USB HID v1.10 Device [American Power Conversion Back-UPS BX750MI FW:219A3541-302202G ] on usb-0000:00:14.0-3/input0 Oct 6 00:31:40 Omar usb_manager: Info: rc.usb_manager usb_add American_Power_Conversion_Back-UPS_BX750MI_FW:219A3541-302202G /dev/bus/usb/003/005 003 005 Oct 6 00:31:41 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:31:41 Omar usb_manager: Info: rc.usb_manager Autoconnect No Mapping found American_Power_Conversion_Back-UPS_BX750MI_FW:219A3541-302202G /dev/bus/usb/003/005 003 005 port 3-3 Oct 6 00:31:41 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything... Oct 6 00:37:11 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:13 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:15 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:16 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:17 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:19 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:21 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:21 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:23 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:25 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:26 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:27 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:29 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:31 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:31 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:33 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:35 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:36 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:37 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:39 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:41 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:41 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:43 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:45 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:46 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:47 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:49 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:51 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:51 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:53 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:55 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:56 Omar upsmon[4732]: Poll UPS [[email protected]] failed - Data stale Oct 6 00:37:57 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything Oct 6 00:37:59 Omar usbhid-ups[4724]: libusb1: Could not open any HID devices: insufficient permissions on everything I also installed netdata to see if that would help, and there's a large spike in iowait right before things go down. Also, in an effort to narrow down the issue, I've also uninstalled the unassigned device plugin family, and switched to ipvlan as well. Still no dice.
  15. Let me give that a shot. I currently use docker folders instead of image. Is there a reason to switch over to docker image?