PhilBarker

I thought as an experiment I'd try removing one of the apps stuck on update, removing the image, and then re-install This seemed to work except now Readarr I think updated but now updates are "not-available"

I'm having this issue also I noticed a week or so ago that a few containers permanently said "apply update" I applied the update, everything seemed to go fine, but then they would still say "apply update" it certainly seems to be updating images - and after the docker run command all completes succesfully UI still shows an update pending if I re-run the update, it pulls the same 5 updated images, reports success - and rinse and repeat this is growing now - it was originally readarr and calibre, now qbittorrentvpn and mediaelch too I've also tried installing the update from the apps page - which does the exact same thing, downloads 5 new images, reports success, still shows update required. My DNS has been set to cloudflare always, but this obviously isn't a DNS issue as it resolves, and isn't a bad checksum issue as there are updated images to pull.. Not sure if there's some other logs I could check somewhere?

So this might be completely unrelated - but I hate not following up on an issue I took a day to have a proper look at my server, there's been some other issues like randomly very slow copying files, even when rsyncing from disk to disk I discovered `top` was showing a massive IO wait.. all the time, causing high CPU usage After trial and error, closing VM's, shutting down docker containers one by one - I discovered this was caused by the file integrity plugin when I had ticked to automatically generate hash's on file creation/change. I turned that off and the IO wait instantly dropped from 60-80 to 0. The whole server feels tons faster now, disk speeds are a lot higher. I had presumed that was a good setting rather than having it churn through everything once a week on a schedule, but obviously not as it sets up a load of inotify tasks and grinds the disks to a halt. Since I've sorted that issue, and rebooted, I've had 3 days now without any problem with the USB stick or the VM's - so I'm not saying they're related but I wonder as there has been loads of people with this issue on the forum and none have actually found a cause for it (unless their USB was actually bad) maybe it's just a side effect of the server basically dying from some other reason.

Well.... this morning same issue with HASS VM - it's dead But looking at logs I didn't have the error about the USB pen and unlicensed Jan 7 06:06:33 Jarvis kernel: br0: port 2(vnet2) entered disabled state Jan 7 06:06:33 Jarvis kernel: device vnet2 left promiscuous mode Jan 7 06:06:33 Jarvis kernel: br0: port 2(vnet2) entered disabled state Jan 7 06:06:34 Jarvis usb_manager: Info: rc.usb_manager vm_action HASS stopped end - Jan 7 06:06:54 Jarvis emhttpd: error: device_read_smart, 9579: Cannot allocate memory (12): device_spinup: stream did not open: sdd Jan 7 06:06:54 Jarvis emhttpd: error: device_read_smart, 9579: Cannot allocate memory (12): device_spinup: stream did not open: sdf Jan 7 06:06:54 Jarvis emhttpd: error: device_read_smart, 9579: Cannot allocate memory (12): device_spinup: stream did not open: sdc Jan 7 06:06:54 Jarvis emhttpd: error: device_read_smart, 9579: Cannot allocate memory (12): device_spinup: stream did not open: nvme0n1 Jan 7 06:06:54 Jarvis emhttpd: error: device_read_smart, 9579: Cannot allocate memory (12): device_spinup: stream did not open: sdi Jan 7 06:06:54 Jarvis emhttpd: error: device_read_smart, 9579: Cannot allocate memory (12): device_spinup: stream did not open: sda Jan 7 06:07:02 Jarvis samba-dcerpcd[9932]: [2024/01/07 06:07:02.017455, 0] ../../lib/util/sys_popen.c:80(sys_popenv) Jan 7 06:07:02 Jarvis samba-dcerpcd[9932]: sys_popenv: fork failed: Resource temporarily unavailable Jan 7 06:10:01 Jarvis kernel: docker0: port 3(veth983191f) entered disabled state Jan 7 06:10:01 Jarvis kernel: veth8a82458: renamed from eth0 Jan 7 06:10:01 Jarvis kernel: docker0: port 3(veth983191f) entered disabled state Jan 7 06:10:01 Jarvis kernel: device veth983191f left promiscuous mode Jan 7 06:10:01 Jarvis kernel: docker0: port 3(veth983191f) entered disabled state Jan 7 06:10:01 Jarvis kernel: docker0: port 3(veth26a5504) entered blocking state Jan 7 06:10:01 Jarvis kernel: docker0: port 3(veth26a5504) entered disabled state Jan 7 06:10:01 Jarvis kernel: device veth26a5504 entered promiscuous mode Jan 7 06:10:01 Jarvis kernel: docker0: port 3(veth26a5504) entered blocking state Jan 7 06:10:01 Jarvis kernel: docker0: port 3(veth26a5504) entered forwarding state Jan 7 06:10:02 Jarvis kernel: eth0: renamed from veth849cba1 Jan 7 06:10:02 Jarvis kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth26a5504: link becomes ready Jan 7 07:00:01 Jarvis root: Restarting CA Auto Turbo Mode Jan 7 07:00:01 Jarvis kernel: mdcmd (56): set md_write_method 1 Jan 7 07:00:01 Jarvis kernel: Jan 7 07:16:58 Jarvis emhttpd: spinning down /dev/sda Jan 7 07:30:54 Jarvis kernel: usb 3-1: USB disconnect, device number 2 Jan 7 07:30:54 Jarvis usb_manager: Info: rc.usb_manager usb_remove Broadcom_Corp_BCM20702A0_5CF370A921D6 /dev/bus/usb/003/002 003 002 Jan 7 07:30:54 Jarvis usb_manager: Info: rc.usb_manager Device Match 003/002 vm: 003 002 Jan 7 07:30:54 Jarvis usb_manager: Info: rc.usb_manager Removed 003/002 vm: nostate 003 002 Jan 7 07:30:54 Jarvis kernel: usb 3-1: new full-speed USB device number 5 using xhci_hcd Jan 7 07:30:54 Jarvis usb_manager: Info: rc.usb_manager usb_add Broadcom_Corp_BCM20702A0_5CF370A921D6 /dev/bus/usb/003/005 003 005 Jan 7 07:30:54 Jarvis kernel: Bluetooth: hci0: BCM: chip id 63 Jan 7 07:30:54 Jarvis kernel: Bluetooth: hci0: BCM: features 0x07 Jan 7 07:30:54 Jarvis kernel: Bluetooth: hci0: BCM20702A Jan 7 07:30:54 Jarvis kernel: Bluetooth: hci0: BCM20702A1 (001.002.014) build 0000 Jan 7 07:30:54 Jarvis kernel: Bluetooth: hci0: BCM: firmware Patch file not found, tried: Jan 7 07:30:54 Jarvis kernel: Bluetooth: hci0: BCM: 'brcm/BCM20702A1-0b05-17cb.hcd' Jan 7 07:30:54 Jarvis kernel: Bluetooth: hci0: BCM: 'brcm/BCM-0b05-17cb.hcd' Jan 7 07:30:54 Jarvis usb_manager: Info: rc.usb_manager Add device with more than one interface & connect serial . Jan 7 07:30:54 Jarvis usb_manager: Info: rc.usb_manager Autoconnect No Mapping found Broadcom_Corp_BCM20702A0_5CF370A921D6 /dev/bus/usb/003/005 003 005 port 3-1 Still looks like my bluetooth dongle randomly disconnected though The only other thing I'm finding that seems to be connected is usually a dead docker container. Today was Lidarr I looked at the rebuild-dndc logs and found this /usr/bin/docker run -d --name="binhex-lidarr" --net="container:binhex-qbittorrentvpn" -e UMASK="000" -e PUID="99" -e PGID="100" -e TZ="Europe/London" -v "/mnt/cache/appdata/data":"/data":rw -v "/mnt/user/Music/":"/media":rw -v "/mnt/short_cache/Downloads/Torrents/":"/downloads":rw -v "/mnt/user/appdata/binhex-lidarr":"/config":rw binhex/arch-lidarr 84c7e97f86ddcc9a8a3b6c8fc351e962e358f6138b986f72b93d54fab8e5e101 /usr/bin/docker: Error response from daemon: failed to start shim: start failed: runtime: failed to create new OS thread (have 7 already; errno=11) runtime: may need to increase max user processes (ulimit -u) fatal error: newosproc This occured at 06:05 and the HASS VM crashed 1 minute after

Yeah same here. My server has been up and running for maybe near a year now. I have used powertop for most of that but as you’ve confirmed the auto suspend is set as default anyway. Not had any issues until the last few weeks. Maybe this will solve it maybe it won’t. But if it does it doesn’t explain why it’s suddenly started happening

Hrmmmmmmmm sorry to spam this thread but.... Do you use powertop? I do and.. I checked /sys/module/usbcore/parameters and mine was 2. I'm going to try changing that to -1 - I'm wondering if powertop is making the usb devices suspend when it drops down to a certain c-state

Just remembered I have all syslogs backed up so checked them all Jan 1 06:04:34 Jarvis usb_manager: Info: rc.usb_manager vm_action HASS stopped end - Jan 1 06:05:00 Jarvis emhttpd: Unregistered - flash device error (ENOFLASH7) Jan 1 06:05:01 Jarvis emhttpd: Pro key detected, GUID: 0781-5571-5501-060921121721 FILE: /boot/config/Pro.key Jan 1 06:06:07 Jarvis emhttpd: error: device_read_smart, 9579: Cannot allocate memory (12): device_spinup: stream did Jan 1 06:06:07 Jarvis emhttpd: error: device_read_smart, 9579: Cannot allocate memory (12): device_spinup: stream did Jan 1 06:06:07 Jarvis emhttpd: error: device_read_smart, 9579: Cannot allocate memory (12): device_spinup: stream did Jan 1 06:06:13 Jarvis emhttpd: Unregistered - flash device error (ENOFLASH7) Jan 1 06:06:14 Jarvis emhttpd: Pro key detected, GUID: 0781-5571-5501-060921121721 FILE: /boot/config/Pro.key Jan 1 06:10:31 Jarvis root: Mealie: Could not download icon https://cdn.statically.io/gh/KakersUK/docker-templates/mas Jan 1 06:48:15 Jarvis kernel: usb 7-1: USB disconnect, device number 6 Jan 1 06:48:15 Jarvis usb_manager: Info: rc.usb_manager usb_remove Broadcom_Corp_BCM20702A0_5CF370A921D6 /dev/bus/usb Jan 1 06:48:15 Jarvis usb_manager: Info: rc.usb_manager Device Match 007/006 vm: 007 006 Jan 1 06:48:15 Jarvis usb_manager: Info: rc.usb_manager Removed 007/006 vm: nostate 007 006 Jan 1 06:48:15 Jarvis kernel: usb 7-1: new full-speed USB device number 7 using xhci_hcd Jan 1 06:48:15 Jarvis usb_manager: Info: rc.usb_manager usb_add Broadcom_Corp_BCM20702A0_5CF370A921D6 /dev/bus/usb/007 Jan 1 06:48:15 Jarvis kernel: Bluetooth: hci0: BCM: chip id 63 Jan 1 06:48:15 Jarvis kernel: Bluetooth: hci0: BCM: features 0x07 Jan 1 06:48:15 Jarvis kernel: Bluetooth: hci0: BCM20702A Jan 1 06:48:15 Jarvis kernel: Bluetooth: hci0: BCM20702A1 (001.002.014) build 0000 Jan 1 06:48:15 Jarvis kernel: Bluetooth: hci0: BCM: firmware Patch file not found, tried: Jan 1 06:48:15 Jarvis kernel: Bluetooth: hci0: BCM: 'brcm/BCM20702A1-0b05-17cb.hcd' Jan 1 06:48:15 Jarvis kernel: Bluetooth: hci0: BCM: 'brcm/BCM-0b05-17cb.hcd' Jan 1 06:48:15 Jarvis usb_manager: Info: rc.usb_manager Add device with more than one interface & connect serial . Jan 1 06:48:15 Jarvis usb_manager: Info: rc.usb_manager Autoconnect No Mapping found Broadcom_Corp_BCM20702A0_5CF370A9 almost the exact same timing when this happened last week but you can see that my bluetooth dongle disconnected, then reconnected - which is probably why HASS crashes as that device is passed through to the VM So it's almost like all the usb devices disconnect and reconnect. But I have absolutely no clue why

Yeah it's weird. I've had a bunch of bizarre issues on my server over Christmas... resolved all of them by swapping out my network card, finally ditched my pcie sata card for a proper HBA... now have nothing coming up in my syslog except every day I find those and one of my VM's is dead. Weirdly my other VM isn't affected (alpine linux) but the home assistant one crashes, and occasionally I find a few dead docker containers too - all die at the time the syslog shows the flash device errors. I've tried testing the usb stick in a PC and it seems fine, I can't see any errors about usb dropping or anything.

I'm having a very similar issue - only started a few weeks ago Nearly every day I get these errors Jan 6 06:06:40 Jarvis emhttpd: Unregistered - flash device error (ENOFLASH7) Jan 6 06:06:41 Jarvis emhttpd: Pro key detected, GUID: 0781-5571-5501-060921121721 FILE: /boot/config/Pro.key Jan 6 06:07:06 Jarvis emhttpd: error: device_read_smart, 9579: Cannot allocate memory (12): device_spinup: stream did not open: sdf Jan 6 06:07:06 Jarvis emhttpd: error: device_read_smart, 9579: Cannot allocate memory (12): device_spinup: stream did not open: sdc Jan 6 06:07:06 Jarvis emhttpd: error: device_read_smart, 9579: Cannot allocate memory (12): device_spinup: stream did not open: nvme0n1 Jan 6 06:07:06 Jarvis emhttpd: error: device_read_smart, 9579: Cannot allocate memory (12): device_spinup: stream did not open: sdi Jan 6 06:07:06 Jarvis emhttpd: error: device_read_smart, 9579: Cannot allocate memory (12): device_spinup: stream did not open: sda Jan 6 06:07:16 Jarvis kernel: br0: port 2(vnet0) entered disabled state Jan 6 06:07:16 Jarvis kernel: device vnet0 left promiscuous mode Jan 6 06:07:16 Jarvis kernel: br0: port 2(vnet0) entered disabled state Jan 6 06:07:16 Jarvis usb_manager: Info: rc.usb_manager vm_action HASS stopped end - Jan 6 06:07:22 Jarvis emhttpd: Unregistered - flash device error (ENOFLASH7) Jan 6 06:07:23 Jarvis emhttpd: Pro key detected, GUID: 0781-5571-5501-060921121721 FILE: /boot/config/Pro.key Jan 6 06:08:00 Jarvis emhttpd: Unregistered - flash device error (ENOFLASH7) Jan 6 06:08:01 Jarvis emhttpd: Pro key detected, GUID: 0781-5571-5501-060921121721 FILE: /boot/config/Pro.key And my home assistant VM is crashed every morning I don't believe there's anything wrong with the USB pen, it's not very old and other thanit disconnecting around 6am it has no issues. I really don't know what's going on - it seems to fairly consistently happen just after 6am but I can't see anything else running/scheduled around that time

Just to close this off in case anyone else has the same issue in the future This 100% was my problem - the disk must have been formatted in my old server and transferred across without being wiped. It was the only disk with a 1MiB GPT, all the others are 4KiB. So when I formatted from exfs to zfs the size of the disk reported changed. The parity restore worked flawlessly and no lost data. The only other weird issue I had was the file integrity plugin started saying the hash's had changed for most of the files on disk3, again likely because of the GPT. I simply checked the files were ok, then removed the hash's and recalculated. All 4 of my other disks reformatted to ZFS without issues 😄

I'm not sure if the actual script itself supports a blank password, but you can try clicking the edit button next to the password field and changing "required" to no Then click save and you'll be able to continue with the field blank If that doesn't work you can try clicking the remove button next to the password fields, then they won't be passed to the container at all Let me know how you get on!

The only issue here is that folders on a 24tb array can't be moved to a 2tb cache and then back again. I'm going to have a play with https://github.com/SpaceinvaderOne/Unraid_ZFS_Auto_Dataset_Update to see if it can do regular folders on array disks. I believe it can

Thanks, I've kicked off a rebuild - fingers crossed! Your explanation probably makes sense, I had a native ubuntu server before moving to unraid, the disk might have been moved over from there and I forgot to zero it out first.

As per the instructions on the other thread I've tried starting the array with the disk removed. All files seem to be emulated ok, I can access them without issues. Attached new diagnostics with the disk now emulated. Does this mean I'm best just to add the disk back in and let it rebuild it from parity? jarvis-diagnostics-20231212-2129.zip

I've just had this exact same issue - I'm going through moving files off of one disk at a time to the rest of the array, then swapping to ZFS The last disk I changed to ZFS, moved all the files off the next disk onto it, when I stopped the array I got the error saying it was the wrong disk. And checking the disk logs the reported size changed after the ZFS format, but the array config obviously didn't update. Not good! Posted a new thread as I'm currently mid-panic and hoping for some help 😱