Crashes since updating to v6.11.x for qBittorrent and Deluge users

30 minutes ago, CiscoCoreX said:

Should i try to have it enabled?

Yes, see if it makes any difference.

42 minutes ago, JorgeB said:

Yes, see if it makes any difference.

It's enabled now.

I'll chime in that I'm having the exact same issues. Was running 6.92 with a year of up time. Did some hardware upgrades and had almost 40 days uptime with a couple lock ups and decided to update after some troubleshooting. Updated to 6.11.1 a week ago I started getting the soft lock where I could only see the unraid banner in the webgui. I can only hard reboot and it seems to lock up after 2-3 days every time now. Keyboard is also unresponsive plugged directly into server after this happens.

 

Everything seems to function except Binhex-Qbittorrent from what I can tell. I've enabled Pre-allocated disk space and removed the following plug ins to see if it makes any difference.

 

Unassigned Devices

Unassigned Devices Plus

File Activity

 

Ok, I removed both plugins below and updating Binhex-DelugeVPN to pre-allocate.  I don't want to delete CA Backup as that is more useful/needed.  If I still lockup, I will try that.

 

dynamix.active.streams.plg - 2020.06.17 (Up to date)

file.activity.plg - 2022.08.19 (Up to date)

Since disabling the 7 plugins that were found to be in common with most users having this issue, I let my server run for 24hrs with no issues before reinstalling each plugin 1 by 1 and letting the server run for a full 24hrs before installing another one.

 

• Oct 12th: Full 24hrs with none of the 7 plugins installed
• Oct 13th: Reinstalled Community Applications (community.applications.plg). Ran full 24hrs, no issues.
• Oct 14th: Reinstalled CA Backup / Restore Appdata (ca.backup2.plg). Ran full 24hrs, no issues.
• Oct 15th: Reinstalled Fix Common Problems (fix.common.problems.plg). Ran full 24hrs, no issues.
• Oct 16th: Reinstalled Unassigned Devices (unassigned.devices.plg). Installed at 1pm EST on the 16th, server crashed at 10am EST on Oct 17th.

 

This could just be a coincidence and be one of the other plugins causing the problem, But it's looking like it might be Unassigned Devices.

 

I have one 500GB ssd on Unassigned Devices for saving downloaded torrents onto via binhex-delugevpn.

Running binhex-delugevpn using wireguard vpn option. The "Pre-allocate disk space" was turned off but I have now enabled it to see if that might help the issue somehow from other posts above.

 

Just for testing purposes I'm going to uninstall the 3 plugins I installed before Unassigned Devices to see if I get another crash within a few days time.

 

Including my diagnostics from the crash today.

impulse-diagnostics-20221017-1545.zip

Thanks for the report @ShadyDeth.  I did NOT remove unassigned devices only because I consider it fairly critical.  My error is identical in almost every way to those posted in this thread - and yet here I am on day 5 of no crashes after removing a few plugins I documented from a few posts up.  I even let qbittorrent run full-bore last night thinking it may just be a load issue - but to my surprise, it was still running this morning.  Maybe I'm just getting lucky.  Another few days should tell the tale...though I haven't been able to run consecutively for 5 days straight since coming off of 9.6.x...

 

Here's to no crashes...

Edited October 18, 2022 by sundown

1 hour ago, sundown said:

Thanks for the report @ShadyDeth.  I did NOT remove unassigned devices only because I consider it fairly critical.  My error is identical in almost every way to those posted in this thread - and yet here I am on day 4 of no crashes after removing a few plugins I documented from a few posts up.  I even let qbittorrent run full-bore last night thinking it may just be a load issue - but to my surprise, it was still running this morning.  Maybe I'm just getting lucky.  Another few days should tell the tale...though I haven't been able to run consecutively for 4 days straight since coming off of 9.6.x...

 

Here's to no crashes...

 

I had another crash today.  

 

Previous troubleshooting steps:

Changed Docker to ipvlan

Uninstalled dynamix.active.streams

Uninstalled file.activity

Set DelugeVPN to pre-allocate

 

Steps taken since the crash.

Uninstalled CA Backup / Restore (ca.backup2)

 

Lets see how long I can run it now without crashing.  if it crashes again, I may just revert back to 6.10.x since I don't think there is anything else I haven't tried mentioned in this thread besides uninstalling UD which is not an option with my setup.

 

Attaching additional diagnostics from today.

debo-server-diagnostics-20221017-1336.zip

On 10/12/2022 at 4:44 PM, binhex said:

As you can probably see from my list of plugins i only have the following installed from your list of commonly installed plugins, which does shorten the list quite a lot, although i am only a metric of 1 so far so take it with a large pinch of salt 🙂

community.applications.plg - 2022.09.30 (Up to date)
unassigned.devices.plg - 2022.10.03 (Up to date)
unassigned.devices-plus.plg - 2022.08.19 (Up to date)

 

I am loathed to uninstall CA for obvious reasons, but i have nuked UD, so we shall see what happens.

 

On 10/13/2022 at 11:14 AM, binhex said:

i was using macvlan for one docker container, but have switched it to ipvlan now.

 

@ich777 thanks for the pm.

no crashes for me since doing the above, uptime 4 days 20 hours and counting, so MAYBE (a big maybe) it is related to Unassigned Devices (uninstalled after first (and only) crash as mentioned above), luckily for me i don't rely on UD, i only had it installed as i was previously playing with pre-clear on a USB connected drive.

7 minutes ago, binhex said:

 

no crashes for me since doing the above, uptime 4 days 20 hours and counting, so MAYBE (a big maybe) it is related to Unassigned Devices (uninstalled after first (and only) crash as mentioned above), luckily for me i don't rely on UD, i only had it installed as i was previously playing with pre-clear on a USB connected drive.

 

I hate that a pattern isn't necessarily emerging here - programmatically it doesn't make a whole lot of sense to me.  Unassigned Devices was really one of the only plugins I DIDN'T remove, and I'm now sitting at an uptime of 6d3h from doing nothing else after having 14 consecutive crashes.  I did have qbittorrent stop seeding torrents a few times, but the container didn't lock and the system stayed responsive and I was able to simply restart the container via the Docker interface and it's been running along just fine since. 

 

There are new versions of Fix Common and the Community Apps plugin available.  I'm hesitant to update just because they would complicate/nullify the testing of this issue...  I checked the logs for both, and nothing mention any changes that would seemingly impact this situation either.

Crashed again last night.  I bit the bullet and reconfigured everything that was using UD and uninstalled UD/UD+ plugins.

 

We'll see....

11 hours ago, binhex said:

 

no crashes for me since doing the above, uptime 4 days 20 hours and counting, so MAYBE (a big maybe) it is related to Unassigned Devices (uninstalled after first (and only) crash as mentioned above), luckily for me i don't rely on UD, i only had it installed as i was previously playing with pre-clear on a USB connected drive.

 

I have to agree. I only did 2 things on this cycle which currently has an uptime of 2 days 9 hours.

1. Switched docker macvlan -> ipvlan (as noted above in another comment, this doesn't seem to be it)
2. Moved all my drives off of Unassigned Devices and made them Pool Devices.

 

All my plugins remain installed and fully updated as of this posts timestamp (i.e. all the ones @JorgeB listed in the OP and more).

 

I'm thinking it was #2 and something is going on when interfacing UD with a lot of IO traffic (like torrenting) on the 6.11 series. I was previously using one of my UD drives for torrent seeding using LinuxServer.io's deluge container and the Gluetun VPN client docker container. I still use the same containers but moved my UD xfs formatted drive to a pool device (which, for those of you that haven't done this yet, will NOT destroy your data if you don't change the filesystem) and have not had another crash since. I also was using UD for my VM drive, Plex database, and scratch drive so UD was previously doing a lot of heavy lifting. But I really think it was the torrent traffic that did it in because I was often able to access the other features of my server (VMs, plex and docker containers) that were utilizing UD AFTER the effects from the OP were noted.

 

I'm going to let it sit for another 15 hours or so and then switch back to docker macvlan to unfortunately, really point the finger at UD.

 

-JesterEE

In my case I don't use UD like most of you do. I only use it for mounting remote share.
3 days uptime now.
I have a dedicated disk for only downloads that qbittorrent are using. Is in my Pool Devices.

The the cache pool, i have appdata, where my qbittorrent are installed.

When this crash happens, i can't kill/restart qbittorrent. And I can't stop the Array, because it says cache disk is busy.....

So i always have to do a soft reboot of my unraid server.

 

Oct 13 05:58:09 Pegasus  emhttpd: shcmd (835): umount /mnt/cache
Oct 13 05:58:09 Pegasus root: umount: /mnt/cache: target is busy.
Oct 13 05:58:09 Pegasus  emhttpd: shcmd (835): exit status: 32
Oct 13 05:58:09 Pegasus  emhttpd: Retry unmounting disk share(s)...
Oct 13 05:58:14 Pegasus  emhttpd: Unmounting disks...
Oct 13 05:58:14 Pegasus  emhttpd: shcmd (836): umount /mnt/cache
Oct 13 05:58:14 Pegasus root: umount: /mnt/cache: target is busy.
Oct 13 05:58:14 Pegasus  emhttpd: shcmd (836): exit status: 32
Oct 13 05:58:14 Pegasus  emhttpd: Retry unmounting disk share(s)...
Oct 13 05:58:19 Pegasus  emhttpd: Unmounting disks...
Oct 13 05:58:19 Pegasus  emhttpd: shcmd (837): umount /mnt/cache

 

What I did:

- Made a script to restart my qbittorrent everyday at the same time. (not sure if this helps

- Enable "Pre-allocate disk space for all files" in my qbittorrent

- Did some other changes in advance tab also. https://imgur.com/a/3dsQ3gF  What i notice in the advance tab. I can't change noting for  Disk IO - types, read mode of write mode. Even if you change something there, it's just get blanked out like you see in the image. Not sure if this has something to do with this crash problem.

- Switched docker macvlan -> ipvlan

 

Edited October 19, 2022 by CiscoCoreX

 Not easy to find the cause so far but please keep posting the things you are trying to see if we can get to a conclusion, I think it's our best bet for now.

12 hours ago, JesterEE said:

I'm going to let it sit for another 15 hours or so and then switch back to docker macvlan to unfortunately, really point the finger at UD.

 

Welp ... glad I waited that extra 15 hours because it happened again just now.

 

Attached diagnostics for those interested.

 

After my restart, I'll be uninstalling Unassigned Devices and reverting to docker macvlan.

 

-JesterEE

 

 

cogsworth-diagnostics-20221019-1156.zip

9 hours ago, JorgeB said:

 Not easy to find the cause so far but please keep posting the things you are trying to see if we can get to a conclusion, I think it's our best bet for now.

You are right, maybe bad idea to make that restart script. I disable it now

 

What I did:

- Enable "Pre-allocate disk space for all files" in my qbittorrent

- Switched docker macvlan -> ipvlan

 

- Still running plugins:

 

1. fix.common.problems.plg ... 2022.10.17
2. community.applications.plg ... 2022.10.16
3. user.scripts.plg ... 2022.08.01
4. unassigned.devices.plg ... 2022.10.12
5. unassigned.devices-plus.plg ... 2022.08.19
6. page.notes.plg ... 2021.07.17
7. open.files.plg ... 2022.08.19
8. nvidia-driver.plg ... 2022.10.05
9. nut.plg ... 2022.03.20
10. intel-gpu-top.plg ... 2022.09.27
11. gpustat.plg ... 2022.02.22
12. dynamix.unraid.net.plg ...
13. dynamix.system.temp.plg ... 2022.09.16b
14. dynamix.system.stats.plg ... 2022.05.20a
15. dynamix.system.buttons.plg ...2022.06.20
16. dynamix.file.manager.plg ... 2022.09.07
17. docker.folder.plg ... 2022.09.24
18. corsairpsu.plg ... 2021.10.05
19. corefreq.plg ... 2022.07.21 DISABLE
20. ca.mover.tuning.plg ... 2022.04.13 DISABLE
21. ca.backup2.plg 2022.07.23

 

I have been crash free for 36 hours now after uninstalling Unassigned Devices plugins.  I wasn't able to go 24 hours in uptime since I installed 6.11.1.

 

I will report back if I see another crash. 

10 hours ago, JesterEE said:

After my restart, I'll be uninstalling Unassigned Devices and reverting to docker macvlan.

 

 

Crashed again after 8 hours. Diagnostics attached (it's the same though)

 

This time, before restarting dirty I wanted to try to get back to my webUI. Still couldn't, but since I can ssh in, I decided to try and stop all my dockers with command:

 

docker stop $(docker ps -q)

 

This hung for a moment but worked. The only container it couldn't stop is my torrent container (deluge). But after I stopped all the others, I was able to get back to my local WebUI and proceed with a normal shutdown. This is kinda weird to me, because I don't think that should have made a difference to the Unraid web backend, but I'm not going to think too much into it; it's all weird right now.

 

When I stopped my array (yes, I usually manually stop my array before a shutdown to see if I can catch bad behavior like this), it hung unmounting the cache drive (where my docker appdata resides), so while I was eventually able to issue a shut down command, I'm pretty sure it wasn't as graceful as it should be (hitting the timeout period and triggering a hard poweroff). I believe something in docker was holding onto that mount from the deluge container being hung.

 

Also notable, I was in the middle of 'yet another' parity check. I was able to see it got to 22.2% and was still chugging along like nothing went wrong.

 

This leaves me to believe it's some Unraid/docker edge case and not a plugin interaction at all.  Docker was updated between 6.10.3 and 6.11.1 so maybe something isn't quite working right on that release:

• 6.10.3 - docker: version 20.10.17 (CVE-2022-29526 CVE-2022-30634 CVE-2022-30629 CVE-2022-30580 CVE-2022-29804 CVE-2022-29162 CVE-2022-31030)
• 6.11.1: docker: version 20.10.18 (CVE-2022-27664 CVE-2022-32190 CVE-2022-36109)

 

I'm going to restart, remove my deluge docker (but keep the appdata of course!) and reinstall Unassigned Devices. If this doesn't work, I think I'm going to head back to a stable 6.10.3 till LimeTech squares this off. If it does, I'm not sure what my next step will be (suggestions?).

 

-JesterEE

 

cogsworth-diagnostics-20221019-2143.zip

Edited October 20, 2022 by JesterEE
Update log with most relevant and more info.

@JesterEE
I see we have the same problems.  And same behave with torrent program hang and busy cache disk.

 

Oct 19 21:42:31 Cogsworth root: umount: /mnt/cache: target is busy.
Oct 19 21:42:31 Cogsworth  emhttpd: shcmd (29660): exit status: 32
Oct 19 21:42:31 Cogsworth  emhttpd: Retry unmounting disk share(s)...
Oct 19 21:42:36 Cogsworth  emhttpd: Unmounting disks...
Oct 19 21:42:36 Cogsworth  emhttpd: shcmd (29661): umount /mnt/cache
Oct 19 21:42:36 Cogsworth root: umount: /mnt/cache: target is busy.
Oct 19 21:42:36 Cogsworth  emhttpd: shcmd (29661): exit status: 32
Oct 19 21:42:36 Cogsworth  emhttpd: Retry unmounting disk share(s)...
Oct 19 21:42:41 Cogsworth  emhttpd: Unmounting disks...

 

Just had another crash. This is what I came back to after a reboot.

 

 

Too early in the morning to fix this.

 

But once I get it fixed. I think im going to remove Unassigned Devices and see how long I can go without a crash. Maybe I didn't give it enough time the first go around.

 

EDIT: Looks like I got a bad SFF 8087 TO 4 x SATA cable. Ordered some new ones. Testing on hold until the new ones arrive.

Edited October 21, 2022 by ShadyDeth
update

I'm optimistic that my problem is solved with unassigned devices removed. I just passed 5 days uptime and I was previously locking up after 2-3 days.

Edited October 21, 2022 by NautilusGT

13 hours ago, ShadyDeth said:

EDIT: Looks like I got a bad SFF 8087 TO 4 x SATA cable. Ordered some new ones. Testing on hold until the new ones arrive.

 

Oooof, I've been there! Terrifying when you see your drives not there all at once.

I'm 15mins away from having stayed up 10 consecutive days.  I'm nearly ready to consider my issue solved.  These are the plugins I removed:

 

ca.backup2.plg - 2022.07.23 (Up to date)

dynamix.active.streams.plg - 2020.06.17 (Up to date)

file.activity.plg - 2022.08.19 (Up to date)

 

Give the current stability, I'm not willing to re-install them for further troubleshooting, as there is nothing there I can't live without.  I still have UD installed and it hosts my primary seed torrents with a little over 12tb total running 24x7.  At least in my situation it does not appear unassigned devices-related.

 

It's like one of my wife's unsolved murder podcasts - I know at the end I probably won't know exactly who caused it, but it doesn't keep me from wanting to know...

Well...I got to a little over 4 days of uptime before it crashed again.  I confirmed what @JesterEE noted I was able to stop all dockers and get the WebUI back so there appears to be something with docker and not just UD plugins.

 

I just reverted back to 6.10.3 as I need this system to be stable.  I wish my dev environment was also having these issues as I have no problem to keep troubleshooting there.  I will keep an eye on this thread to see if this gets resolved.

 

Attaching diagnostics.

debo-server-diagnostics-20221022-1601.zip

WELP.  After 10 days and 4 hours of uptime, I encountered this issue yet again.

 

@JesterEE Thank you tons for the docker command - I've tried several incarnations previously, but unable to get a successful response.  This allowed me to basically restart the dockers and be on my merry way. 

 

Best I can tell going back through the posts, it appears everyone experiencing the issue is running some type of torrent client - mostly deluge or qbittorrent.  Is this somehow docker engine/IO related?  There are several here not running their torrents off of UD, so that doesn't appear as obvious, either.

 

Over the 10 days I stayed running, I transferred a bit over 10tb of torrents.  I don't think that's unsubstantial - so maybe not load related at all?

 

Latest diags attached for funsies.

unraid-diagnostics-20221023-0827.zip

I also am running a docker stack with ipvlan Ryzen 5900x windows VM freezes as others stated above. Rolling back to 6.10 for now. I also had unassigned devices and community applications plugins. Windows VM typically freeze after a couple hours of idle time.