JesterEE
-
Posts
169 -
Joined
-
Last visited
Content Type
Profiles
Forums
Downloads
Store
Gallery
Bug Reports
Documentation
Landing
Report Comments posted by JesterEE
-
-
49 minutes ago, Shomil Saini said:
Any update on stability of linuxserver/deluge:2.1.1-r3-ls179 container.
Thanks in Advance.
Been running it continuously since my 11/18/2022 post. No issues.
-
1
-
-
8 hours ago, ShadyDeth said:
Assuming you want to stay on Deluge and have the VPN option, easiest workaround I found was to install the linuxserver.io Deluge 2.1.1 libtorrent v1 @JesterEE posted about.
I checked the repo again just now, this is still the latest LSIO release on libtorrent v1.
8 hours ago, ShadyDeth said:One thing I will say is make sure you have Privoxy running first or else Deluge will use your IP and not the VPN IP. Not sure why when it's network is routed through the Privoxy container. I set a start wait on deluge (Docker > Advanced View > fill in the wait on autostart) to fix this on any reboots. Also if you're running "CA Backup / Restore Appdata" that messes with this whole set up since it restarts all your docker containers at the same time when its doing its back up, and deluge has used my ip every time instead of the vpn. So I've disabled it for now.
I use the Gluetun container for my VPN and I've never seen this issue. Actually just the opposite. I intentionally test this from time to time to see if I'm leaking my IP and when the VPN is off and does not revert to the default internet connection (essentially a built in kill switch). I do not create a custom docker network as this write-up has shown. Instead, in the template for the container you want to use the VPN network, I set:
Network Type: None
and add
--network=container:VPN_CONTAINER_NAME
on the extra parameters line. I'm pretty sure this is essentially doing the same thing except without naming the network, so I'm not sure why we have different experiences with dropped connections.
What is important to note, doing it this way will require the client containers to rebuild when the VPN container is updated. This is because docker needs to point the clients (deluge, etc.) to the new endpoint since it has a new hash associated with the VPN container. So when you update your VPN container via the WebUI, since Unraid 6.9 (I think), the OS has been smart enough to rebuild the attached containers automatically, and after a minute or so for rebuild and restarting the client containers, all is well.
However, if the VPN container gets updated automatically by the Auto Update Applications plugin, the rebuild will not be triggered (since this rebuild control is implemented in the Docker WebUI php code), and all clients will lose their network connection. This will still not leak my IP and revert to the default network, but the client containers will just have no network connectivity. So, in the Auto Update Applications settings, I turn autoupdate off for the VPN client and do that one manually from time to time.
Hope this helps!
-
3+ days torrent uptime, no crashes in 6.11.5.
I'm satisfied with the current resolution that libtorrent v2 is to blame.
If you all want to get back on v2, I suggest following the open issue on the libtorrent tracker to see when they correctly support transparent hugepages.
-
1
-
-
Quick note to deluge users that are now using the older version I linked a couple days ago, I updated the post with a newer release since I had an issue with state corruption between restarts. See the updated comment here.
I did a couple quick tests (starting/stopping/restarting the container) and I don't see the same corruption occurring on the newer version of deluge still with libtorrent v1.
-
1
-
1
-
-
@binhex you run one of the more widely used qbittorrent images in the community. Have you seen many more reports on your support channels?
-
Day 4 and I got this somewhat related error spammed in the deluge docker container log.
Quoteterminate called after throwing an instance of 'libtorrent::libtorrent_exception'
what(): invalid type requested from entryThe client is not accessible, but Unraid itself seems unaffected (WebUI, SSH, etc.). I tried to restart the docker container and the deluge daemon is not starting correctly yielding the same error being reported right away.
I'm going to restart the server and see what happens. I'll edit this post when I test out the container after the restart.
POST RESTART UPDATE
The Unraid OS restart stopped my array without issue so no parity check on reboot! I'm very happy about that, I was getting sick of dirty restarts. After restarting my deluge container, I was still getting the same error as above. I did some debugging, and it looks like my session.state file in the appdata got corrupted (this is a known issue in the before-days which I haven't had in a while - maybe about a year). Once I pulled in a copy of the state file from the backup, everything returned to normal. FYI for those that don't deluge, it makes one backup of the settings and state files in the app directory, but in my experience, it is not good about backing up only known good files ... I've had the built-in backup store a corrupted file before so milage may vary. It would be a good idea to get the state file from the Unraid CA Backup / Restore Appdata plugin if you have it installed (which, come on, we all should).
I'll restart my container and see how long it goes till this comes up again. But overall, this was much nicer bad experience in that libtorrent didn't trigger a kernel error that took down the server. 😆
First, I'm going to upgrade to 6.11.5 and go from there.
-
72 hours of running Deluge on libtorrent v1 and no crashes. In every other test it has crashed before this point, so all signs point to this is the true issue.
Looking at this thread on the kernel bug tracker is looks like some sort of operating system configuration bug. I guess we are at the mercy of the Linux gods.
What I find interesting is that this should be a very widespread and a somewhat prohibitive issue with an internet full of bittorent users on newer Linux distros. Yet the chatter is relatively low on +1s and reports to having the same issue. Is it not affecting everyone the same way?
-
1
-
1
-
-
4 minutes ago, SergeantCC4 said:
I am using neither Qbittoreent or deluge and my server has crashed multiple times over the last two weeks since using 6.11.1.
Looking at your syslog, this doesn't look like the same problem as this thread is addressing.
You have a ton of BTRFS errors. Maybe search that on the forum and see what you find.
-
1
-
-
You know, now that this has all come to light ... it makes sense that this is a libtorrent issue because the only people seeing it are either running deluge or qbittorent; the 2 most popular clients that use this library (Libtorrent - Projects).
-
UPDATE: I had an issue where the 'old' version of deluge (2.0.5) was causing state corruption on docker power off. I know this has been resolved more recently (maybe by libtorrent v2, maybe in deluge, I'm not sure) as I have not had this problem in months running the latest LSIO docker. So, I looked in the source repo, again, and found a newer version of deluge (2.1.1) built with libtorrent-rasterbar v1. This may fix the issue, I do not know yet. Either way, it's newer than the link from the original post.
2.1.1-r3-ls179 - Dockerhub Link
ORIGINAL POST:
For those of us that use Deluge (from Linuxserver), I snooped the source and found that the last libtorrent-rasterbar with a version < 2 was 2.0.5-0202202181752ubuntu20.04.1-ls140 - Dockerhub Link.
This was released 9 months ago! At that point they switched the base image to Alpine and pointed to either 3.16 or edge; which now only has libtorrent-rasterbar v2 in the Alpine package index. I've updated this docker daily since forever, so I have definitely been using libtorrent-rasterbar v2 for months. So, @Altwazar and @binhex are probably right in that there is some new Linux kernel interaction in the 6.11 series that is triggering the error.
I'll try and roll another container with this version over my lunch break and see if the kernel gets mad.
Update: Pulled and running. We'll see what we see 😆.
-
3
-
-
I think the most important part is this is seemingly not an Unraid specific issue and seems like some funky interaction between libraries.
Does it still make sense to track this issue here?
-
I use Deluge, and in the linuxserver/deluge:latest docker it's pulling libtorrent 2.0.8.0 which causes the same issue. I'm at work, but later I'm going to try and load up some older versions that use libtorrent v1 and see if it persists.
-
23 hours ago, Altwazar said:
I don't use unraid, but same problem occurs with heavy usage and qbittorrent with libtorrent-rasterbar version 2 (https://github.com/arvidn/libtorrent/issues/6952). Qbittorrent compiled with version 1.2 don't solved issue for me.
Thank you very much for joining our community forum just to let us know that this is repeatable on other Linux systems and it's an application error rather than a kernel error!
-
On 11/15/2022 at 12:58 PM, JesterEE said:
fio --directory=/torrents --name=iops-test-job --ioengine=libaio --rw=randrw --bs=4k --iodepth=256 --direct=1 --group_reporting --eta-newline=1 --end_fsync=1 --time_based --size=2GB --numjobs=25 --runtime=86400
I'll run this for a few days and ... we'll see what we see.
48 hours of continuous RW and still not triggering the error. 24 more and I'm ganna call it a pass.
Still looking for insight on network stress testing.
-
1
-
-
On 11/14/2022 at 11:06 AM, JesterEE said:
Still running 6.11.2, but I'll run this test for a couple days and see what happens w/o my torrent app on. Since @ShadyDeth reported the crash still happens in 6.11.3 and nothing concerning this issue has changed, we'll see what we see.
I was playing with fio some and I settled on this command to test the system for random read/writes to emulate how I think a torrent client should perform while simultaneously downloading/seeding multiple files:
fio --directory=/torrents --name=iops-test-job --ioengine=libaio --rw=randrw --bs=4k --iodepth=256 --direct=1 --group_reporting --eta-newline=1 --end_fsync=1 --time_based --size=2GB --numjobs=25 --runtime=86400
This creates 25 2GB files (50GB total) in the /torrents directory (must be mapped in the docker template) and tests random read/write performance for the duration of runtime (in seconds). I used timeanddate.com to create meaningful durations for my schedule.
I'll run this for a few days and ... we'll see what we see. 😉
If this turns up a big nothing-burger I think the next logical test would be the docker network interface. I have never done this but found that either netstress or iperf may make 2 good candidates. Does anyone have experience with these tools or Linux network stress testing in general?
-
On 11/10/2022 at 1:23 PM, JorgeB said:
Since the best indication we have for now is that high i/o with a docker container is causing this issue anyone willing try please test with fio running in a docker:
Still running 6.11.2, but I'll run this test for a couple days and see what happens w/o my torrent app on. Since @ShadyDeth reported the crash still happens in 6.11.3 and nothing concerning this issue has changed, we'll see what we see.
-
13 minutes ago, dlandon said:
Are you writing your downloads to an encrypted disk? Array or UD?
Unencrypted XFS cache (separate from the array cache).
I was using UD at first (earlier part of this thread) because I never got around to moving the static non-array storage to cache when multiple caches were enabled in Unraid a few releases back. To debug and remove UD as the potential culprit of this issue, I removed all my static drive UD dependence, and ran a test with UD both installed and uninstalled and got the same error. I don't think this is a UD problem.
-JesterEE
-
No joy with 6.11.2. Almost 2 days of uptime before 💣.
Back to 6.10.3 when I get some time.
-
1
-
-
11 hours ago, JorgeB said:
There's a newer kernel, so it's worth trying for anyone affected.
Hadn't gotten around to putting 6.10.3 back on, so I guess I'll give this a shot first.
-
1
-
-
On 10/26/2022 at 7:23 PM, JesterEE said:
7 days uptime after removing my deluge docker and doing normal tasks with the server (VM, docker, plex streaming, databasing, file storage, parity/integrity checks, etc.). I'm going to install it again and see how long it stays stable. I'd bet, less than 3 days. 🤔
Well, it was actually only 26 hours and poof! This is pretty conclusive to me. Something is messed up, and it's not something I did 😝. I'm not 100% sure if it's high IO, the network interface, or docker, but any way you slice it, it's an OS issue and needs to be addressed by the devs in a future release.
If someone at @limetech posts here (or contacts me privately) by 10/29/22 asking to help debug, I will gladly do so. If not, back to 6.10.3 and a stable server for me.
-JesterEE
-
Do you have a way of using that utility in Unraid to see if it comes up without docker?
I couldn't find a Slackware package for it.
-
2 minutes ago, JorgeB said:
I'm not an advanced docker user, can you easily install something like fio and run it in a docker?
Short answer... Yes! I've never used that utility, but I just googled it and if there's value, I could run some tests.
-
3 minutes ago, JorgeB said:
It's not difficult to generate high i/o, it would be more difficult to replicate if it's docker with high i/o, anyone having this issue without using docker, or at least without using high i/o docker containers, like torrents, etc?
Why would this be any harder to do in docker than on the host?? If you have a preferred method of doing this at a shell prompt, it's easy enough to just run that command in any docker image you want.
-
5 minutes ago, JorgeB said:
That suggests high i/o might not be the cause, unless it's a delayed crash.
Sure, it could be a lot of things since torrent clients do a lot from networking to file IO. Plus, in Unraid, this includes the docker abstraction layer so that's yet another whole set of potential interactions.
This is why I asked above if there is any testing we can do natively in Unraid. I was hoping a dev would chime in with a unit test or something along those lines so we can start to get some resolution on this.
-JesterEE
-
1
-
Crashes since updating to v6.11.x for qBittorrent and Deluge users
in Stable Releases
Posted
Here is a link to the libtorrent bug tracker for this issue: https://github.com/arvidn/libtorrent/issues/6952