sundown

Attempting to call, via qbittorrent's "run external command" functionality, an API call from another container (cross-seed) to check if a torrent can be cross seeded at the torrent's completion. The command is: curl -XPOST http://ipaddress:2468/api/webhook --data-urlencode "name=%N". I was hoping that the LAN_ADDRESS worked both inbound and outbound - but appears to only impact incoming requests to qbittorrent (for the UI). Do I have any options?

Could anyone provide additional information here? @H2O_King89 when you mention "tried the docker ip" - are you referring to the docker internal addresses (in my case 172.17.x.x)? To clarify, I'm also attempting to run a Crossseed command (identical to the one quoted here). I have my LAN_NETWORK correctly set to 192.168.2.0/24 and have NO problems reaching the webUI from my home network (192.168.2.x) (not to be confused with the other user's issues here). However running "nc -zv 192.168.2.222 2468" from qbittorrent's command line returns a "connection timeout." Executing this from any of my other dockers succeeds.

Within the last 24hrs, I've had 2 12TB drives either go disabled or throw so many errors to be non-functional. Best I can figure it's just a fluke accident, as I've checked/replaced cables and there have been no other "events" that I have experience with that would have led to this type of failure. So at this point, I'm looking for the best approach to remediating the situation. Based on the logs, I'll assume neither are salvageable/trustworthy, and will have to be replaced. Unfortunately, replacements are still a few days out at this point. I have had prior drives fail, and I've used unbalance to move the files off, remove the drive, rebuilt the array and then add a new drive back into the mix. But with one failed and the other as good as, I'm unsure of the best approach. Any input is appreciated. Screenshot attached and diags as well: unraid-diagnostics-20221130-1825.zip

@binhex Looks like you have a new build of qbittorrent out on the same day that qbittorrent themselves released v4.5.0 with their comment: Assuming your latest build is leveraging their 4.5 release, should those of us on your release be relatively secure in reverting back to "latest"?

60+ hrs here. qbittorrent showing over 8TB up and 12TB down. It would have crashed before now under that type of load. (Not to mention I didn't catch the older version of qbittorrent reset my cache directory and moved over 2TB off of long term storage and on to NVME cache drives, that I had to then move back...). To @JesterEE's point: You'd think some of these large seedbox companies would be shouting from the rooftops that this is broken, but it's almost impossible to find any mention of this issue...

And for users with questions on this, @binhex as already provided you an answer (from https://github.com/binhex/documentation/blob/master/docker/faq/unraid.md)!

At minimum I believe @ShadyDeth mentioned he was using Deluge with this problem. Though it could be related to it also leveraging the libtorrent 2.x libraries in the same manner. I'm not terribly familiar with Deluge unfortunately.

@ShadyDeth Thanks for your continued analysis! Yes, I am using the VPN functionality of the binhex-qbittorrentvpn docker...the wireguard portion. I maintain an OpenVPN container (ich777's OpenVPN-Client) for NZBGet, etc. Additionally, I use the VPN Manager Tunnel's Wireguard so my son at college has access to some sandbox VMs with some oomph behind them. I'm a big fan of researching docker engine changes, and would love to know the answer to this!

Okay all - crashes continue. Logs have stayed the same - but the NATURE of the crash has changed. I'm not sure when this began happening, but if I had to speculate it happened with the upgrade to 6.11.3. Previously, my system interface would hang - no UI, nothing. I could still SSH into the box and run diagnostics and navigate, but I couldn't do anything else. I understand from others this was their functionality as well. Performing a "docker stop $(docker ps -q)" would kill all the dockers and allow me to restart without a reboot. Now, the only thing that hangs on the system is my torrent docker (in my case, binhex-qbittorrentvpn). The UI continues to be accessible as well as SSH. This would seem to be a good sign, however "docker stop $(docker ps -q)" no longer stops all my dockers - it leaves binhex-qbittorrentvpn "running" (zombie). Nothing I've found has been able to fix this, short of a reboot. For instance, I can run a "/etc/rc.d/rc.docker stop", which won't succeed. a "/etc/rc.d/rc.docker force_stop" does work, but upon executing a "/etc/rc.d/rc.docker start" qbittorrent looks like it is running but running a "docker logs binhex-qbittorrentvpn" shows no new entries from the last. In short, while the UI now remains functional and the rest of the system can be accessed (apparently as normal), there is no fixing the qbittorrent docker without a reboot. It's unclear to me as to what rebooting does to correct the issue as I have nothing else to go on. I say all this to sadly say I know not everyone here is using qbittorrent while experiencing this issue, thus my findings are likely in some way unique to my config. There is some sliver of hope in that upon trying to start the docker engine back up, I receive the error message "Error response from daemon: failed to allocate secondary ip address (server:192.168.3.1): Address already in use". That's not a subnet that I've been using, so maybe there's some cross-over that I'm not seeing. I'm safe to disable/delete it, so I'll turn that knob and see if it affects anything. Sorry, not much of an update from my situation.

Since I'm 100% on crashes since I've increased the qBittorrent load, I took today's crash to dig around a bit more when trying to figure out how to cleanly shutdown things. As I mentioned before, "docker stop $(docker ps -q)" no longer shuts my docker engine down completely - the qBittorrent docker continues to zombie. Yesterday turning the docker engine off via the settings tab throws an error and a reboot to fix that error throws the array into parity check. Today, instead of killing the docker engine (which apparently does zero good for my situation) I simply tried to take the array offline, which resulted in a repetitive error message above (it's repeated 100+ times during the writing of this). cache I can understand - I do write to it when I categorize a torrent into a "Movie_LongTerm" state (stored on the array). However disk5 was news to me. Disk 5 is the only disk with a "games torrent" directory on it. I'm not sure if this is an indicator of specific functionality being suspect or simply that it failed randomly why trying to do games processing (raring, moving, etc). I can say that the game torrents tend to be the most accessed and largest files on the system (some are 300-400gigs each). So while I look into @JorgeB's proposed test today, I wondered if anyone might be able to share command line commands that would be more "intrusive" in taking my array offline and/or being able to kill the qbittorrent docker WITHOUT throwing my entire system into a state of parity recheck? As a BTW: I'm at 6.11.3 now, and this is occurs. Not that there was any help that a random version increment would have corrected it, but still...

I switched a heavier IO load back over onto my Unraid qBittorrent docker yesterday. Last night at 2am the issue occurred again. Same exact (to the row) error messages. The last several times I've had this occur, I've noted that the "docker stop $(docker ps -q)" would kill every docker EXCEPT qBittorrent, which would continue in a "running" state. I could force it closed by turning off the docker engine via the Unraid UI, but turning the docker engine back on in the same manner always presented me with an error message: The only way I've been able to find to correct this issue is with a reboot. Though admittedly I'm no command line docker expert. Thus, it looks like if I put my heart into it, I'm able to reproduce this issue with a fair amount of regularity. As far as my configuration of qBittorrent goes, its PRIMARY drives are both Unassigned Devices mounted: a 1TB NVME for the initial download and a 12TB spinner also mounted by Unassigned Devices. That 12TB drive is identical to the other 6 that I have mounted as part of Unraid's primary array, so my logic is that there shouldn't be any difference of hardware there causing this particular issue. Depending on the category of torrent, however, qBittorrent may also access files on the primary Unraid array. It isn't common, but it does have that ability. Just updating with this week's random Unraid problems on this topic...

I have a need for dockers on two different docker networks to communicate with one another. My situation is as follows: I run a "none" network for most media dockers that routes traffic through an "OpenVPN" container. I run an independent "qbittorrent-vpn" docker on my "bridge" network (that I'll eventually move across to my "docker-net" network when I get this situation sorted). There are apps (such as cross-seed and autobrr) that I need to have access to the qBittorrent-vpn docker. I'm in search of a configuration/setup/command that allows routing across network, if possible. I've attempted creating a third network and connecting the containers therein, but I receive an error message that I'm unable to sort: Any input as to either an easier way to make this happen or troubeshooting steps for the error message above? Thanks.

Also Ryzen. Moved most of my torrents from unraid to a seedbox (might be averaging 1.5mibs now), and the frequency for me has stayed about the same: Once every 5-8 days - longest I've ever gone is 10.5 days. My logs are the same as the logs I've posted 12 or so times before, so I won't bore with duplicate details. Now I'm just down to Plex pulling on the system, and I refused to off-load it. If this config won't function as a basic media server serving up 2-3 movie views per 24hrs, well... Again, the (temporary) fix for me every time is to stop the docker engine and restart. Works 100% of the time.

Next closest log entry prior to that was 36+hrs before

@JorgeB I don't have the diags, but do have the log. And also note: This "crash" must have happened earlier than I had noticed (according to the log). A trim action took place two hours later when things really hit the fan. Doubt any relevance, but still:

@JorgeB I just so happened to be sitting in front of mine when it locked earlier today. One thing I did notice - aside from a nearly identical log - was the fact that my mover was running at the time. I can't point to it in the log anywhere, but at least the GUI said it was moving at the time of the crash. Not sure if related, but wow is this frustrating. Killing the dockers always brings it back to life, so I'm no longer thrashing my drives with each hard boot, but having things down until I find its hung is a little more than mildly inconvenient. I'm rebooting to clear a few logs, and while I'm doing it I'm updating to 6.11.1. I feel like I've given 6.11.0 almost 3 weeks to shake out and it hasn't gotten any better with any changes introduced, so no further harm in upgrading in the off-chance something there solves the problem.

After my 10 day cycle, then crash, this happens 5 days later. I'm going to move my torrents to a seedbox and shut down qbittorrent to see what type of impact that may or may not have. There was NOTHING else going on at 1:36am aside from torrents - no backups, no plex, no nzbget, nothing - besides qbittorrent. And the load was no different than throughout the last 5 days. Oct 28 01:36:14 Unraid kernel: BUG: kernel NULL pointer dereference, address: 00000000000000b6 Oct 28 01:36:14 Unraid kernel: #PF: supervisor read access in kernel mode Oct 28 01:36:14 Unraid kernel: #PF: error_code(0x0000) - not-present page Oct 28 01:36:14 Unraid kernel: PGD 0 P4D 0 Oct 28 01:36:14 Unraid kernel: Oops: 0000 [#2] PREEMPT SMP NOPTI Oct 28 01:36:14 Unraid kernel: CPU: 5 PID: 24951 Comm: Disk Tainted: P D O 5.19.9-Unraid #1 Oct 28 01:36:14 Unraid kernel: Hardware name: System manufacturer System Product Name/TUF GAMING X570-PLUS (WI-FI), BIOS 3405 02/01/2021 Oct 28 01:36:14 Unraid kernel: RIP: 0010:folio_try_get_rcu+0x0/0x21 Oct 28 01:36:14 Unraid kernel: Code: e8 84 5e 63 00 48 8b 84 24 80 00 00 00 65 48 2b 04 25 28 00 00 00 74 05 e8 15 95 64 00 48 81 c4 88 00 00 00 5b e9 43 99 86 00 <8b> 57 34 85 d2 74 10 8d 4a 01 89 d0 f0 0f b1 4f 34 74 04 89 c2 eb Oct 28 01:36:14 Unraid kernel: RSP: 0000:ffffc90000887cc0 EFLAGS: 00010246 Oct 28 01:36:14 Unraid kernel: RAX: 0000000000000082 RBX: 0000000000000082 RCX: 0000000000000082 Oct 28 01:36:14 Unraid kernel: RDX: 0000000000000001 RSI: ffff88870b351b60 RDI: 0000000000000082 Oct 28 01:36:14 Unraid kernel: RBP: 0000000000000000 R08: 0000000000000028 R09: ffffc90000887cd0 Oct 28 01:36:14 Unraid kernel: R10: ffffc90000887cd0 R11: ffffc90000887d48 R12: 0000000000000000 Oct 28 01:36:14 Unraid kernel: R13: ffff88842cefe038 R14: 000000000000e029 R15: ffff88842cefe040 Oct 28 01:36:14 Unraid kernel: FS: 000014db233f96c0(0000) GS:ffff888fee940000(0000) knlGS:0000000000000000 Oct 28 01:36:14 Unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 28 01:36:14 Unraid kernel: CR2: 00000000000000b6 CR3: 0000000a06ee8000 CR4: 0000000000350ee0 Oct 28 01:36:14 Unraid kernel: Call Trace: Oct 28 01:36:14 Unraid kernel: <TASK> Oct 28 01:36:14 Unraid kernel: __filemap_get_folio+0x98/0x1ff Oct 28 01:36:14 Unraid kernel: ? _raw_spin_unlock+0x14/0x29 Oct 28 01:36:14 Unraid kernel: filemap_fault+0x6e/0x524 Oct 28 01:36:14 Unraid kernel: __do_fault+0x30/0x6e Oct 28 01:36:14 Unraid kernel: __handle_mm_fault+0x9a5/0xc7d Oct 28 01:36:14 Unraid kernel: handle_mm_fault+0x113/0x1d7 Oct 28 01:36:14 Unraid kernel: do_user_addr_fault+0x36a/0x514 Oct 28 01:36:14 Unraid kernel: exc_page_fault+0xfc/0x11e Oct 28 01:36:14 Unraid kernel: asm_exc_page_fault+0x22/0x30 Oct 28 01:36:14 Unraid kernel: RIP: 0033:0x14db33d246c1

WELP. After 10 days and 4 hours of uptime, I encountered this issue yet again. @JesterEE Thank you tons for the docker command - I've tried several incarnations previously, but unable to get a successful response. This allowed me to basically restart the dockers and be on my merry way. Best I can tell going back through the posts, it appears everyone experiencing the issue is running some type of torrent client - mostly deluge or qbittorrent. Is this somehow docker engine/IO related? There are several here not running their torrents off of UD, so that doesn't appear as obvious, either. Over the 10 days I stayed running, I transferred a bit over 10tb of torrents. I don't think that's unsubstantial - so maybe not load related at all? Latest diags attached for funsies. unraid-diagnostics-20221023-0827.zip

I'm 15mins away from having stayed up 10 consecutive days. I'm nearly ready to consider my issue solved. These are the plugins I removed: ca.backup2.plg - 2022.07.23 (Up to date) dynamix.active.streams.plg - 2020.06.17 (Up to date) file.activity.plg - 2022.08.19 (Up to date) Give the current stability, I'm not willing to re-install them for further troubleshooting, as there is nothing there I can't live without. I still have UD installed and it hosts my primary seed torrents with a little over 12tb total running 24x7. At least in my situation it does not appear unassigned devices-related. It's like one of my wife's unsolved murder podcasts - I know at the end I probably won't know exactly who caused it, but it doesn't keep me from wanting to know...

I hate that a pattern isn't necessarily emerging here - programmatically it doesn't make a whole lot of sense to me. Unassigned Devices was really one of the only plugins I DIDN'T remove, and I'm now sitting at an uptime of 6d3h from doing nothing else after having 14 consecutive crashes. I did have qbittorrent stop seeding torrents a few times, but the container didn't lock and the system stayed responsive and I was able to simply restart the container via the Docker interface and it's been running along just fine since. There are new versions of Fix Common and the Community Apps plugin available. I'm hesitant to update just because they would complicate/nullify the testing of this issue... I checked the logs for both, and nothing mention any changes that would seemingly impact this situation either.

Thanks for the report @ShadyDeth. I did NOT remove unassigned devices only because I consider it fairly critical. My error is identical in almost every way to those posted in this thread - and yet here I am on day 5 of no crashes after removing a few plugins I documented from a few posts up. I even let qbittorrent run full-bore last night thinking it may just be a load issue - but to my surprise, it was still running this morning. Maybe I'm just getting lucky. Another few days should tell the tale...though I haven't been able to run consecutively for 5 days straight since coming off of 9.6.x... Here's to no crashes...

Just an update from me: It's been nearly 3 days without a failure for me. This is what I uninstalled from a plugin perspective: ca.backup2.plg - 2022.07.23 (Up to date) dynamix.active.streams.plg - 2020.06.17 (Up to date) file.activity.plg - 2022.08.19 (Up to date) Again, I do run binhex's qbittorrent client, largely writing to an unassigned drive. But also again, no failures since removing these plugins. I'm cautiously optimistic.

I hate jumping to conclusions, but I also experience this with binhex's version of qBittorrent when my systems locks up (though it didn't last night after the removal of some of the above plugins, so we'll see). Everything I have is using macvlan. I'm going to follow old troubleshooting lore and only change one thing at a time. I'll make this modification if/when the next system lockup occurs and keep you posted.

As a data point, my near-daily failures have the same versions as above, however my fix common problems plugin is still a bit out of date. And still having the problem. It just took a spell where it was being updated (what seemed like) a few times a day, and I was tired of clicking the button every time fix.common.problems.plg - 2022.09.26 (Update available: 2022.10.09) I'll disable the above plugins, with the exception of the unassigned devices. If anyone can tell me how to mount my drives seamlessly in Unraid without this plugin, I'll try that too. Just a linux command-level mount? Thanks for this follow-up post.

I went a few days without a crash, and thought I was out of the woods, and this this morning - BAM. Good news is, it looks like the flash syslog storage did indeed catch it. Bad news is, to me as an idiot, I don't see anything else more insightful than I already posted. I hope I'm wrong and you smart people spot something! For good measure, I've also added the latest diagnostics. Thanks for the follow-up, @JorgeB! syslog unraid-diagnostics-20221012-0812.zip