• 6.6.0RC4 - nginx upstream timed errors

    • Closed Annoyance

    I'm getting a lot of nginx upstream timed out errors - when it happens my webui and dockers become inaccessible until it resolves itself


    Sep 19 13:32:24 Highlander nginx: 2018/09/19 13:32:24 [error] 383#383: *740117 upstream timed out (110: Connection timed out) while reading upstream, client:, server: , request: "GET /Dashboard HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "1d087a25aac48109ee9a15217a105d14c06e02a6.unraid.net", referrer: "https://1d087a25aac48109ee9a15217a105d14c06e02a6.unraid.net/Main"


    User Feedback

    Recommended Comments

    I've removed the ones I barely use e.g. Unbalance but the others are all well maintained except for maybe the rclone-beta plugin



    Link to comment

    I think the culprit might be the Rclone plugin - all was ok when I tried to find the culprit until I started my rclone upload job.


    @Waseh I'm not sure if there's anything you can or need to do to fix.  I have a rclone vfs mount which seemed fine, but when I start my upload that seems to be the trigger for problems


    Will try some more testing over the weekend


    rclone move /mnt/user/rclone_upload/google_vfs/ gdrive_media_vfs: -vv --drive-chunk-size 512M --checkers 3 --fast-list --transfers 2 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle** --exclude *.backup~* --exclude *.partial~* --bwlimit 9500k --tpslimit 4


    Link to comment

    Considering this only happens when you actually run a rclone command i don't think it's related to the plugin but perhaps more likely the rclone command itself.

    If somehow the wrapper is causing the problem you would be able to solve it by invoking:

    rcloneorig --config /boot/config/plugins/rclone-beta/.rclone.conf move /mnt/user/rclone_upload/google_vfs/ gdrive_media_vfs: -vv --drive-chunk-size 512M --checkers 3 --fast-list --transfers 2 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle** --exclude *.backup~* --exclude *.partial~* --bwlimit 9500k --tpslimit 4

    This bypasses anything the wrapper is doing to the command. I would be surprised if this was the culprit though.

    Edited by Waseh
    Link to comment

    I've been running for 10 hours on the 6.6.0 release and I've only seen the problem twice in my logs, so I don't think (i) rclone is the source and (ii) it's not a biggie now whatever it is


    Sep 21 01:08:00 Highlander nginx: 2018/09/21 01:08:00 [error] 8699#8699: *6530 upstream timed out (110: Connection timed out) while reading upstream, client:, server: , request: "GET /update.htm?cmd=/webGui/scripts/share_size&arg1=appdata&arg2=ssz1&csrf_token=50C36541BB26A4FA HTTP/2.0", upstream: "http://unix:/var/run/emhttpd.socket:/update.htm?cmd=/webGui/scripts/share_size&arg1=appdata&arg2=ssz1&csrf_token=50C36541BB26A4FA", host: "1d087a25aac48109ee9a15217a105d14c06e02a6.unraid.net", referrer: "https://1d087a25aac48109ee9a15217a105d14c06e02a6.unraid.net/Shares"
    Sep 21 09:02:51 Highlander nginx: 2018/09/21 09:02:51 [error] 8699#8699: *186147 upstream timed out (110: Connection timed out) while reading response header from upstream, client:, server: , request: "POST /plugins/dynamix.docker.manager/include/Events.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "1d087a25aac48109ee9a15217a105d14c06e02a6.unraid.net", referrer: "https://1d087a25aac48109ee9a15217a105d14c06e02a6.unraid.net/Dashboard"



    @Waseh I can't find anything on the rcloneorig command - what's the difference between it and 'rclone'?  Is it best to use rcloneorig?  Thanks

    Link to comment

    'rcloneorig' is in fact the true rclone binary. When invoking 'rclone' (when using my plugin) you actually invoke a wrapper script that calls rcloneorig and injects the config path just as the example i posted above.
    This is done so the config can be kept on the flash drive as to keep it persistent across reboots, but to keep the command consistent with what it would be on other systems. :) 

    Edited by Waseh
    • Like 1
    Link to comment

    ahh - thanks.  I'm going to start running my commands that way - rclone usage is huge for me as I've uploaded about 80% of my library, so anything that gets closer to 'raw' performance seems a good thing

    Link to comment

    Oh it won't affect performance at all - as soon as the command is entered the original binary handles everything so you won't gain anything by running it "directly" . If you check your running processes you'll already see rcloneorig and not rclone even though you invoked rclone. :)

    Edited by Waseh
    Link to comment

    Ahh ok.


    rclone is such an amazing project - I can't help thinking how much money Nick would make if he went commercial, although he's a successful businessman already I guess.

    • Upvote 1
    Link to comment

    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.

  • Status Definitions


    Open = Under consideration.


    Solved = The issue has been resolved.


    Solved version = The issue has been resolved in the indicated release version.


    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.


    Retest = Please retest in latest release.

    Priority Definitions


    Minor = Something not working correctly.


    Urgent = Server crash, data loss, or other showstopper.


    Annoyance = Doesn't affect functionality but should be fixed.


    Other = Announcement or other non-issue.