• 6.6.0RC4 - nginx upstream timed errors


    DZMM
    • Closed Annoyance

    I'm getting a lot of nginx upstream timed out errors - when it happens my webui and dockers become inaccessible until it resolves itself

     

    Sep 19 13:32:24 Highlander nginx: 2018/09/19 13:32:24 [error] 383#383: *740117 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.30.10, server: , request: "GET /Dashboard HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "1d087a25aac48109ee9a15217a105d14c06e02a6.unraid.net", referrer: "https://1d087a25aac48109ee9a15217a105d14c06e02a6.unraid.net/Main"

    highlander-diagnostics-20180919-1343.zip




    User Feedback

    Recommended Comments

    I've removed the ones I barely use e.g. Unbalance but the others are all well maintained except for maybe the rclone-beta plugin

     

    1038189924_FireShotCapture30-Highlander_Plugins_-https___1d087a25aac48109ee9a15217a.thumb.png.161834275d0b1390c4eec3522ea4c758.png

    Link to comment

    I think the culprit might be the Rclone plugin - all was ok when I tried to find the culprit until I started my rclone upload job.

     

    @Waseh I'm not sure if there's anything you can or need to do to fix.  I have a rclone vfs mount which seemed fine, but when I start my upload that seems to be the trigger for problems

     

    Will try some more testing over the weekend

     

    rclone move /mnt/user/rclone_upload/google_vfs/ gdrive_media_vfs: -vv --drive-chunk-size 512M --checkers 3 --fast-list --transfers 2 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle** --exclude *.backup~* --exclude *.partial~* --bwlimit 9500k --tpslimit 4

     

    Link to comment

    @DZMM
    Considering this only happens when you actually run a rclone command i don't think it's related to the plugin but perhaps more likely the rclone command itself.

    If somehow the wrapper is causing the problem you would be able to solve it by invoking:

    rcloneorig --config /boot/config/plugins/rclone-beta/.rclone.conf move /mnt/user/rclone_upload/google_vfs/ gdrive_media_vfs: -vv --drive-chunk-size 512M --checkers 3 --fast-list --transfers 2 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle** --exclude *.backup~* --exclude *.partial~* --bwlimit 9500k --tpslimit 4

    This bypasses anything the wrapper is doing to the command. I would be surprised if this was the culprit though.

    Edited by Waseh
    Link to comment

    I've been running for 10 hours on the 6.6.0 release and I've only seen the problem twice in my logs, so I don't think (i) rclone is the source and (ii) it's not a biggie now whatever it is

     

    Sep 21 01:08:00 Highlander nginx: 2018/09/21 01:08:00 [error] 8699#8699: *6530 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.30.10, server: , request: "GET /update.htm?cmd=/webGui/scripts/share_size&arg1=appdata&arg2=ssz1&csrf_token=50C36541BB26A4FA HTTP/2.0", upstream: "http://unix:/var/run/emhttpd.socket:/update.htm?cmd=/webGui/scripts/share_size&arg1=appdata&arg2=ssz1&csrf_token=50C36541BB26A4FA", host: "1d087a25aac48109ee9a15217a105d14c06e02a6.unraid.net", referrer: "https://1d087a25aac48109ee9a15217a105d14c06e02a6.unraid.net/Shares"
    Sep 21 09:02:51 Highlander nginx: 2018/09/21 09:02:51 [error] 8699#8699: *186147 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.30.10, server: , request: "POST /plugins/dynamix.docker.manager/include/Events.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "1d087a25aac48109ee9a15217a105d14c06e02a6.unraid.net", referrer: "https://1d087a25aac48109ee9a15217a105d14c06e02a6.unraid.net/Dashboard"

    highlander-diagnostics-20180921-1007.zip

     

    @Waseh I can't find anything on the rcloneorig command - what's the difference between it and 'rclone'?  Is it best to use rcloneorig?  Thanks

    Link to comment

    @DZMM
    'rcloneorig' is in fact the true rclone binary. When invoking 'rclone' (when using my plugin) you actually invoke a wrapper script that calls rcloneorig and injects the config path just as the example i posted above.
    This is done so the config can be kept on the flash drive as to keep it persistent across reboots, but to keep the command consistent with what it would be on other systems. :) 

    Edited by Waseh
    • Like 1
    Link to comment

    ahh - thanks.  I'm going to start running my commands that way - rclone usage is huge for me as I've uploaded about 80% of my library, so anything that gets closer to 'raw' performance seems a good thing

    Link to comment

    Oh it won't affect performance at all - as soon as the command is entered the original binary handles everything so you won't gain anything by running it "directly" . If you check your running processes you'll already see rcloneorig and not rclone even though you invoked rclone. :)

    Edited by Waseh
    Link to comment

    Ahh ok.

     

    rclone is such an amazing project - I can't help thinking how much money Nick would make if he went commercial, although he's a successful businessman already I guess.

    • Upvote 1
    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.