weirdcrap

Members
  • Posts

    446
  • Joined

  • Last visited

Everything posted by weirdcrap

  1. Final update. I regret to report that I am now 100% positive this entire issue is caused by WireGuard itself as @claunia mentioned and no amount of setting changes is going to fix it. I re-enabled direct SSH access to NODE through the firewall and restricted it to my current comcast IP. When using the exact same setup and going entirely around the WG tunnel I get my full consistent speeds without any random drops or issues. I transferred roughly 300GB of data over the course of about 12 hours yesterday and NOT ONCE did the speed drop to an unacceptable level (only minor variances of +/-0.5MBps). This is the performance I have been seeking from the WireGuard tunnel and I am still no closer to figuring out why this simply doesn't work consistently. @ljm42 I assume WireGuard was not built with CONFIG_DYNAMIC_DEBUG enabled in the Kernel? I would really like to see this issue solved. However with nothing but anecdotal evidence and my personal testing I don't feel right just showing up on the WireGuard mailing list (or wherever the devs hang out) and saying there's a problem without some actionable proof. That and I have no flipping clue what about WireGuard causes this performance issue. EDIT: I guess I'll join #Wireguard on Freenode and see if any of the devs have a clue on what, if anything else, I can try to change to fix this or get some actionable proof and a fix made.
  2. Some promising results with an MTU of 1380. I was able to complete a 60GB transfer from start to finish with zero drop in speed. I've started a much larger 250GB transfer, we shall see if I can maintain speed through the entire thing. EDIT: ANNNNDDDDD just like that there it goes. This is just ridiculous. EDIT2: I've re-opened SSH up to the world, restricted to my current comcast IP only. I'm running my 250GB transfer completely outside of the WireGuard tunnel and so far so good, but I've said that dozens of times up to this point so I won't hold my breath. For whatever reason, despite making the appropriate changes in sshd_config and rebooting for good measure, I cannot get UnRAID to completely disable password auth.
  3. With the GPU changes (https://wiki.unraid.net/Unraid_OS_6.9.0#GPU_Driver_Integration) the old method of enabling Intel QuickSync via the go file is no longer recommended then? # Enable Intel QuickSync HW Transcoding modprobe i915 chmod -R 777 /dev/dri
  4. It's that time of the month again! Just upgraded both servers to 6.9.0 stable. I haven't done my test with iperf3 yet that I mentioned above, I figured I'd try to adjust my MTU first and see what happens. I have lowered MTU to 1400 on both ends of the tunnel and have started a data transfer. EDIT: 1400 didn't work, trying 1380 as recommended in that link I posted. I don't know what else to try.
  5. I imagine not as those files are unpacked fresh from the flash each boot. you would need to add it to your GO file.
  6. For now those of you who want the full syslog in the GUI can run: sed -i 's/1000/3000/' /usr/local/emhttp/plugins/dynamix/include/Syslog.php in the terminal to fix it (tested and working for me). Thanks to @SimonF for the command.
  7. Not that I'm aware of at this time. I assume it would be rather trivial to increase the buffer size but I'm not sure where in UnRAID this is handled without digging through code. If you follow that bug report @John_Mopened I bet LimeTech will come up with a workaround pretty quick for those who want it.
  8. I believe this is due to the size of the syslog buffer provided to the WebUI and it was mentioned in one of the beta or RC threads. The full syslog should still be available on the filesystem itself. EDIT: Bug report already opened for it:
  9. I don't use this docker but hydra stores all its config in its data/nzbhydra.yml file afaik. I imagine your path would be something like /appdata/nzbhydra2-binhex/data/nzbhydra.yml
  10. I'm not sure why you can't seem to get a good speed only when downloading from his remote server. It could be an intermediate issue somewhere along your route? Do you see poor response times from any particular hop when you run a traceoute? I've seen reports of WireGuard having poor performance when your MTU is to high and causes packet fragmentation, that could be another possible avenue of investigation. I just made a post about it over in my thread as that is where my investigation is at right now.
  11. Post your diagnostics (Tools > Diagnostics). It would be impossible for anyone to speculate without it.
  12. I've been playing with iPerf3 to see if I can saturate my wireguard tunnel with other programs/protocols. My hope is to determine if this is in fact an SSH/RSYNC issue or if this random speed issue is WireGuard itself. With NODE running "iperf3 -s" and VOID acting as the client with "iperf3 -c 10.253.1.1 -t 30 -P 8" I can only seem to get about 20-25Mbps.... Cranking parallels up to 30 made no difference, I still maxed out at a meager 25Mbps. Is this indicative of a bigger problem or am I just not using iperf correctly? This is my first time messing with it so I'm inclined to think I'm doing something wrong with my test. EDIT: Could my speed woes be an MTU issue? https://www.reddit.com/r/WireGuard/comments/jn7d7e/slow_file_transfer/ https://keremerkan.net/posts/wireguard-mtu-fixes/ https://superuser.com/questions/1537638/wireguard-tunnel-slow-and-intermittent running traceroute --mtu <endpoint> on both ends of the server to server tunnel show my first hop MTU of 1500 so I assume that means wireguards default of 1420 should be fine for me right? I may try to tune it lower like the second link suggests just to test during my monthly data transfer to see if it has any real affect on the issue or not. EDIT2: Well my iperf3 issues appear to be my home internet's limited upload. In my described setup above VOID as the client was uploading data to the server rather than downloading data from it. Using --reverse fixes it and I seem to be able to fully maximize my bandwidth both directions. I'll have to pick a time this evening to run an iperf3 test where I can saturate the server bandwidth for like 45 minutes and see if it is able to maintain it the entire time.
  13. On UnRAID SSH is enabled by default. QNAP probably requires SSH to be enabled in the GUI somewhere. EDIT: https://wiki.qnap.com/wiki/How_to_SSH_into_your_QNAP_device I'm not aware of a QNAP and UnRAID specific guide, but there are hundreds of guides online about setting up server to server sync with SSH and RSYNC: https://www.digitalocean.com/community/tutorials/how-to-use-rsync-to-sync-local-and-remote-directories
  14. I can't speak to the SMB shares not detecting space correctly, though your speed issues sounds kind of similar to mine. I've tried copying files between my two servers using SSH & rsync, NFS shares, SSHFS, and they all suffer from severe performance loss somewhere mid transfer with no warning: I have been unable to definitively nail down where the problem is, though the more I work at it the more it seems like something to do with WireGuard rather than my chosen transfer protocol, my servers, or my network/internet. Sorry I don't have more help to offer, but I just wanted to chime in that you aren't alone in your performance issues with moving data over WireGuard.
  15. Bump, still having the same issues with this. Transferred about 30GB before the transfer speed goes to shit. Finishing the transfer in Windows because I don't know what else to do about this issue.
  16. Good morning itimpi, this may be related to the issue ScottinOkla is describing? I had a parity check running all day yesterday (manually started), went to bed, woke up and it was paused for no reason. I do have the parity check tuner plugin installed, and my only pause scenario is disks overheating. I see no indication in the logs that any disks were overheating (additionally my house has quite a cold ambient temperature). I did have "Send notification for temperature related pause" set to NO, but I assume it would have still logged the pause and reason in the logs? It stopped right at 3:30AM CST with no corresponding log entries around it as to why... Feb 12 13:44:24 VOID webGUI: Successful login user root from 192.168.1.8 Feb 12 13:45:00 VOID root: Fix Common Problems Version 2021.01.27 Feb 12 13:47:23 VOID kernel: mdcmd (38): check nocorrect Feb 12 13:47:23 VOID kernel: md: recovery thread: check P ... Feb 12 13:47:48 VOID kernel: mdcmd (39): set md_write_method 1 Feb 12 13:47:48 VOID kernel: Feb 12 13:49:04 VOID root: Stopping Auto Turbo Feb 12 13:49:04 VOID root: Setting write method to unRaid defined Feb 12 13:49:04 VOID kernel: mdcmd (40): set md_write_method auto Feb 12 17:04:35 VOID dhcpcd[1683]: br0: failed to renew DHCP, rebinding Feb 12 18:19:34 VOID emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog Feb 12 23:00:01 VOID Plugin Auto Update: Checking for available plugin updates Feb 12 23:00:06 VOID Plugin Auto Update: Checking for language updates Feb 12 23:00:06 VOID Plugin Auto Update: Community Applications Plugin Auto Update finished Feb 13 00:00:07 VOID crond[1849]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null Feb 13 00:10:37 VOID emhttpd: spinning down /dev/sdh Feb 13 00:10:37 VOID emhttpd: spinning down /dev/sdg Feb 13 00:10:37 VOID emhttpd: spinning down /dev/sdb Feb 13 00:10:37 VOID emhttpd: spinning down /dev/sdf Feb 13 00:10:37 VOID emhttpd: spinning down /dev/sdq Feb 13 00:10:37 VOID emhttpd: spinning down /dev/sdo Feb 13 02:00:01 VOID Docker Auto Update: Community Applications Docker Autoupdate running Feb 13 02:00:01 VOID Docker Auto Update: Checking for available updates Feb 13 02:00:07 VOID Docker Auto Update: No updates will be installed Feb 13 03:30:06 VOID kernel: mdcmd (49): nocheck PAUSE Feb 13 03:30:06 VOID kernel: Feb 13 03:30:06 VOID kernel: md: recovery thread: exit status: -4 Feb 13 05:02:21 VOID emhttpd: read SMART /dev/sdh Feb 13 05:02:29 VOID emhttpd: read SMART /dev/sdg Feb 13 05:02:35 VOID emhttpd: read SMART /dev/sdb Feb 13 05:02:42 VOID emhttpd: read SMART /dev/sdo Feb 13 05:02:49 VOID emhttpd: read SMART /dev/sdf Feb 13 05:02:54 VOID emhttpd: read SMART /dev/sdq void-diagnostics-20210213-0521.zip
  17. Interesting, you may have more than one plugin to blame here then. As Ken-ji mentioned in that other thread, you will probably need to go through the plugins one by one to figure out which one is changing your ownership and ask the author to fix it.
  18. You shouldn't even need to remove the plugin. At least for me, the proper ownership of the / directory was restored by simply rebooting UnRAID.
  19. You were right @ken-ji My other server hadn't had the parity check tuning plugin updated yet so I installed. Sure enough I lost pub key access and my usr folder is owned by nobody:users now EDIT: So I went digging into the folders it seems to have touched and I ended up in the plugins folder and it looks like some of the other plugins ive installed over the years aren't owned by root either. Is it a rule of thumb that all of these files should be owned by root? Or is it different from plugin to plugin?
  20. Interesting. The reboot actually fixed it for me so UnRAID must be correctly applying the default permissions and overriding anything the plugins may be messing with at boot. I don't have any more plugin updates currently but I will keep my eye out for an update and see if my ssh breaks after applying it. EDIT: Yeah it was almost certainly the "usr" folder that caused this as I recall seeing that it was the only folder in / that wasn't owned by root. I ignored it as I didn't think that folder would have anything to do with my issue since my key and ssh files weren't stored there.
  21. Yeah I was able to get in with my password still but I could not find anything wrong with my / directory. Next time it happens ill be sure to get a screenshot and share it so I can get some more eyes on it. If it was a plugin though wouldn't it affect me from the moment I booted the system? This occurred after 6 days of uptime and multiple uses of the key up to that point. I haven't added or removed any plugins, I think I may have updated one, the parity check tuning plugin.
  22. I also just got hit with my first case of "Authentication refused: bad ownership or modes for directory /" I have SSH'd into this server every day this week and used my public key. Now, with zero changes on my end, UnRAID says my permissions are wrong. I have a second server with the same exact setup and the public key works fine.... I'm mid appdata backup, I'm going to restart after and see if this persists. I use Termius, but also have putty installed. Updating putty from 0.7.3 to 0.7.4 did nothing to fix the issue. EDIT: While I'm waiting for the appdata backup to finish I've been poking around and I can't find anything wrong with the ownership of any of the files or folders according to output of ls -al. I've even tried re-owning and chmodding the relevant folders as suggested above and on other sites, it makes no difference. Public key auth is just flat out broken for no discernable reason. EDIT: Yeah a reboot fixes it with no changes to my config. Hopefully whatever causes this gets fixed before stable, its alarming to suddenly not be able to connect to your server. Luckily I don't disallow password auth right now, though I would like to turn it off at some point for heightened security.
  23. Well I guess I'll stick with the MX500 then, even though just ignoring the issue gives me a deep seeded (seated?) feeling of wrongness lol. Maybe the SmartMonTools update you linked to will resolve this once and for all by adjusting the alerting behavior for this drive. A question about the 860 EVO drive on my LSI and it's lack of TRIM: If I don't ever fill my SSD up (it generally only hovers around 100-200GB used) then will my lack of TRIM support have much of any noticeable performance and/or longevity effects? I've been reading about the subject and people talk about the write amplification implications of not having TRIM or garbage collection for heavily utilized SSDs. However it sounds like if you have lots of blocks without needed data there shouldn't need to be a lot of shuffling around done by the controller firmware, right?
  24. @JorgeB So what are good 1TB SATA (Non M.2) SSDs that are known to work well with UnRAID without frustratingly weird firmware issues like my 860 EVO issue or apparently Crucial MX500's randomly reporting bogus bad sectors ? I just bought an MX500 to put into my remote server as a replacement for my other aging 850 EVO. But if it's going to randomly report bad sectors I don't really want anything to do with that and I would rather return it for something else. Intel seems to have all but abandoned consumer line SSDs bigger than 512GB. I don't particularly want to pay a premium for a "data center quality" SATA SSD: https://www.intel.com/content/www/us/en/products/memory-storage/solid-state-drives/data-center-ssds.html I've not had good experience with Kingstons and I don't really know much about the Western Digital line of SSDs. EDIT: So I found the main topic for these crucial drives: So disabling the monitoring for 197 just prevents email/push alerts but it will still track and report bad sectors that stick around in the webui? UnRAID only disables a disk if it fails a write test, so this bug shouldn't cause any sort of disabling issues, right?
  25. You are correct on all counts. I have no clue what to make of it, I've recreated the tunnel and peer numerous times... Could this be some sort of weird networking or router tech? My only thoughts would be to turn on allowed connection logging and comparing the "good" connections to the "bad" connections.