weirdcrap

lol oh dang I totally forgot about that. That would probably do it, it was for my previous mobo. Testing now. EDIT: yeah that was it. You da man squid. It would have taken me days of frustration before I found that on my own haha.

Solution: Check your syslinux config and make sure its standard, I had changed mine for an old mobo and never removed a switch I had added. I recently upgraded my gaming PC to a Ryzen 5000 build so I moved my old hardware to VOID as a major upgrade. I have an Asus Maximus VII Hero and an Intel i7-4790k. Intel Ark indicates I have VT-x and VT-d support on the CPU and I can see options to enable them in the BIOS: However whenever I boot into UnRAID it reports that IOMMU is disabled? This is not a deal breaker for me as I wasn't planning on using VMs on here but I thought it would be nice to have the option at least. Is there another setting I'm missing here? My Googling suggests that those should be the only two options I need to turn on. I updated my BIOS to the pre-beta BIOS (I'm not keen on running a beta BIOS) and that didn't make a difference. void-diagnostics-20210318-1942.zip

Yeah I do run monthly parity checks, NODE gets it on the 1st and VOID on the 15th. NODE completed its parity check on 3/1 with no errors and this disk reported no issues. Then a week later I remove it from NODE drive it 4 hours back to my house to put it in VOID and it fails in 30 minutes. I'm not sure how else I could have caught this earlier except with monthly SMART testing of all disks.

Alright thanks. Lesson learned, don't trust a used disk just because it didn't report any problems. I also need to be better about regularly running SMART tests on my array disks so I can hopefully catch this stuff before its a catastrophic failure and my array is no longer protected.

@JorgeBIt immediately fails both tests. SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 31759 45736176 # 2 Short offline Completed: read failure 10% 31759 45739672 That's why I was surprised. NODE hadn't recorded any pending or reallocated sectors for this disk. Then I brought it home (carefully packed in a HDD box with padding) and it just immediately fails right off the bat. Thankfully it looks like B&H Photo has the 6TB red plus drives in stock. Final question: Since I put that 6TB disk in and started the rebuild, I have to replace it with a 6TB or bigger correct?

Did you see my edit about the 31 reallocated sectors? Those weren't there when I took the disk out of NODE. Do you still think its a power/cable issue? These are all hot swap bays where I don't have to mess with cabling and the old disk had no issues in this bay. Is it safe to cancel the rebuild? It will just leave disk 6 emulated?

I recently put a new 8TB drive in NODE and was going to use the existing 6TB drive I replaced in VOID. I stupidly did not preclear the drive as I have done this many times before without issue, but this time, barely 2% into the disk rebuild on VOID, the disk threw 1500 + read errors, write errors, then disabled the disk. What is my best course of action here? I can't put the old 2TB disk I replaced it with back into the server since I already started the rebuild right? My parity is in a paused "Read Check" state. Should I cancel it? Because UnRAID disabled the disk I can't run SMART tests from the GUI nor can i seem to call the disk via the command line to try and test it using smartctl. I of course don't have any spare drives on hand. void-diagnostics-20210307-0621.zip EDIT: I can't find it in the terminal because UnRAID seems to have dropped the disk and changed its drive letter? The log shows it changed to sdw after it shit the bed. EDIT2: Holy shit that went south quickly. I took it out of NODE and there were no reallocated sectors. Now its got 31 reallocated sectors in a matter of minutes SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 224 196 021 Pre-fail Always - 7775 4 Start_Stop_Count 0x0032 098 098 000 Old_age Always - 2967 5 Reallocated_Sector_Ct 0x0033 199 199 140 Pre-fail Always - 31 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 057 057 000 Old_age Always - 31759 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 40 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 19 193 Load_Cycle_Count 0x0032 190 190 000 Old_age Always - 30273 194 Temperature_Celsius 0x0022 124 094 000 Old_age Always - 28 196 Reallocated_Event_Count 0x0032 169 169 000 Old_age Always - 31 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0 So I guess this disk was just a ticking time bomb from the get go?

The standard go file contains only: #!/bin/bash # Start the Management Utility /usr/local/sbin/emhttp & So remove "-p 8088" You can change the port in the webui under Settings > Management Access

Final update. I regret to report that I am now 100% positive this entire issue is caused by WireGuard itself as @claunia mentioned and no amount of setting changes is going to fix it. I re-enabled direct SSH access to NODE through the firewall and restricted it to my current comcast IP. When using the exact same setup and going entirely around the WG tunnel I get my full consistent speeds without any random drops or issues. I transferred roughly 300GB of data over the course of about 12 hours yesterday and NOT ONCE did the speed drop to an unacceptable level (only minor variances of +/-0.5MBps). This is the performance I have been seeking from the WireGuard tunnel and I am still no closer to figuring out why this simply doesn't work consistently. @ljm42 I assume WireGuard was not built with CONFIG_DYNAMIC_DEBUG enabled in the Kernel? I would really like to see this issue solved. However with nothing but anecdotal evidence and my personal testing I don't feel right just showing up on the WireGuard mailing list (or wherever the devs hang out) and saying there's a problem without some actionable proof. That and I have no flipping clue what about WireGuard causes this performance issue. EDIT: I guess I'll join #Wireguard on Freenode and see if any of the devs have a clue on what, if anything else, I can try to change to fix this or get some actionable proof and a fix made.

Some promising results with an MTU of 1380. I was able to complete a 60GB transfer from start to finish with zero drop in speed. I've started a much larger 250GB transfer, we shall see if I can maintain speed through the entire thing. EDIT: ANNNNDDDDD just like that there it goes. This is just ridiculous. EDIT2: I've re-opened SSH up to the world, restricted to my current comcast IP only. I'm running my 250GB transfer completely outside of the WireGuard tunnel and so far so good, but I've said that dozens of times up to this point so I won't hold my breath. For whatever reason, despite making the appropriate changes in sshd_config and rebooting for good measure, I cannot get UnRAID to completely disable password auth.

With the GPU changes (https://wiki.unraid.net/Unraid_OS_6.9.0#GPU_Driver_Integration) the old method of enabling Intel QuickSync via the go file is no longer recommended then? # Enable Intel QuickSync HW Transcoding modprobe i915 chmod -R 777 /dev/dri

It's that time of the month again! Just upgraded both servers to 6.9.0 stable. I haven't done my test with iperf3 yet that I mentioned above, I figured I'd try to adjust my MTU first and see what happens. I have lowered MTU to 1400 on both ends of the tunnel and have started a data transfer. EDIT: 1400 didn't work, trying 1380 as recommended in that link I posted. I don't know what else to try.

I imagine not as those files are unpacked fresh from the flash each boot. you would need to add it to your GO file.

For now those of you who want the full syslog in the GUI can run: sed -i 's/1000/3000/' /usr/local/emhttp/plugins/dynamix/include/Syslog.php in the terminal to fix it (tested and working for me). Thanks to @SimonF for the command.

Not that I'm aware of at this time. I assume it would be rather trivial to increase the buffer size but I'm not sure where in UnRAID this is handled without digging through code. If you follow that bug report @John_Mopened I bet LimeTech will come up with a workaround pretty quick for those who want it.

I believe this is due to the size of the syslog buffer provided to the WebUI and it was mentioned in one of the beta or RC threads. The full syslog should still be available on the filesystem itself. EDIT: Bug report already opened for it:

I don't use this docker but hydra stores all its config in its data/nzbhydra.yml file afaik. I imagine your path would be something like /appdata/nzbhydra2-binhex/data/nzbhydra.yml

I'm not sure why you can't seem to get a good speed only when downloading from his remote server. It could be an intermediate issue somewhere along your route? Do you see poor response times from any particular hop when you run a traceoute? I've seen reports of WireGuard having poor performance when your MTU is to high and causes packet fragmentation, that could be another possible avenue of investigation. I just made a post about it over in my thread as that is where my investigation is at right now.

Post your diagnostics (Tools > Diagnostics). It would be impossible for anyone to speculate without it.

I've been playing with iPerf3 to see if I can saturate my wireguard tunnel with other programs/protocols. My hope is to determine if this is in fact an SSH/RSYNC issue or if this random speed issue is WireGuard itself. With NODE running "iperf3 -s" and VOID acting as the client with "iperf3 -c 10.253.1.1 -t 30 -P 8" I can only seem to get about 20-25Mbps.... Cranking parallels up to 30 made no difference, I still maxed out at a meager 25Mbps. Is this indicative of a bigger problem or am I just not using iperf correctly? This is my first time messing with it so I'm inclined to think I'm doing something wrong with my test. EDIT: Could my speed woes be an MTU issue? https://www.reddit.com/r/WireGuard/comments/jn7d7e/slow_file_transfer/ https://keremerkan.net/posts/wireguard-mtu-fixes/ https://superuser.com/questions/1537638/wireguard-tunnel-slow-and-intermittent running traceroute --mtu <endpoint> on both ends of the server to server tunnel show my first hop MTU of 1500 so I assume that means wireguards default of 1420 should be fine for me right? I may try to tune it lower like the second link suggests just to test during my monthly data transfer to see if it has any real affect on the issue or not. EDIT2: Well my iperf3 issues appear to be my home internet's limited upload. In my described setup above VOID as the client was uploading data to the server rather than downloading data from it. Using --reverse fixes it and I seem to be able to fully maximize my bandwidth both directions. I'll have to pick a time this evening to run an iperf3 test where I can saturate the server bandwidth for like 45 minutes and see if it is able to maintain it the entire time.

On UnRAID SSH is enabled by default. QNAP probably requires SSH to be enabled in the GUI somewhere. EDIT: https://wiki.qnap.com/wiki/How_to_SSH_into_your_QNAP_device I'm not aware of a QNAP and UnRAID specific guide, but there are hundreds of guides online about setting up server to server sync with SSH and RSYNC: https://www.digitalocean.com/community/tutorials/how-to-use-rsync-to-sync-local-and-remote-directories

I can't speak to the SMB shares not detecting space correctly, though your speed issues sounds kind of similar to mine. I've tried copying files between my two servers using SSH & rsync, NFS shares, SSHFS, and they all suffer from severe performance loss somewhere mid transfer with no warning: I have been unable to definitively nail down where the problem is, though the more I work at it the more it seems like something to do with WireGuard rather than my chosen transfer protocol, my servers, or my network/internet. Sorry I don't have more help to offer, but I just wanted to chime in that you aren't alone in your performance issues with moving data over WireGuard.

Bump, still having the same issues with this. Transferred about 30GB before the transfer speed goes to shit. Finishing the transfer in Windows because I don't know what else to do about this issue.

Good morning itimpi, this may be related to the issue ScottinOkla is describing? I had a parity check running all day yesterday (manually started), went to bed, woke up and it was paused for no reason. I do have the parity check tuner plugin installed, and my only pause scenario is disks overheating. I see no indication in the logs that any disks were overheating (additionally my house has quite a cold ambient temperature). I did have "Send notification for temperature related pause" set to NO, but I assume it would have still logged the pause and reason in the logs? It stopped right at 3:30AM CST with no corresponding log entries around it as to why... Feb 12 13:44:24 VOID webGUI: Successful login user root from 192.168.1.8 Feb 12 13:45:00 VOID root: Fix Common Problems Version 2021.01.27 Feb 12 13:47:23 VOID kernel: mdcmd (38): check nocorrect Feb 12 13:47:23 VOID kernel: md: recovery thread: check P ... Feb 12 13:47:48 VOID kernel: mdcmd (39): set md_write_method 1 Feb 12 13:47:48 VOID kernel: Feb 12 13:49:04 VOID root: Stopping Auto Turbo Feb 12 13:49:04 VOID root: Setting write method to unRaid defined Feb 12 13:49:04 VOID kernel: mdcmd (40): set md_write_method auto Feb 12 17:04:35 VOID dhcpcd[1683]: br0: failed to renew DHCP, rebinding Feb 12 18:19:34 VOID emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog Feb 12 23:00:01 VOID Plugin Auto Update: Checking for available plugin updates Feb 12 23:00:06 VOID Plugin Auto Update: Checking for language updates Feb 12 23:00:06 VOID Plugin Auto Update: Community Applications Plugin Auto Update finished Feb 13 00:00:07 VOID crond[1849]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null Feb 13 00:10:37 VOID emhttpd: spinning down /dev/sdh Feb 13 00:10:37 VOID emhttpd: spinning down /dev/sdg Feb 13 00:10:37 VOID emhttpd: spinning down /dev/sdb Feb 13 00:10:37 VOID emhttpd: spinning down /dev/sdf Feb 13 00:10:37 VOID emhttpd: spinning down /dev/sdq Feb 13 00:10:37 VOID emhttpd: spinning down /dev/sdo Feb 13 02:00:01 VOID Docker Auto Update: Community Applications Docker Autoupdate running Feb 13 02:00:01 VOID Docker Auto Update: Checking for available updates Feb 13 02:00:07 VOID Docker Auto Update: No updates will be installed Feb 13 03:30:06 VOID kernel: mdcmd (49): nocheck PAUSE Feb 13 03:30:06 VOID kernel: Feb 13 03:30:06 VOID kernel: md: recovery thread: exit status: -4 Feb 13 05:02:21 VOID emhttpd: read SMART /dev/sdh Feb 13 05:02:29 VOID emhttpd: read SMART /dev/sdg Feb 13 05:02:35 VOID emhttpd: read SMART /dev/sdb Feb 13 05:02:42 VOID emhttpd: read SMART /dev/sdo Feb 13 05:02:49 VOID emhttpd: read SMART /dev/sdf Feb 13 05:02:54 VOID emhttpd: read SMART /dev/sdq void-diagnostics-20210213-0521.zip

Interesting, you may have more than one plugin to blame here then. As Ken-ji mentioned in that other thread, you will probably need to go through the plugins one by one to figure out which one is changing your ownership and ask the author to fix it.