weirdcrap

I am, I for some reason thought it was written by one of the unraid admins, I didn't realize it was a user guest blog. Either way I didn't mean to come off like I expected him to come help me, I just meant in comparing my self made setup to theirs I did everything I could "by the book" so to speak. It varies, sometimes I can get through 10-20 files at 1-2GB a pop. Other times the speed will tank half way through the first file. I did test a cache to cache transfer in my flurry of work yesterday. It did not improve the situation from what I recall but I did not document the results well so it bears another test to be sure. I do not unfortunately and the primary server is actually hosted about 4 hours away from me so its not something I can just go and visit on a whim. When I first setup WireGuard I definitely tested this and both ends were able to start the connection so I'm not sure what changed there. My router here with VOID is a pfsense router and from what I can tell my port forwards are setup correctly: I'll look those threads over again as I'll admit my grasp of wireguard when i configured it was tentative at best and its all looking greek to me now looking back at it. I think I see my problem though, on NODE I have the peer address set to the same thing as my local endpoint. The peer for VOID should be my dynamic DNS name for my home IP right? EDIT: Started a cache (NODE) to cache (VOID) transfer and barely made it into the first file before the speed tanked. Meanwhile I've been uploading files from VOID to NODE for the last hour at full speed. 10GB files at 2.8MB/s (the max for my crappy comcast upload). This is the part that drives me nuts, I can't seem to find a pattern to what transfers utilize their full potential speed while others just languish in the land of dial-up.

I didn't think to try this until now, when I run multiple transfers in parallel, one slowing down does NOT cause the other to slow down. Why is that? If this was a resource issue (whether CPU, RAM, bandwidth, etc) I would think parallel transfers would be affected equally...

@ljm42 I'm noticing an odd behavior with my server to server wireguard tunnel... When I'm signed into NODE, I can't ping VOID initially. Once I start a ping from VOID to NODE however, replies start flowing both ways... This is not something I had noticed before. Should I enable keep alive to prevent this? I'm going to snag a video of this behavior. EDIT: So as you can see in the video, I start to ping VOID's WG IP on NODE and get no replies until I start a ping from VOID to NODE. Then like magic all of a sudden NODE realizes VOID is, in fact, available. I'm to the point in this I'm willing to offer a cash reward if someone can just tell me WTF is wrong with rsync/SSH.

Yup, that's the thread I found. I linked to LJM42's reply specifically as that appears to be what my issue was. On a side note, I see the tips and tweaks plugin recommends the performance scaler, does this offer much difference for day to day use over the OnDemand option?

It would be really nice if someone, anyone, could come in here and post about their experiences with the RC vs the stable. Is anyone else doing what I'm doing with SSH and RSYNC? Are you having the head smashingly frustrating performance problems I have? Who wrote LimeTech's guide on this? Are they having these kinds of problems? I'm absolutely shocked that in the three weeks I've been posting here about this not a single person has come forward to either confirm or deny that this is a legitimate issue vs something with my setup. I'm sorry for my ranting, I'm beyond frustrated and disappointed that I keep thinking I've found the answer, only to be proven wrong time and time again mere hours later. EDIT: OK so I've got NODE and VOID on RC2 again and NODE is finally behaving itself after discovering my CPU governor setting got changed. I'm not sure what sort of voodoo magic I summoned yesterday when upgrading NODE to RC2 but the performance gains I saw the entire day yesterday disappeared that night. RSYNC over SSH is back to 1MB/s > when using VOID to pull from NODE. Whether dockers are running or not makes no difference. Pushing from NODE to VOID continues to give me my full 10MB/s. I've placed new diagnostics files in the OP. Every single time I think I'm starting to figure out the problem, UnRAID throws me a god damn curveball. So now on 6.9.0-RC2 whether I run the script on NODE or VOID, the performance starts out good and then falls to 1MB/s > or less.

alright well I spoke to soon. I was having some issues with my plex docker running poorly on NODE so I rolled it back to 6.8.3 as a test. This appears to have fixed my plex issue but now VOID is only managing to pull about 1MB/s again (I was getting 10MB/s before). I can push from NODE to VOID at full speed as always... It's late and I'm tired of messing with this, if the speed issue continues I may try to upgrade back to 6.9.0-rc2 and see if that makes the problem go away again. Alternatively, could I downgrade back to 6.8.3 on VOID? I can't roll back using the update tool, can I just download 6.8.3 from the website and extract it onto my flash? EDIT: The answer is yes, just replace all the bz files. Downgrading VOID to 6.8.3 to see if it makes a difference. EDIT2: JFC, so I can't downgrade to 6.8.3 on VOID because of the new f*cking partition layout for BTRFS. So my choices are either blow away my cache again (I just upgraded it on Monday), or stay on 6.9.0-RC2 and never know if my problems are because of the beta or not.

Yeah it has been an extremely frustrating journey for me, but I'm glad I've finally cracked it. What led me to suspect resource utilization despite no clues that it was resource starved was that, with all other things equal, I could use rsync/ssh to push from NODE at my full speeds while trying to have VOID pull the files resulted in terrible speeds (no matter what protocol I used). So I assumed I had to have some sort of resource limitation where rsync/ssh were running from. It's been running since this morning and speeds are stable, something I could never accomplish over the last couple of months for more than an hour at a time. I've always had Plex monitoring the library for changes, so I'm not sure why it has suddenly become a big deal. The only major plex change that comes to my mind around that time was them adding intro detection to the server. ^This is wrong, the issue is back and as hard to pin down as ever.

Loop2 is for the docker image and BTRFS is its file system. Post your diagnostics (Tools > Diagnostics) please. If I recall correctly LimeTech made some underlying changes to address the excessive writes in the betas but users also had to change the partition alignment:

It's my monthly data transfer and the performance is still crap. I've played with every rsync and SSH option I can think of (--whole-file, --inplace). It doesn't matter if the data goes straight into the array or all gets written to a cache disk first. Copying outside of SSH (ie using Windows file explorer) saturates my bandwidth (full 10MB/s) as expected but SSH can barely manage to maintain 1MB/s EDIT: Copying via rsync/SSH from other servers I have access to do not suffer form performance issues. I'm able to max out my home internet bandwidth. EDIT2: I'm going to try setting up an SSHFS mount between the servers and see if it behaves any differently. I'm grasping at straws here so if anyone has a better idea I'm all ears. EDIT3: SSHFS seemed to work at first, but its speed seems to crater eventually as well. EDIT4: This is driving me up a wall. I have hundreds of gigabytes to transfer and at the horrendous dial-up level speeds I'm getting, I could WALK the files to the backup server faster than I could move them over the internet. EDIT5: I'm playing with NFS shares over WireGuard since I can't figure out what is wrong with my SSH speeds between these two servers. Results are promising though I'm going to have to rescript my entire backup process. EDIT6: I think I cracked it! I never considered stopping the few dockers on the server running the script as the stats in the webui and htop never indicated the CPU was anywhere near busy (it hovered around 10-15%). But sure enough, with everything stopped my speeds are back to normal and have stayed consistent. Specifically, it seems to be Plex's automatic library change scanning that was absolutely crippling my transfer rate. With dockers started and that turned off my speeds are holding steady so far. EDIT7: EDIT6 is wrong, see latest posts.

Really? That's interesting, the old one worked great for the last 5 years, not a single error. After the most recent two errors all has been quiet and I have a large data transfer going so I'll have to try this in a couple days. EDIT: and just like that I jinxed myself, tons more errors now. Fantastic. EDIT2: Ok, I shuffled stuff around and put the SSD on the asmedia controller I have the parity drive on and so far so good on boot up. I'll have to start copying data and see if it shows up still. EDIT3: Yeah moving it off the AMD controller solved it. Per usual JorgeB has the answers.

I purchased a 1TB Samsung SSD 860 EVO before Christmas and just kind of left it on the shelf till today. Well I installed it in my server to replace an old but reliable 500GB 860 EVO and that's where the trouble started. Upon copying data to the drive I started getting warnings about CRC errors and the SMART stats for it going up and variations of this continue to show up in the logs: Feb 1 11:22:43 VOID kernel: ata6.00: irq_stat 0x08000000, interface fatal error Feb 1 11:22:43 VOID kernel: ata6.00: failed command: WRITE FPDMA QUEUED Feb 1 11:22:43 VOID kernel: ata6.00: cmd 61/e8:40:f8:ce:f0/09:00:02:00:00/40 tag 8 ncq dma 1298432 ou Feb 1 11:22:43 VOID kernel: res 40/00:40:f8:ce:f0/00:00:02:00:00/40 Emask 0x10 (ATA bus error) Feb 1 11:22:43 VOID kernel: ata6.00: status: { DRDY } Feb 1 11:22:43 VOID kernel: ata6.00: failed command: WRITE FPDMA QUEUED Feb 1 11:22:43 VOID kernel: ata6.00: cmd 61/50:48:e0:d8:f0/09:00:02:00:00/40 tag 9 ncq dma 1220608 ou Feb 1 11:22:43 VOID kernel: res 40/00:40:f8:ce:f0/00:00:02:00:00/40 Emask 0x10 (ATA bus error) Feb 1 11:22:43 VOID kernel: ata6.00: status: { DRDY } Feb 1 11:22:43 VOID kernel: ata6.00: failed command: WRITE FPDMA QUEUED Feb 1 11:22:43 VOID kernel: ata6.00: cmd 61/d8:70:30:e2:f0/09:00:02:00:00/40 tag 14 ncq dma 1290240 ou Feb 1 11:22:43 VOID kernel: res 40/00:40:f8:ce:f0/00:00:02:00:00/40 Emask 0x10 (ATA bus error) Feb 1 11:22:43 VOID kernel: ata6.00: status: { DRDY } Feb 1 11:22:43 VOID kernel: ata6.00: failed command: WRITE FPDMA QUEUED Feb 1 11:22:43 VOID kernel: ata6.00: cmd 61/90:78:08:ec:f0/09:00:02:00:00/40 tag 15 ncq dma 1253376 ou Feb 1 11:22:43 VOID kernel: res 40/00:40:f8:ce:f0/00:00:02:00:00/40 Emask 0x10 (ATA bus error) Feb 1 11:22:43 VOID kernel: ata6.00: status: { DRDY } Feb 1 11:22:43 VOID kernel: ata6.00: failed command: WRITE FPDMA QUEUED Feb 1 11:22:43 VOID kernel: ata6.00: cmd 61/f8:80:98:f5:f0/09:00:02:00:00/40 tag 16 ncq dma 1306624 ou Feb 1 11:22:43 VOID kernel: res 40/00:40:f8:ce:f0/00:00:02:00:00/40 Emask 0x10 (ATA bus error) Feb 1 11:22:43 VOID kernel: ata6.00: status: { DRDY } Feb 1 11:22:43 VOID kernel: ata6.00: failed command: WRITE FPDMA QUEUED Feb 1 11:22:43 VOID kernel: ata6.00: cmd 61/e0:a8:90:ff:f0/09:00:02:00:00/40 tag 21 ncq dma 1294336 ou Feb 1 11:22:43 VOID kernel: res 40/00:40:f8:ce:f0/00:00:02:00:00/40 Emask 0x10 (ATA bus error) Feb 1 11:22:43 VOID kernel: ata6.00: status: { DRDY } So I shut it down and swapped out: 3 different SATA cables, 2 molex to sata power adaters, and the port on the board with another working port on the motherboard. When I changed the port on the mobo the error changed somewhat, and appeared when i first booted before copying any data: Feb 1 18:13:05 VOID kernel: ata1.00: READ LOG DMA EXT failed, trying PIO Feb 1 18:13:05 VOID kernel: ata1.00: exception Emask 0x0 SAct 0xffffffff SErr 0x0 action 0x6 Feb 1 18:13:05 VOID kernel: ata1.00: irq_stat 0x40000008 Feb 1 18:13:05 VOID kernel: ata1.00: failed command: READ FPDMA QUEUED Feb 1 18:13:05 VOID kernel: ata1.00: cmd 60/20:20:28:1a:25/00:00:0e:00:00/40 tag 4 ncq dma 16384 in Feb 1 18:13:05 VOID kernel: res 41/84:20:28:1a:25/00:00:0e:00:00/00 Emask 0x410 (ATA bus error) <F> Feb 1 18:13:05 VOID kernel: ata1.00: status: { DRDY ERR } Feb 1 18:13:05 VOID kernel: ata1.00: error: { ICRC ABRT } Feb 1 18:13:05 VOID kernel: ata1: hard resetting link Feb 1 18:13:05 VOID kernel: ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Feb 1 18:13:05 VOID kernel: ata1.00: supports DRM functions and may not be fully accessible Feb 1 18:13:05 VOID kernel: ata1.00: supports DRM functions and may not be fully accessible Feb 1 18:13:05 VOID kernel: ata1.00: configured for UDMA/133 Feb 1 18:13:05 VOID kernel: sd 1:0:0:0: [sdb] tag#4 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s Feb 1 18:13:05 VOID kernel: sd 1:0:0:0: [sdb] tag#4 Sense Key : 0xb [current] Feb 1 18:13:05 VOID kernel: sd 1:0:0:0: [sdb] tag#4 ASC=0x47 ASCQ=0x0 Feb 1 18:13:05 VOID kernel: sd 1:0:0:0: [sdb] tag#4 CDB: opcode=0x28 28 00 0e 25 1a 28 00 00 20 00 Feb 1 18:13:05 VOID kernel: blk_update_request: I/O error, dev sdb, sector 237312552 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Feb 1 18:13:05 VOID kernel: ata1: EH complete Feb 1 18:13:05 VOID kernel: ata1.00: Enabling discard_zeroes_data It hasn't re-appeared yet, I'm copying more data as a test to see if it happens again. But so far the issue continues to follow the drive, so I think I might have gotten a defective one. Can anyone confirm? Other ideas I could try? On a related note, anyone have any experience with warrantying this type of issue with samsung? Like in idiot I have missed my return window with Amazon. void-diagnostics-20210201-1818.zip syslog-20210201-114006.txt syslog-20210201-161732.txt syslog-20210201-175504.txt EDIT: Still doing it, same errors as before: Feb 1 19:08:20 VOID kernel: ata1.00: exception Emask 0x10 SAct 0x80800000 SErr 0x0 action 0x6 frozen Feb 1 19:08:20 VOID kernel: ata1.00: irq_stat 0x08000000, interface fatal error Feb 1 19:08:20 VOID kernel: ata1.00: failed command: WRITE FPDMA QUEUED Feb 1 19:08:20 VOID kernel: ata1.00: cmd 61/60:b8:00:7e:db/00:00:05:00:00/40 tag 23 ncq dma 49152 out Feb 1 19:08:20 VOID kernel: res 40/00:b8:00:7e:db/00:00:05:00:00/40 Emask 0x10 (ATA bus error) Feb 1 19:08:20 VOID kernel: ata1.00: status: { DRDY } Feb 1 19:08:20 VOID kernel: ata1.00: failed command: WRITE FPDMA QUEUED Feb 1 19:08:20 VOID kernel: ata1.00: cmd 61/60:f8:80:7e:db/00:00:05:00:00/40 tag 31 ncq dma 49152 out Feb 1 19:08:20 VOID kernel: res 40/00:b8:00:7e:db/00:00:05:00:00/40 Emask 0x10 (ATA bus error) Feb 1 19:08:20 VOID kernel: ata1.00: status: { DRDY } Feb 1 19:08:20 VOID kernel: ata1: hard resetting link Feb 1 19:08:21 VOID kernel: ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Feb 1 19:08:21 VOID kernel: ata1.00: supports DRM functions and may not be fully accessible Feb 1 19:08:21 VOID kernel: ata1.00: supports DRM functions and may not be fully accessible Feb 1 19:08:21 VOID kernel: ata1.00: configured for UDMA/133 Feb 1 19:08:21 VOID kernel: ata1: EH complete Feb 1 19:08:21 VOID kernel: ata1.00: Enabling discard_zeroes_data

bump, still having issues with this.

You can run the User Scripts plugin to schedule your scripts. Public key auth would be your solution to allowing it to run with no manual intervention. LimeTech actually published a guide about the very thing you are trying to do: https://unraid.net/blog/unraid-server-to-server-backups-with-rsync-and-wireguard It uses WireGuard, but if you have another solution just skip that part and start at the section titled "Setting up Site A and Site B for RSYNC" SpaceInvaderOne I believe has also done a walk through video on the subject. EDIT: I was wrong or I can't find it. As for adding SSHPass, you could see if it is listed in or ask for it to be added to the list of available packages for the plugin. Though IMO Public key auth is the better way to go anyways.

UPDATE 3/3/2021: I have definitively determined my performance issues are caused by WireGuard. I do not yet know if or when I'll find a solution. UPDATE 8/1/2022: This is still very much broken. I try a file transfer every couple of months and it continues to be horribly slow. Using RSYNC over SSH outside the Wireguard tunnel works great and is what I will continue to use until I can figure this sh*t out. FINAL UPDATE 11/24/2022: See my last post here for solution and TL;DR: Let me preface all of this by saying I'm not sure where my issue lies, so I'm going to layout what I know and hopefully get some ideas on where to look for my performance woes. The before times: Before setting up WireGuard I had SSH open to the world (with security and precautions in place) on my main server so that once a month my backup server could connect and push and pull content as defined in my backup script. This all worked splendidly for years and I always got my full speeds up to the bandwidth limit I set in my rsync parameters. Now: With the release of WireGuard for UnRAID I quickly shutdown my SSH port forward and setup WireGuard. I have one tunnel for my administrative devices and a second tunnel which serves as sever2server access between NODE and VOID. NODE is my main server, and runs 6.8.3 stable. It is located on a 100Mbps/100Mbps fiber line. UPDATE: As a last ditch effort I upgraded NODE to 6.9.0-RC2 as well, no change in the issue. VOID is my backup, runs 6.9.0-RC2 and lives in my home on a 400Mbps/20Mbps cable line. In this setup, my initial rsync session will go full speed for anywhere from 5-30 minutes, then suddenly and dramatically drop in speed, down to 10Mbps or less and stay there until I cancel the transfer. I can restart the transfer immediately and regain full speed for a time, but it always eventually falls again. Here is my rsync call: rsync -avu --stats --numeric-ids --progress --delete -e "ssh -i /mnt/cache/.watch/id_rsa -T -o Compression=no -x -o StrictHostKeyChecking=no" root@NODE:/mnt/user/TV/Popeye/ /mnt/user/TV/Popeye/ Here is a small sample of the rsync transfer log to illustrate the sudden and sharp drop in speed: Season 1938/Popeye - S1938E09 - Mutiny Ain't Nice DVD [BTN].mkv 112,422,538 100% 10.80MB/s 0:00:09 (xfr#24, to-chk=58/135) Season 1938/Popeye - S1938E10 - Goonland DVD [BTN].avi 72,034,304 100% 9.76MB/s 0:00:07 (xfr#25, to-chk=57/135) Season 1938/Popeye - S1938E11 - A Date to Skate DVD [BTN].mkv 138,619,127 100% 10.44MB/s 0:00:12 (xfr#26, to-chk=56/135) Season 1938/Popeye - S1938E12 - Cops Is Always Right DVD [BTN].mkv 127,109,972 100% 11.02MB/s 0:00:10 (xfr#27, to-chk=55/135) Season 1939/Popeye - S1939E01 - Customers Wanted DVD [BTN].mkv 114,673,044 100% 10.50MB/s 0:00:10 (xfr#28, to-chk=54/135) Season 1939/Popeye - S1939E02 - Aladdin and His Wonderful Lamp DVD [BTN].mkv 325,996,501 100% 11.69MB/s 0:00:26 (xfr#29, to-chk=53/135) Season 1939/Popeye - S1939E03 - Leave Well Enough Alone DVD [BTN].mkv 105,089,182 100% 11.30MB/s 0:00:08 (xfr#30, to-chk=52/135) Season 1939/Popeye - S1939E04 - Wotta Nitemare DVD [BTN].mkv 149,742,115 100% 754.78kB/s 0:03:13 (xfr#31, to-chk=51/135) Season 1939/Popeye - S1939E05 - Ghosks Is The Bunk DVD [BTN].mkv 114,536,257 100% 675.53kB/s 0:02:45 (xfr#32, to-chk=50/135) Season 1939/Popeye - S1939E06 - Hello, How Am I DVD [BTN].mkv 92,083,730 100% 700.03kB/s 0:02:08 (xfr#33, to-chk=49/135) Season 1939/Popeye - S1939E07 - It's The Natural Thing to Do DVD [BTN].mkv 110,484,799 100% 715.66kB/s 0:02:30 (xfr#34, to-chk=48/135) Season 1939/Popeye - S1939E08 - Never Sock a Baby DVD [BTN].mkv 97,660,132 100% 716.88kB/s 0:02:13 (xfr#35, to-chk=47/135) Season 1940/Popeye - S1940E01 - Shakespearian Spinach DVD [BTN].mkv 102,543,357 100% 632.64kB/s 0:02:38 (xfr#36, to-chk=46/135) Season 1940/Popeye - S1940E02 - Females is Fickle DVD [BTN].mkv 102,363,188 100% 674.34kB/s 0:02:28 (xfr#37, to-chk=45/135) Season 1940/Popeye - S1940E03 - Stealin' Ain't Honest DVD [BTN].mkv 100,702,236 100% 732.80kB/s 0:02:14 (xfr#38, to-chk=44/135) Season 1940/Popeye - S1940E04 - Me Feelins is Hurt DVD [BTN].mkv 111,018,052 100% 672.35kB/s 0:02:41 (xfr#39, to-chk=43/135) Season 1940/Popeye - S1940E05 - Onion Pacific DVD [BTN].mkv 103,088,015 100% 650.18kB/s 0:02:34 (xfr#40, to-chk=42/135) Season 1940/Popeye - S1940E06 - Wimmin is a Myskery DVD [BTN].mkv 61,440,000 59% 757.02kB/s 0:00:56 ^C rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(701) [generator=3.2.3] and my accompanying stats page during the same transfer. You can see the sudden decline around 11:46 which coincides with my sudden drop in transfer speed above: I don't see anything telling in the system logs on either server when this speed drop happens. It almost seems like a buffer is filling up and not being emptied quick enough, causing the speed to tank. What I don't think it is: I don't think my issue is with WireGuard or my ISP speeds on either end. While the transfer is crawling along over SSH at sub-par speeds I can easily browse to NODE over WireGuard from my Windows or Mac computer and pick any file to copy over the tunnel and I can fully saturate the sending servers upload with no issues while SSH is choking in the background: Could it have something to do with the SSH changes that took place between 6.8.3 and 6.9.0? None of the changes I'm aware of sound like the culprit but I could be wrong. So besides that I'm pretty much out of ideas on what it could be without just playing with random ssh and rsync options. Let me know if there is some other info I can provide, below are both servers diagnostic files: node-diagnostics-20210204-0751.zip void-diagnostics-20210204-0752.zip EDIT: I just realized LimeTech has a guide about this published: https://unraid.net/blog/unraid-server-to-server-backups-with-rsync-and-wireguard I looked it over and I'm not really doing anything different except not passing -z (compression) to rsync and disabling compression for the SSH connection. a lot of what is transferred for me is video and doesn't compress well so why waste the CPU cycles on it.

Yeah I noticed that when I upgraded my test server to beta 31 (the first beta I tried this round). My non-root users were still able to login but I was getting random permission errors when trying to do things like launch MC. Unfortunately rather than mess with troubleshooting it I decided to just scrub it and wait until the new stable comes out to see what DocGyver and the community would come up with for non-root access again. I'll be following your thread though to see what you find out so do keep us posted.

The changes are detailed in beta 25 release notes: So they changed about 3 lines in the config which if I had to take a stab are: only root user is permitted to login via ssh (remember: no traditional users in Unraid OS - just 'root'): AllowUsers root non-root tunneling is disabled: Match Group root AllowTcpForwarding yes non-null password is now required: Not sure about this one, I think by default SSH already didn't allow blank passwords. PermitEmptyPasswords is set to No but commented out in a freshly generated UnRAID sshd_config file (at least for me). Are you using DocGyver's SSH plugin to allow non-root users access? I decided before trying out the betas to scrub his plugin and its changes from my system on the off chance I was going to have conflicts since he doesnt appear to have updated it for the betas. I have attached my sshd_config for you to compare with if you want as it should be entirely UnRAID defaults besides my changing of where to look for the authorizedkeysfile (line 44). sshd_config

Single or dual parity? UnRAID should be smart enough to keep track of your drive locations without you having to do anything special with a single parity disk. Dual parity is something I'm not familiar with as I don't currently use it but it does make a difference. With that in mind you should probably get a copy of your diagnostics zip and take a screenshot of the main tab where all your drives and serial #s are displayed in the current order just in case.

Oh yeah I thought about trying to dig through the PLGs myself but I didn't really know what I was looking for so I figured asking would be quicker. I'd just have a million questions about the PLGs instead lol. @dlandon thank you for the quick fix! Glad it wasn't anything serious.

Nice, how did you figure this out? Something in my diags?

I did remove Docgyver's SSH and denyhosts plugins after the beta update as they were quite old and I saw the SSH changes made in beta 25. I initially thought they might be my problem but even after removing those and rebooting this persisted.

The update checker in UnRAID disagrees with you. I assumed I was all good. How do I tell which are out of date if I can't trust the updater? EDIT: I even have automatic weekly plugin updates turned on in Squid's plugin.

Decided to dive straight into beta 35 (from stable) with my backup server to get ahead of any issues I might encounter as my backup and main are configured very similarly. I've noticed two innocuous errors on the monitor attached to the server. They don't show up in the logs anywhere that I can find and there is nothing preceding or following them. I rebooted into safe mode and those two lines don't appear. So it has something to do with a plugin, but I'm not sure how to go about figuring out which one. I poked through the beta threads and tried some searching but didn't see this mentioned anywhere else yet. I haven't noticed any lost or broken functionality yet, so they appear to be harmless but I would feel better knowing where they are coming from and what if any damage it may cause. Diagnostics: void-diagnostics-20201203-1515.zip

UnRAID v6.8.3 Diagnostics and the previous two system logs capturing roughly my last 30 days of uptime: node-diagnostics-20201129-0852.zip A very strange issue I have just now started experiencing. Twice in the last 48 hours UnRAID has completely list its ability to resolve DNS names without a reboot of the server. All attempts to ping by name result in "name or service not known". UnRAID is unable to resolve any names for update checks and the like. Pinging by IP address works without issue and my wireguard sever continues to provide access to the system. I have made no recent changes to network setup or DNS. My network settings are statically configured and I utilize 3 upstream DNS servers which all remain pingable during this outage: 8.8.8.8, 1.1.1.1, 8.8.4.4 When this issue occurs I can get into other devices on the network and they can resolve names just fine so this is something exclusive to my machine it would appear. If you look at my syslog from 11/28-11/29 (included in the zip above) you will see I rebooted the server and after a few minutes DNS resolution started working again and I was able to update some plugins and whatnot. Nov 28 11:49:12 Node emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin update fix.common.problems.plg Nov 28 11:49:12 Node root: plugin: creating: /boot/config/plugins/fix.common.problems/fix.common.problems-2020.11.28-x86_64-1.txz - downloading from URL https://raw.githubusercontent.com/Squidly271/fix.common.problems/master/archive/fix.common.problems-2020.11.28-x86_64-1.txz Nov 28 11:49:12 Node root: plugin: checking: /boot/config/plugins/fix.common.problems/fix.common.problems-2020.11.28-x86_64-1.txz - MD5 Nov 28 11:49:12 Node root: plugin: running: /boot/config/plugins/fix.common.problems/fix.common.problems-2020.11.28-x86_64-1.txz Nov 28 11:49:13 Node root: plugin: running: anonymous Nov 28 12:49:00 Node kernel: veth375a643: renamed from eth0 Nov 28 12:49:00 Node kernel: docker0: port 1(veth20db4d4) entered disabled state Nov 28 12:49:00 Node kernel: docker0: port 1(veth20db4d4) entered disabled state Nov 28 12:49:00 Node kernel: device veth20db4d4 left promiscuous mode Nov 28 12:49:00 Node kernel: docker0: port 1(veth20db4d4) entered disabled state Nov 28 13:51:25 Node kernel: mdcmd (47): spindown 4 Nov 28 13:51:26 Node kernel: mdcmd (48): spindown 6 Nov 28 13:51:26 Node kernel: mdcmd (49): spindown 7 Nov 28 13:51:26 Node kernel: mdcmd (50): spindown 9 Nov 28 13:51:52 Node kernel: mdcmd (51): set md_write_method 0 Nov 28 13:51:52 Node kernel: Nov 28 14:00:43 Node kernel: mdcmd (52): spindown 2 Nov 28 14:01:36 Node kernel: mdcmd (53): spindown 1 Nov 28 15:21:54 Node kernel: mdcmd (54): set md_write_method 1 Nov 28 15:21:54 Node kernel: Nov 28 17:25:38 Node kernel: mdcmd (55): spindown 1 Nov 28 17:25:48 Node kernel: mdcmd (56): spindown 2 Nov 28 17:25:50 Node kernel: mdcmd (57): spindown 9 Nov 28 17:25:53 Node kernel: mdcmd (58): spindown 7 Nov 28 17:25:56 Node kernel: mdcmd (59): set md_write_method 0 Nov 28 17:25:56 Node kernel: Nov 28 19:40:22 Node kernel: mdcmd (60): spindown 4 Nov 28 19:52:48 Node kernel: mdcmd (61): spindown 8 Nov 28 20:41:07 Node kernel: mdcmd (62): spindown 6 Nov 28 21:27:50 Node kernel: mdcmd (63): spindown 9 Nov 28 23:20:56 Node kernel: mdcmd (64): spindown 4 Nov 28 23:27:19 Node kernel: mdcmd (65): spindown 5 Nov 28 23:41:38 Node kernel: mdcmd (66): spindown 7 Nov 29 00:00:01 Node Docker Auto Update: Community Applications Docker Autoupdate running Nov 29 00:00:01 Node Docker Auto Update: Checking for available updates Nov 29 00:00:02 Node Docker Auto Update: No updates will be installed Nov 29 00:15:03 Node kernel: mdcmd (67): spindown 8 Nov 29 00:43:31 Node kernel: mdcmd (68): spindown 2 Nov 29 01:00:01 Node root: Fix Common Problems Version 2020.11.28 Nov 29 01:00:01 Node root: Fix Common Problems: Error: Unable to communicate with GitHub.com Nov 29 01:00:01 Node root: Fix Common Problems: Other Warning: Could not check for blacklisted plugins Nov 29 01:00:12 Node root: Fix Common Problems: Other Warning: Could not perform docker application port tests Nov 29 01:00:12 Node sSMTP[31740]: Unable to locate smtp.gmail.com Nov 29 01:00:12 Node sSMTP[31740]: Cannot open smtp.gmail.com:465 Nov 29 01:01:11 Node kernel: mdcmd (69): set md_write_method 1 Everything was great until 1AM when FCP wanted to run and was unable to resolve github.com again... I see nothing between those two events that explains my sudden loss of DNS or why this seems to be all of a sudden happening daily. I'd love some help in figuring this out as it brings my server and dockers to a grinding halt with so many parts depending on name resolution. EDIT: In the time it took me to draft this topic I have lost DNS resolution again. I'm at a loss as to what has suddenly changed to cause this... EDIT2: hmm it almost seems to be related to docker? I stopped the dockers and docker service and now its back up? I'm honestly just grasping at straws here though, I can't seem to find a rhyme or reason yet. Bringing docker back online doesn't seem to immediately break it. UPDATE: Well It's now been 48 hours with no issues...I still have no idea what caused this but it seems to have resolved itself....

Is your AT&T router in bridge mode or otherwise allowing pfsense to handle IP addressing, routing, DNS, etc? Putting the AT&T device in bridge mode essentially turns it into a dumb modem, allowing pfsense to handle your network traffic and security. If not you may be double NATing yourself which may be your biggest problem here. I couldn't find home user instructions and am not familiar with AT&T provided routers but this should be a start: https://www.att.com/support/smallbusiness/article/smb-internet/KM1188700/ Beyond that, I had on and off issues with remote access until I added: server:private-domain: "plex.direct" to the DNS resolver custom options box in PFSENSE. https://forums.plex.tv/t/secure-connections-lan-and-pfsense/123319/9 ^I didn't find the NAT+Proxy setting in that thread necessary, mine is set to system default which for me is keeping NAT reflection disabled. That and making sure the port forward was setup properly was, IIRC, all I had to do to get plex working behind PFSense. As mfwade mentioned, if you use PFBlockerng and have GeoIP filtering on you may want to turn it off until you get plex working to eliminate it as a potential problem.