Read/Write speeds slow after some issues

[chesh]

Yesterday I replaced a 4TB Red that was having read errors.  The disk I replaced it with I had already precleared (3 pass) and had it sitting aside as a hot spare.  At the same time, I upgraded from 6.3.1 to 6.3.3. So, not sure which, if either caused this issue.  But, after the disk rebuild finished, I seem to be having extremely slow read/write speeds.  I first noticed it when everyone connected to my Plex remote were continuously buffering.  I tried copying some files between shares via my laptop and noticed some slow copy speeds there, too (1mb - 20mb fluctuating). So, just for the hell of it, I just kicked off a parity check.  It's going at a whopping 55kb/sec.  Is this due to rebuilding the disk, or do I have something else going wrong?  I've attached a diagnostic report.  Thanks for any help anyone can provide!

tower-diagnostics-20170504-2129.zip

[Squid]

19 minutes ago, chesh said:

Yesterday I replaced a 4TB Red that was having read errors.  The disk I replaced it with I had already precleared (3 pass) and had it sitting aside as a hot spare.  At the same time, I upgraded from 6.3.1 to 6.3.3. So, not sure which, if either caused this issue.  But, after the disk rebuild finished, I seem to be having extremely slow read/write speeds.  I first noticed it when everyone connected to my Plex remote were continuously buffering.  I tried copying some files between shares via my laptop and noticed some slow copy speeds there, too (1mb - 20mb fluctuating). So, just for the hell of it, I just kicked off a parity check.  It's going at a whopping 55kb/sec.  Is this due to rebuilding the disk, or do I have something else going wrong?  I've attached a diagnostic report.  Thanks for any help anyone can provide!

tower-diagnostics-20170504-2129.zip

Yes, a parity check or rebuild will impact the RW speed of the array.  But, 55KB/s is too slow.  But, also bear in mind that at startup of the various apps, they might be doing something that is going to also impact the speeds - scanning the disks for available media...  Dynamix FIP might be creating hashes, etc.

 

There's nothing in the syslog to indicate any problem.  (And, you're not running 6.3.3 until you reboot).  Give it a bit to see if it speeds up.  If nothing happens, then try a reboot

[chesh]

That's just it.  While the disk was rebuilding yesterday, all speeds were fine.  As of today, the R/W speeds have been crap.  I've rebooted 3 times tonight (including the final one to downgrade to see if that was the issue), and it's still acting weird.  I'll continue testing in the morning to see if anything changes.  Since I originally posted, I've tested w/ my local Win7 VM on the UnRAID server copying files between shares and I'm getting anywhere from 750kb/sec - 99mb/sec. For instance, I started watching a show from my Roku 4 through my Plex docker and it was taking forever to spool.  RDP'd into my Win7 VM and started a copy process between shares and it started slow and once it started speeding up, my spool sped up too.  Not sure if that's indicative of something. Like I said, I'll continue to watch and see if it speeds up any, and post any follow up if it doesn't get better tomorrow.  I've already cancelled the parity check for the night to see if it gets better tomorrow. I'll kick off another parity in the morning to see if it's goes any quicker. 

 

PS: Squid, thanks for being a rock in the forums.  Don't ever leave. You're one of the best community support guys in here.

Edited May 5, 2017 by chesh

[chesh]

So, I let the server run over night and tried some moving of files between shares via a local VM on UnRaid.  Still have extremely slow write speeds (500kb/sec).  So, before I got in the showing this morning, I fully shutdown my server, let it sit for 10mins, and then powered back on.  This was 3hours ago.  I just tested copying some files between shares again, and I'm still having an issue.  Any thoughts on troubleshooting further? Attached is a new diagnostics.

tower-diagnostics-20170505-0850.zip

[chesh]

So, in further troubleshooting, I set all of my containers to not auto-start and then rebooted my server again.  Once back up, I kicked off another parity check and I was getting the expected speeds (80mb/sec - 140mb/sec). It ran like this for about 30mins and then dropped back down to 56kb/sec.  This is with literally nothing running but UnRaid.  Not sure what is causing the disk usage. Attached is a picture of the current disk usage with nothing running.  Anyway to figure out what's eating it?

[chesh]

After rebuilding just about everything today, I was finally able to figure out what was causing my slowdowns.  Putting the answer here, so in case anyone else runs against the same problem.  After starting dockers one at a time and watching the disk throughput via a parity check, it was SickRage causing the issue.  Turns out I had a ton of crap in the post-processing folder and when it would go to post-process, it was extracting a ton of crap all at the same time and trying to move it to my specified folders.  Weird thing about SickRage, it was also extracting movies, decided it didn't know what it was, and then moved on.  I had a lot of movies in my post-process folder.  So, I moved everything out of that folder that had already been post-processed, started back up SickRage, and here we are almost 45mins later and no issues.  I had successfully run Deluge and Plex for most the day and had no issues.  But, as soon as SickRage hit the post-processing task, there would go my speeds again.  It must spawn off a bunch of child threads when post-processing, as killing the SickRage container wouldn't bring my speed back.  It would take almost 20-30mins for it to finish what it was doing (even though the container was stopped) before it would go back to normal.  Hence the hard time trying to pin out the root cause.  Anyway, I hope this helps someone else.

[nommiiss]

Thanks for this Chesh! I was having very slow read/writes on my unRaid to the extent that I was going to move back to my old system. Turns that as I am a noob I decided to load all my Dockers on the system at once with a very bad result and once I have disabled the majority of them things have come back to life... 

 

Thanks again