Typing up a reply with all that I have found... oh yea.. your gonna love this LOL.
Cliffnotes = bad drive(s) but not the new ones
First I downgrade to 4.7 and BAM all the problems are gone... for a few minutes then the speeds start to act up again.
So I telnet in and check the drive stats, MD2 has errored and so on. I run down to check it out and can smell HD Death, CRAP!
Turns out issue #1 was a failing cache drive, temps didnt reflect it but she was in thermal meltdown!
I think "Great" problems solved right? NOPE, im still on 4.7 at this point so I think maybe the files have been corrupted.. I load up 5.0b6 and things are great for a few minutes then go to crap again.
WTF at this point im ready to light it up like the fireworks we'd been blowing up all night!!!
So, I load up a fresh batch of 5.0b7 which for some reason completely pissed off the network and locked up the modem, oh yea we're havin fun now! I get everything reset and load the webui. Its fresh and no drives assigned at this point and baby SHE'S FLYIN. I begin testing each drive one by one to see which one slows the system down. I hit drive #4 (WD 2tb 2mos old) and it slows to a halt. I'm able to load all drives but that one and the box is better than its ever been.
I'm seeing write speeds of 80MB/s and read speeds of over 120MB/s, this is without a cache drive too. Never seen a system move this fast.
I guess it didnt like being messed with the first time around, when the drives came back up it was not a happy camper. Now I need to see if the WD is bad or if its just got a few misplaced files with errors?