Drive speeds are abyssmal

[icedragonslair]

dgaschk,

1 - I removed and started with stock system as per instructions in your sig, (wasn't sure about the '&' in the go file so I added the space before it).

2 - Re-installed unmenu and edited the second line in the go file to start unmenu upon reboot.

3 - I did not update webgui.

4 - bios for supermicro card was already at latest, flashed again anyway

Will test transfer speed tomorrow

New log

syslog-2014-05-10.txt

[icedragonslair]

Nope speeds are still horrible, now to tackle all of the network stuff...sigh

 

But here's a neat one; I set every share to exclude disk8, and yet no matter what it still writes data to disk8 when I transfer...I want to keep this disk empty...is there a way to do this as I am moving data around to possibly rebuild my server from scratch?

 

 

[trurl]

Nope speeds are still horrible, now to tackle all of the network stuff...sigh

 

But here's a neat one; I set every share to exclude disk8, and yet no matter what it still writes data to disk8 when I transfer...I want to keep this disk empty...is there a way to do this as I am moving data around to possibly rebuild my server from scratch?

Are there any folders already on disk8? Any folder at the top level of cache or an array disk is automatically a user share. Possibly this would mean that any top level folder already on disk8 will cause disk8 to be considered part of that folder's user share regardless of what you set in the webGUI.

[icedragonslair]

Nope, it creates them as I transfer, this is excruciatingly slow as it is, having to copy stuff an extra time is just downright cruel...;(

 

At this point frustration is really getting to me, as I haven't been able to run a complete parity check in the last two months since it it says it will take thousands of minutes to complete at 3.2mb/s and it completely renders the server useless at the same time, also the slow transfer speeds are now making me look elsewhere for solutions (flexraid, etc).

 

Obviously, I know how to do very little of what is needed for this and having time to actually learn it all without very clear instructions is just making matters even more frustrating.

Any help is always appreciated, but at this point I don't know what else to do but accept it as the mediocre performance server it has turned into.

 

I guess I really do not understand how this happened with nothing seeming to under-perform as far as hardware is concerned and having changed nothing with anything else...it just started happening.

 

Let's just throw another item out there, I just transferred the exact same files between two windows systems, the same source system and the same destination system (hardware used for unraid server) along with the win7 install I setup a couple of moths ago. The files are transferring at 80mb/s stable.

Sigh...

 

Thanks,

Ice(RickH)

[DaleWilliams]

*** I was reading this thread from the start, just to see what was going on...

This may have been covered earlier and I just missed it or don't see it, but:

 

Have you checked the BIOS of your unRAID server? To make sure ALL your harddrives are set up for AHCI? (also called SATA/AHCI in some BIOS) 

If not set, unRAID will still work, but disk read/write will be very slow.

 

[icedragonslair]

Dale,

Yes they are, However I do not know about the MV8 card, I am assuming it is since there are no settings for it in the marvel GUI

[icedragonslair]

Maybe this will help, I unasigned disk 8 (It is empty so it is where it should have been writing to) from the array and it wrote a 3gb file at 77mb/s (lightspeed...lol) however I am assuming it was written directly to the parity drive, I then re-wrote it to the correct drive (#7) and it wrote at it regular speed (3.5mb/s) what would cause this behavior?

 

 

[doorunrun]

Maybe this will help, I unasigned disk 8 (It is empty so it is where it should have been writing to) from the array and it wrote a 3gb file at 77mb/s (lightspeed...lol) however I am assuming it was written directly to the parity drive, I then re-wrote it to the correct drive (#7) and it wrote at it regular speed (3.5mb/s) what would cause this behavior?

The way I understand unRAID works, you're not writing directly to the parity drive in that sense. When you save a file to unRAID it will be written to one of the drives, but at the same time a parity is computed and that value is written to the parity drive. For simplicity's sake, parity is a sum of the data on all drives at one position. Oh, that's a crappy explanation; you'd better check the wiki for the official answer!

It sounds like you are confusing parity with cache.

 

When you wrote it the first time what drive got written to? I may not understand what you meant by "correct drive (#7).

 

unRAID Wiki Parity page:

http://lime-technology.com/wiki/index.php/Parity

 

[doorunrun]

Another thing that may not have been covered fully; what about posting SMART reports on each of the drives? And this would be those generated from unRAID with the drives as connected.

 

There's some instructions on doing this "the right way" http://lime-technology.com/wiki/index.php?title=Troubleshooting#Obtaining_a_SMART_report

[icedragonslair]

Here are the requested short tests  for drives 1-4 (sdc-sdf)

 

Thank you,

 

Please let me know if I needed to do something else

smartsdc.txt

smartsdd.txt

smartsde.txt

smartsdf.txt

[icedragonslair]

Here they are for drives 5-8 (sdg-sdj)

smartsdg.txt

smartsdh.txt

smartsdi.txt

smartsdj.txt

[icedragonslair]

And drive 9 (sdk)

 

smartsdk.txt

[doorunrun]

Thanks for posting the test results. I don't see where there's a big problem. The odd-duck is your WD20EVDS it's a 512 byte sector logical/physical drive; all the others are "advanced format" (512/4096) I think it's called. One drive "sdg" shows 2 UDMA-CRC errors which isn't a big deal as far as I know and mostly due to cabling/connector issues.

 

Almost a year ago you posted a thread with much the same problem only it was read errors http://lime-technology.com/forum/index.php?topic=28180.0 In that case I think, there was a drive replacement, right? So, I guess things got better for a while and now it's back to the old problem?

 

Earlier in this thread you posted that parity checks run very slowly  and that would indicate data transfer within the box has problems. I'm just about out of ideas.......

 

Have you run extended memory tests (memtest)? Power supply up to snuff?

 

You should probably work on testing internal file testing. I think you can use Midnight Commander to copy a file between drives and get some stats about transfer speeds. I haven't tried it but it could give you an idea if there's one disk creating the problem.

[icedragonslair]

doorunrun,

 

I did have an issue before, but then I removed the drive and precleared it again and it tested fine, speeds were back up to where they were than this happened.

Power supply is up to snuff and I have been trying to figure out mc (an hour trying to find commands and instructions...lol) but, evidently that will going to take longer than rebuilding the server...lol.

 

At this time I am transferring my data down to 2 4tb externals and then i will decide what to do. Since I have a spare copy of win7 pro 64 kicking around I may try other stuff, but I already paid for the unRAID pro so I will at least give it another try before trying FR and win7 (the only advantage I see of that is dlna built in)

[doorunrun]

Ice, thanks for the explanation. If you do rebuild unRAID you might want to add do a little testing as in setting up a single drive array (no parity) and try some transfers. Then add parity and try transfers again and possibly run a parity check and build up from there.

 

Whatever you decide to to do good luck with your system!

[icedragonslair]

Doorunrun,

Right now that is the plan, I have the data migrating at this moment (2x 4tb externals...lol) and once its done I will be killing the setup and starting fresh and as you said I will be adding one drive at a time and testing all thoroughly. Starting with the one drive and using different HW setups/cables  as well.

Hopefully I can get this nailed down, I have one question...

Is it worth it for me to try v6 beta as this time, especially since I use no additional addons?

 

Thanks loads, will post here when it is done

Ice (Rick)

[doorunrun]

.............Is it worth it for me to try v6 beta as this time, especially since I use no additional addons?

Yes! Since it's 64-bit based you'll be taking any of the PAE weirdness out of the equation. As you know you don't have to use Xen as the default is for it not to run. I only use it on a test server with Xen to catch some "nightly" TV shows and it's working well. I use the apcupsd plugin which adds the powerdown script. Of course it's Beta, but reliability of data storage is #1 on Tom's list.

 

Now, it wouldn't hurt to read the V6 release thread either, but since you're not going to add plugins I think you should be OK.

 

[icedragonslair]

doorunrun, all data removed and resetting disks, one at a time. I will be preclearing them again, once they are fully tested ()first disk is already flying (138mb/s parity check speed), which raised my eyebrows considerable and I now feel there was more than one issue at work here. Possibly a poorly performing drive along with a suspected slowdown caused by either a faulty breakout cable or even both mv8 cards may have issues, I am leaning toward the cables. I will continue to test on just the motherboard controller for the first few drives, I will then switch to the card and see the difference if any. If there is indeed a difference I will order a new cable (again) to confirm it.

 

So if I do indeed setup the v6 and run my second server (backup) am I better off running the new primary server with a cache drive and will it make any difference since I transfer all of my data manually (teracopy), without using a mover, I never really saw the need for it before, but if it would make a difference (like a 250gb ssd?), I would then try it.

 

Thanks,

Ice (Rick)

[doorunrun]

.......So if I do indeed setup the v6 and run my second server (backup) am I better off running the new primary server with a cache drive and will it make any difference since I transfer all of my data manually (teracopy), without using a mover, I never really saw the need for it before, but if it would make a difference (like a 250gb ssd?), I would then try it.

Ice, I'm glad to see you're making some headway towards isolating the slowdown problems!

 

In response to your question it's going to be hard for me to say how much performance you'd gain since I don't utilize my unRAID system in the same way. So I'll have to just give you an opinion. When you have a cache drive it  becomes a temporary holding spot. During the transfer parity is not computed or written. If your cache is a SSD and operating at 6 Gbps, then it should give you the best disk  transfer performance. Can the source at the other end of the wire provide matching performance?  Your data is in limbo (not protected) while it sits on the cache drive and you can set the mover to move more frequently.

 

Given the cost of the SSD and vulnerability of the data, I don't think it's not going to make that much of a difference in transfer speed to matter. But that's just me. Maybe posting the the "Cache or not to cache" thread you'll get a more "real world" answer.

 

BTW, I did some checking on iPerf, the network testing utility. There's a package available in unMENU that will load it onto your unRAID server. You can download a Windows command line version from http://iperf.fr/. For the basic test the commands are pretty simple; on the unRAID end type in "iperf  -s" to make it the server, on the Windows end type in "iperf -c" to make it the client and the test will run. You can then switch ends, so to speak. There are additional web support pages linked on that URL. There's also an enhanced Windows version called JPERF that uses Java and is more graphical. I don't think the unMENU package will persist through a reboot and just gets loaded into unRAID's RAM file system. It uses iperf v. 2.0.5, from circa 2011 which should match the URL's version pretty closely.

[icedragonslair]

iPerf does not install correctly through unMenu have posted in the iperf thread, hoping for help

 

[doorunrun]

I worked out the script needed to install iPerf through unMENU for  V6b5a running 64-bit. It's posted to the iPerf thread: http://lime-technology.com/forum/index.php?topic=10771.msg308804#msg308804.

 

The 32bit version takes a couple more steps. You can get the tgz file from the link in this post: http://lime-technology.com/forum/index.php?topic=10771.msg308532#msg308532 . You don't need the conf file since it should be in your /boot/packages folder already. Just drop that tgz file in there and unMENU should be OK seeing it and installing it. If not I should be able to rework the conf file.

[icedragonslair]

Thanks Doorunrun,

Awaiting a new key (second key deal ).

I am completely precleared.

I have tested each drive speed through parity rebuilding solo and as a group with v5.0.5 but I am waiting for new key to set this up with v6 from scratch, I have used my old key to setup my backup server.

For now I am just using 3-4TB USB3 drives to house my media and all is fine at this time, can't wait to give v6 a try 

 

Thanks,

Ice

 

Pairty running:

One drive only (test) at 158.3 mb/s (far cray from 3-4mb/s...lol)

Three drives (3 x WD20EURX) (test) at 141.9 mb/s, next up; test read & write speeds for the sata 6g drives, then on to the sata 3g drives

Six drives running parity build and check at about 135 mb/s down to 97 mb/s at the very end, this is so fast in comparison to what it was before I started rebuilding

 

RESULTS:

Read & write, at this stage, for these drives are comparable at 40-60 mb/s with writes slowing to 17 mb/s at sometimes & read bursting to 81 mb/s

This is withing my previous high boundaries, looks like the server is back on track

 

Now for the sata3g drives (3 x WD20EURS)

RESULTS:

Read & write, at this stage, for all drives (1 WD30EURS (Parity), 3 x WD20EURX + 3 x WD20EURS) is somewhat comparable to the previous results, but slowed slightly (I can assume due to adding the sata 3g drives to what was a sata 6g array).

They are almost identical to above peak write speed on the array, with a complete directory tree totaling 690gb (4 folders deep) peaked at 72mb/s and bottomed out at 16mb/s and this is spread across 3 drives as I have it set, just amazing the peak is just about 25% faster than it was before

[icedragonslair]

@doorunrun,

Thanks for the cache info, I will probably forgo the cache for now until needed for something with xen if I choose to use it.

 

As for the server, I have now gotten everything back together and I am running v6.0-beta5a, wow, just...wow...lol. (see above post for numbers)

I think I nailed the problem, between two very slow performing drives (RMA'd) and the possibility of other 'weirdness' going on I think we are all set.

Once the system is completely reloaded I will capture a log and post it and keep 'fingers crossed' that when the new drives are in and installed everything is still the same.

 

As far as unRAID v6, it is absolutely amazing great work to all and thanks for all the help

 

Ice (Rick)

[doorunrun]

That's great news! I suggested to TomH another poll question about V6 and those using it in production. But, no word back on it. I would switch my primary system over to it but I use a couple of plugins and I want to see the plugin manager issue stabilize.

I appreciate your thanks and hope your system continues to run well.