I got unRAID around 8 months ago, and I have issues 2-3x a month. I had an issue a month or so ago where my parity checks were very slow (2MB/s). I fixed it by moving one of my PCI-X cards down to the other PCI Bus, and then it was 87MB/s. Now a month later, after basically no changes, I'm getting 2MB/s-11MB/s on my parity check. There are people with my exact SATA cards, motherboard, and hard drives having no issues. This cannot be a PCI-Bus issue as I was using 14 drives and parity syncing at 86MB/s less than a month ago! I've seen members here getting 86MB/s on 20 drive systems with my exact hardware.

 

1) No SMART errors.

2) All drives were precleared and tested.

3) System worked fine a month ago, nothing has changed other than replacing a SATA cable.

 

(Solved) Rewired case, got rid of round robin cabling, and all hard drives are using only onboard, and SATA card 1. I'll see if this problem returns after I get enough hard drives to start using SATA card 2.

(Solved Again) Post

syslog.txt

I'm now getting getting a max of 15MB/s at 2.2%, and it's widely varying between 2MB/s-15MB/s.

Could tons of small files on one hard drive cause this?

If you are talking about parity build/check speeds, then No.  Parity doesn't care about the file system on the disk.

But if you are talking about file transfer speeds, then Yes.  Tons of small files on one hard drive can cause this.  Especially if the disk is almost full.

 

I'm now getting getting a max of 15MB/s at 2.2%, and it's widely varying between 2MB/s-15MB/s.

Could tons of small files on one hard drive cause this?

If you are talking about parity build/check speeds, then No.  Parity doesn't care about the file system on the disk.

But if you are talking about file transfer speeds, then Yes.  Tons of small files on one hard drive can cause this.  Especially if the disk is almost full.

 

 

Then i'm stumped on why my parity build is going 2-15MB/s, each time I refresh it seems like it's a drastically different number.

Then i'm stumped on why my parity build is going 2-15MB/s, each time I refresh it seems like it's a drastically different number.

 

I see that you have cache_dirs running at the same time as you are running the parity build.  That may be bad idea.

 

Then i'm stumped on why my parity build is going 2-15MB/s, each time I refresh it seems like it's a drastically different number.

 

I see that you have cache_dirs running at the same time as you are running the parity build.  That may be bad idea.

 

 

Sorry, I forgot to mention that I noticed that happened so I stopped parity sync, waited for cache to finish, then restarted the parity sync. Same thing.

 

I'll post a new, easier to read, syslog in a second.

Posted the new syslog on main post.... restarted the server, dir cache is disabled, started parity sync... still 2MB/s.

try moving yoru hdds, you may have a faulty backplane.

 

just a guess.

try moving yoru hdds, you may have a faulty backplane.

 

just a guess.

I disabled 5 of my hard drives, and it return to 82MB/s. I re-added 2 more hard drives, and it's stuck at 2MB/s. So I disabled those 2 hard drives and re-added the other 3. Still stuck at 2MB/s. Keep in mind that every single one of these hard drives are using different SATA cables, SATA ports, and power cables.

 

I've ruled out cables, PCI-bus bandwidth, and faulty drives by trying these configurations:

Parity + Data Disks 1-5 = 82MB/s

Parity + Data Disks 6-10 = 82MB/s

Parity + Data Disks 5-10 = 2MB/s

Parity + Data Disks 1-6 = 2MB/s

Parity + Data Disks 1,3,5,7,9 = 82MB/s

Parity + Data Disks 1,2,3,5,7,9 = 2MB/s

Parity + Data Disks 1,3,5,7,9,10 = 2MB/s

 

I'm not sure what to do now. This seems like an unraid problem. Anything over 6 data disks causes slow parity syncs.

Does anything show up in /var/log/messages that isn't showing in /var/log/syslog ?

 

I dont suppose you have a 'spare' motherboard you could try? This seems odd, almost like there's stray interrupts causing havok.

Does anything show up in /var/log/messages that isn't showing in /var/log/syslog ?

 

I dont suppose you have a 'spare' motherboard you could try? This seems odd, almost like there's stray interrupts causing havok.

 

Can you tell me how to access /var/log/messages? I'm very inexperienced with unraid.

 

No spare motherboards.

I completely rewired my case so that the hard drives were only on the onboard, and first sata controller.. and it seems to be working. So it almost seems like the 2nd sata card has limited bandwidth and causing a bottleneck, even though it's the same as the other one.

 

Either that or it's just a downside to "round robin" wiring.

Go to the Settings page under 'Disk settings' and disable spinup groups & then see if this has any effect (it shouldn't).  Assuming issue still exists, you are on the right track by isolating groups of drives per controller.  It could very well end up being a bios issue with your motherboard and/or the controller cards and/or this particular h/w config used together.  Would be nice to see a syslog after 'slow' parity check has run for a few minutes to see if anything unusual in there.  When you mention similar configs of other people work fine - what are some examples of these configurations?

I completely rewired my case so that the hard drives were only on the onboard, and first sata controller.. and it seems to be working. So it almost seems like the 2nd sata card has limited bandwidth and causing a bottleneck, even though it's the same as the other one.

 

Either that or it's just a downside to "round robin" wiring.

 

Keep in mind.

 

There are two PCI-X buses on that board.

Each one can be configured to run at a different speed.

1 bus can run at a maximum of 133MHZ (upper PCI-X slots near CPU).

1 bus can run at a maximum of 100MHZ (lower PCI-X slots near edge of board).

 

Each bus can be configured for AUTO - PCI 33, PCI 66, PCI-X 66, PCI-X 100, PCI-X 133

 

Check your bios.

 

In the past I have had an issue with cables, A noisy cable or connection would slow everything down.

 

 

Also how do you have the drives/controllers laid out?

Is the parity on the motherboard or on a controller? (S/B motherboard).

 

Do you have any other PCI cards in the slots any where.

 

For the record, I used a promise TX4 PCI in a PCI-X slot at PCI-66 and I had excellent transfer speeds, however I did specifically set the bus to PCI-66 to make sure it was at a higher speed then PCI-33.

 

Also keep in mind, if you keep refreshing the web page, it interferes with the parity check speed as each hard drive's identity and temperature are read.

 

Do not refresh to often. I usually do it a couple times then wait 10 minutes before doing it again.

Go to the Settings page under 'Disk settings' and disable spinup groups & then see if this has any effect (it shouldn't).  Assuming issue still exists, you are on the right track by isolating groups of drives per controller.  It could very well end up being a bios issue with your motherboard and/or the controller cards and/or this particular h/w config used together.  Would be nice to see a syslog after 'slow' parity check has run for a few minutes to see if anything unusual in there.  When you mention similar configs of other people work fine - what are some examples of these configurations?

 

Spinup Groups is one of the first things I disabled. I let it ran to 5% (3 1/2 hours) and there wasn't anything new in my syslog.

 

Here's a post with a guy my exact mobos, and 2 of the same SATA cards:

http://lime-technology.com/forum/index.php?topic=5477.msg51052#msg51052

 

He's running 85MB/s with 20 EADs with no problems. He seems to have had the same issue as me, and moved one of his SATA cards down to the 100mhz slot which solved his problem. Somehow that only worked for me for around a month (Which honestly doesn't make sense, PCI-Bus should all a sudden not be able to handle the bandwidth)

 

I completely rewired my case so that the hard drives were only on the onboard, and first sata controller.. and it seems to be working. So it almost seems like the 2nd sata card has limited bandwidth and causing a bottleneck, even though it's the same as the other one.

 

Either that or it's just a downside to "round robin" wiring.

 

Keep in mind.

 

There are two PCI-X buses on that board.

Each one can be configured to run at a different speed.

1 bus can run at a maximum of 133MHZ (upper PCI-X slots near CPU).

1 bus can run at a maximum of 100MHZ (lower PCI-X slots near edge of board).

 

Each bus can be configured for AUTO - PCI 33, PCI 66, PCI-X 66, PCI-X 100, PCI-X 133

 

Check your bios.

 

In the past I have had an issue with cables, A noisy cable or connection would slow everything down.

 

 

Also how do you have the drives/controllers laid out?

Is the parity on the motherboard or on a controller? (S/B motherboard).

 

Do you have any other PCI cards in the slots any where.

 

For the record, I used a promise TX4 PCI in a PCI-X slot at PCI-66 and I had excellent transfer speeds, however I did specifically set the bus to PCI-66 to make sure it was at a higher speed then PCI-33.

 

Also keep in mind, if you keep refreshing the web page, it interferes with the parity check speed as each hard drive's identity and temperature are read.

 

Do not refresh to often. I usually do it a couple times then wait 10 minutes before doing it again.

 

I originally had them laid out round robin, however I made sure the parity drive was onboard, and I only used 4/6 of the ports for onboard. Now that I layed them out to be grouped on the same controllers, it's working but I have nothing connected to the 2nd SATA card now.

 

Originally it was this:

Drive 1 (Parity) - Onboard

Drive 2 - SATA Card 1

Drive 3 - SATA Card 2

Drive 4 - Onboard

Drive 5 - SATA Card 1

Drive 6 - SATA Card 2

etc

 

I just set it like this:

Drive 1 (Parity) - Onboard

Drive 2 - Onboard

Drive 3 - Onboard

Drive 4 - Onboard

Drive 5 - SATA Card 1

Drive 6 - SATA Card 1

etc

 

Both SATA cards we're full, and onboard was 4/6 ports used. However there wasn't a drive connected to all hotswap bays.

Since your using a norco-4020 case, have you ruled out that one of the backplanes might have developed an issue?

Reaching here, but could one of the SATA cards have issues which only cropped up now.

 

Did you try swapping just the cards?

Since your using a norco-4020 case, have you ruled out that one of the backplanes might have developed an issue?

 

Yes, I tested all hard drives by themselves. All hard drives are still in the same place, but now just wired differently to ignore my 2nd SATA card and my speed is now fine.

 

It's something relating to the 2nd SATA card, but it was working fine before.

 

Reaching here, but could one of the SATA cards have issues which only cropped up now.

 

Did you try swapping just the cards?

 

Good idea, never thought of that.. though it will be a headache because I have 8 wired connected to each. I think i'm going to just let it be for now, my speeds are fine, and I just got a new hard drive and it's still fine. My next hard drive will start using the 2nd SATA card so if the issue pops up again that will be the first thing I try.

 

Here's something i've been wondering:

My motherboard allows me to set my PCI-X slots to 100mhz or 133mhz. Right now they are both on AUTO. Can I just set them box to 133MHz even though the bottom one is rated for 100MHz? I'm just confused on why they would allow you to overclock the PCI-X Slot.. seems risky.

Since your using a norco-4020 case, have you ruled out that one of the backplanes might have developed an issue?

 

Yes, I tested all hard drives by themselves. All hard drives are still in the same place, but now just wired differently to ignore my 2nd SATA card and my speed is now fine.

 

It's something relating to the 2nd SATA card, but it was working fine before.

 

Just because it was working fine before does not mean it is now.  I have had many a motherboard go bad on me without warning and after having worked for quite some time.

Since your using a norco-4020 case, have you ruled out that one of the backplanes might have developed an issue?

 

Yes, I tested all hard drives by themselves. All hard drives are still in the same place, but now just wired differently to ignore my 2nd SATA card and my speed is now fine.

 

It's something relating to the 2nd SATA card, but it was working fine before.

 

Just because it was working fine before does not mean it is now.  I have had many a motherboard go bad on me without warning and after having worked for quite some time.

 

I will test the 2nd SATA card when I get another hard drive. It's already out of it's 30 day warranty and I still have the company warranty for a long time, so no rush. With everything on onboard/SATA card 1 my parity sync finished overnight... but I also rewired everything differently so it is possible that it was something to do with the wiring.

Once again, it's doing it again. Haven't changed a damn thing. I did the parity sync, and it went around 52MB/s. Now i'm trying to do a parity check to verify everything was fine and it's sitting at 2MB/s. It should be at 82MB/s. I still have 8 drives connected to the first SATA card, and 4 on the onboard. Didn't open the case, didn't touch any settings - It worked on this configuration 10 minutes ago!

 

This problem makes no sense. Once again, this is pointing to an issue with unraid or a hardware conflict with unraid. The parity sync stayed at 52MB/s the entire time, 10 minutes later I start a parity check with everything the same at it's back to 2MB/s. I have a very hard time believing this is hardware now. My case's HD leds show that the drives work for about 1 second, then stop working for a second, and that keeps repeating. When it's working they just constantly stay on.

Reaching here, but could one of the SATA cards have issues which only cropped up now.

 

Did you try swapping just the cards?

 

Good idea, never thought of that.. though it will be a headache because I have 8 wired connected to each. I think i'm going to just let it be for now, my speeds are fine, and I just got a new hard drive and it's still fine. My next hard drive will start using the 2nd SATA card so if the issue pops up again that will be the first thing I try.

 

It will. No doubt about it.

 

Here's something i've been wondering:

My motherboard allows me to set my PCI-X slots to 100mhz or 133mhz. Right now they are both on AUTO. Can I just set them box to 133MHz even though the bottom one is rated for 100MHz? I'm just confused on why they would allow you to overclock the PCI-X Slot.. seems risky.

 

I don't think it's possible to overclock the PCI-X slot. In any case, If there were any question as to compatibility, I would swap cards in slots.

I had a question about an issue like this. Turns out I went to swap cards and noticed I did not have it in all the way (or it was heat/vibration creep out).

 

Also if there is any question, I set the clock speed manually just to be sure.

I purposely did this during my testing. (and under-clocked too just to get benchmarks).

Once again, it's doing it again. Haven't changed a damn thing. I did the parity sync, and it went around 52MB/s. Now i'm trying to do a parity check to verify everything was fine and it's sitting at 2MB/s. It should be at 82MB/s. I still have 8 drives connected to the first SATA card, and 4 on the onboard. Didn't open the case, didn't touch any settings - It worked on this configuration last night.

 

This problem makes no sense. Once again, this is pointing to an issue with unraid or a hardware conflict with unraid. The parity sync stayed at 52MB/s the entire time, 10 minutes later I start a parity check with everything the same at it's back to 2MB/s. I have a very hard time believing this is hardware now. My case's HD leds show that the drives work for about 1 second, then stop working for a second, and that keeps repeating. When it's working they just constantly stay on.

 

Honestly this points more to a problem with Hardware then unRAID.  I was running an Abit board for the longest time without any problems, then all of the sudden, drives started dropping out of the array.  I thought the drives were just dying (and one was) but the other 3 were for a different reason.  The onboard SIL controller decided to take a shit on me.  The board still work, as long as i am using the onboard ports and none that are related to the embedded SIL controller.  I ended up giving that board to a room mate adn told him about the problem.  I told him i was not sure if it would work like it was supposed to but that i thought the problem was only with the SIL controller.

Once again, it's doing it again. Haven't changed a damn thing. I did the parity sync, and it went around 52MB/s. Now i'm trying to do a parity check to verify everything was fine and it's sitting at 2MB/s. It should be at 82MB/s. I still have 8 drives connected to the first SATA card, and 4 on the onboard. Didn't open the case, didn't touch any settings - It worked on this configuration 10 minutes ago!

 

This problem makes no sense. Once again, this is pointing to an issue with unraid or a hardware conflict with unraid. The parity sync stayed at 52MB/s the entire time, 10 minutes later I start a parity check with everything the same at it's back to 2MB/s. I have a very hard time believing this is hardware now. My case's HD leds show that the drives work for about 1 second, then stop working for a second, and that keeps repeating. When it's working they just constantly stay on.

 

Swap the cards to rule out slots and take the "other card" totally out of the system.

 

I would suspect a drive is timing out or retrying. Is there anything in the syslog.

Any clicking?

 

You can rapidly initiate a SMART short test on the drives which will take mere minites.

If need be you can run a SMART long test over night on an "idle" system.

 

Check the temperature of the heatsinks on the motherboard (carefully).

Make sure there is airflow across them and they are not burning/sizzling hot.

Once again, it's doing it again. Haven't changed a damn thing. I did the parity sync, and it went around 52MB/s. Now i'm trying to do a parity check to verify everything was fine and it's sitting at 2MB/s. It should be at 82MB/s. I still have 8 drives connected to the first SATA card, and 4 on the onboard. Didn't open the case, didn't touch any settings - It worked on this configuration 10 minutes ago!

 

This problem makes no sense. Once again, this is pointing to an issue with unraid or a hardware conflict with unraid. The parity sync stayed at 52MB/s the entire time, 10 minutes later I start a parity check with everything the same at it's back to 2MB/s. I have a very hard time believing this is hardware now. My case's HD leds show that the drives work for about 1 second, then stop working for a second, and that keeps repeating. When it's working they just constantly stay on.

 

Swap the cards to rule out slots and take the "other card" totally out of the system.

 

I would suspect a drive is timing out or retrying. Is there anything in the syslog.

Any clicking?

 

You can rapidly initiate a SMART short test on the drives which will take mere minites.

If need be you can run a SMART long test over night on an "idle" system.

 

 

Nothing in Syslog, no clicking.. all drives appear fine in SMART. Is there a way to do a SMART test on all the drives in the system without having to do them one-by-one?

 

EDIT: Just ran Short SMART test on all drives, no drives had any errors.

 

Check the temperature of the heatsinks on the motherboard (carefully).

Make sure there is airflow across them and they are not burning/sizzling hot.

 

Airflow is good, heatsinks are barely warm and BIOS claims they are fine.. all hard drive temps are fine (Highest is 35).