Occasional sluggish performance: X9SCM-F/Sandy Bridge 2.3GHz

[Auggie]

When I outgrew my maxxed-out 14-drive server, I built another server on a used X9SCM-F mobo and Sandy Bridge 2.3GHz processor.  All other components, including RAM, were brand new.  The original server I re-tasked as a general purpose file server with a whole different set of drives. 

 

I moved all the drives and expanded to the maximum capacity of 24 drives into the new server.  It has always been somewhat sluggish accessing it's shares, both user shares and individual disks.

 

I kept thinking perhaps the performance issue was Mac related (especially under Mavericks), but I've never had any performance issues accessing the Atom-based old server from the same Mac, before or after.  And when I tried accessing the new server from a non-Mac machine (PCH C-200 to be specific), it, too, seems to get hung up for awhile at times.  And so too with the web interface: sometimes there would be a long delay before the page is completed or refreshed.

 

There are never any errors generated in any form or fashion.  The machine itself never crashes, and if I don't have delays accessing it via telnet or working the command directly on the machine.

 

I've already sent the mobo to SuperMicro for a dead PCI slot issue (covered under warranty) so I assume they would have performed other diagnostics to determine if the motherboard was defective in any other way.

 

I'm at my wits end and contemplating buying a factory brand-new motherboard and CPU just to check if the used motherboard and/or CPU are suspect.  Or is it because of more overhead with 24 drives?

 

Are there any other troubleshooting methods I can try before taking such a drastic step?

[BobPhoenix]

Hopefully someone with more knowledge can respond but your plan to start with a factory fresh MB/CPU sounds like a plan.  Another would be to replace "3x Supermicro AOC-SASLP-MV8 SAS/SATA PCIe" from your signature with LSI based controllers like the IBM M1015s or the updated SuperMicro AOC-SAS2LP-MV8 if the MB/CPU replacement doesn't work well enough - or you find some controllers you can borrow first as a test.  The SASLP-MV8 is a slower controller then the SAS2LP-MV8 and the M1015 so that might make a difference in your response times.  But that is just my opinion and may not mean much.

[c3]

You don't mention what version you are running. Also some details like amount of RAM (yes, I can see 16GB in your signature, but you mention a new build).

http://lime-technology.com/forum/index.php?topic=27783.msg246595#msg246595

[Auggie]

Hopefully someone with more knowledge can respond but your plan to start with a factory fresh MB/CPU sounds like a plan.  Another would be to replace "3x Supermicro AOC-SASLP-MV8 SAS/SATA PCIe" from your signature with LSI based controllers like the IBM M1015s or the updated SuperMicro AOC-SAS2LP-MV8 if the MB/CPU replacement doesn't work well enough - or you find some controllers you can borrow first as a test.  The SASLP-MV8 is a slower controller then the SAS2LP-MV8 and the M1015 so that might make a difference in your response times.  But that is just my opinion and may not mean much.

 

I got the SASLP-MV8's for the "new" server since I had no problems with my "old" server, they were supported by UnRAID, and I didn't know there were any newer versions of MV8's.  Before resorting to getting a new mobo/CPU, I'll probably consider switching to the SAS2LP's; I can replace them one at a time to see if I see any marked performance improvements.

 

 

You don't mention what version you are running. Also some details like amount of RAM (yes, I can see 16GB in your signature, but you mention a new build).

http://lime-technology.com/forum/index.php?topic=27783.msg246595#msg246595

 

Yes, my sig of the "new" server is the target server in question that I'm experiencing issues with.  Running the most current 5.0.5.

[Auggie]

BTW, if I read the mobo specs correctly, it only has two x8 PCIe slots, with the other two being "x4 in x8 slots" so I'm not sure if having a third SAS2LP card will provide any performance benefit over the older SASLP card in that x4 slot...?

[BobPhoenix]

BTW, if I read the mobo specs correctly, it only has two x8 PCIe slots, with the other two being "x4 in x8 slots" so I'm not sure if having a third SAS2LP card will provide any performance benefit over the older SASLP card in that x4 slot...?

The SASLP-MV8 is a PCIe 1.0 card but the SAS2LP-MV8 is PCIe 2.0 so even though the x8 card would be running at half speed in a x4 slot it will be twice as fast with the 2.0 PCIe.  If possible you could try to reduce your drive count per SASLP-MV8 to 6 drives or less and see if that helps - without a purchase.  I've seen posts in other threads from those that know more that the SASLP-MV8 will saturate it's connection at about 6 drives per controller.  So if you can reduce your drive count per controller to that (temporarily at least) and see improvements then upgrading controllers may help.  If you don't see any improvement with 18 drives on 3 controllers then your problem is likely elsewhere.

[Auggie]

The SASLP-MV8 is a PCIe 1.0 card but the SAS2LP-MV8 is PCIe 2.0 so even though the x8 card would be running at half speed in a x4 slot it will be twice as fast with the 2.0 PCIe.  If possible you could try to reduce your drive count per SASLP-MV8 to 6 drives or less and see if that helps - without a purchase.  I've seen posts in other threads from those that know more that the SASLP-MV8 will saturate it's connection at about 6 drives per controller.  So if you can reduce your drive count per controller to that (temporarily at least) and see improvements then upgrading controllers may help.  If you don't see any improvement with 18 drives on 3 controllers then your problem is likely elsewhere.

 

I may give that a try, but my original server had it's lone SASLP maxxed out at 8 drives (plus 6 drives on the motherboard ports) and did not experience any of the perceived performance issues that I'm seeing on my new server.

 

At least it's good to know I will still see performance improvements on any SAS2LP card installed in a x4 slot.

[Auggie]

I will first replace a suspect SASLP card with a new SAS2LP card that I just ordered.

 

This card is never POSTed during boot up (displays port statuses and option to enter into its setup page), yet in all other respects, seems to work and all drives are accessible with no errors, SMART or otherwise.  I've pulled all other SASLPs and tried it in all four different PCI slots, but it never POSTs.

[Ford Prefect]

...here are some more things that spring to mind, that I'd try before buying a new set.

 

- is there a pattern when sluggish responses occur?...is it on write or read'?...on specific data, residing on specific disks?

- you added new disks...what models are these...could one or more of these add an effect (like beeing super-green, super-slow)?

- what about network equipment?...change cables and switch-ports...the mobo has two NICs..try the other

- disable all Advanced Power Management features in the BIOS...this could cause NICs and PCIe cards to go into sleep/wakeup cycles and sometimes even cause instability.

[Auggie]

- is there a pattern when sluggish responses occur?...is it on write or read'?...on specific data, residing on specific disks?

 

I've been trying to nail down a pattern.  These sluggishness episodes affect both read (extremely long delays in accessing a drive or user share which spans all drives, which may result in all server shares being summarily unmounted by OS X due to unresponsiveness, even though I can usually remount them just fine), as well as writes (extremely slow writes resulting in file transfer failures, but never leading to OS X unmounting the server shares).  If I suspect a specific drive and if I have a spare, I replace the drive or I try switching it to a different SASLP; I've even replaced a SASLP with another brand new one.  It may appear to resolve issues specific to that drive/HBA, but overall, the sluggishness continues.

 

- you added new disks...what models are these...could one or more of these add an effect (like beeing super-green, super-slow)?

 

When WD was absent in the 4TB market, I started adding Hitachi Deskstar 5K4000 drives; all drives <4TB are WD Greens, which were a mixture of 2 and 3TBs.  These same drives I did not experience performance issues when they were installed on the Atom setup, so I don't believe the types of drives are the culprit.  I expanded the new server with WD 2TB Greens that I had plenty on-hand from when I had gradually upgraded the Atom to larger 3 then 4 TB drives.  And I'm now slowly replacing those 2TBs with WD 4TB Greens, delaying as long as I can for the WD 5TBs to be released.

 

- what about network equipment?...change cables and switch-ports...the mobo has two NICs..try the other

 

I was told that unRAID will only recognize one of multiple mobo network interface ports and it's non-negotiable.  How unRAID determines which port is unclear (for mine it's port #2), but it will always be that same port.  Regarding switches and cabling, when I replaced the Atom server, I used the same cables and gigabit switches as I physically placed the new server in the same place; as I stated, I had no issues with the Atom server with the exact same network equipment while the new server had performance issues from the start.  I even have the Atom server now attached to the same switch with no issues.  However, I will replace the cable to the new server just in case...

 

- disable all Advanced Power Management features in the BIOS...this could cause NICs and PCIe cards to go into sleep/wakeup cycles and sometimes even cause instability.

 

I will try this.  I assume unRAID has sufficient power management features to negate the need for BIOS managed power management?  I would prefer to be as power-efficient as possible, but of course, will disable APM if need be.

 

 

I have been running the new SAS2LP for a week now, ensuring that the parity drive and the most accessed data drive (containing the YAMJ media library data files) is attached, and there is definitely a remarkable speed improvement of drives attached to it.  I still see the sluggish performance episodes, but so far, they appear to be on drives not attached to the SAS2LP (I access the data drives directly when moving/transferring media files).  I will continue observing the performance and if they continue to be as I've described, will get a second SAS2LP.

[Ford Prefect]

I've been trying to nail down a pattern.  These sluggishness episodes affect both read (extremely long delays in accessing a drive or user share which spans all drives, which may result in all server shares being summarily unmounted by OS X due to unresponsiveness, even though I can usually remount them just fine), as well as writes (extremely slow writes resulting in file transfer failures, but never leading to OS X unmounting the server shares).  If I suspect a specific drive and if I have a spare, I replace the drive or I try switching it to a different SASLP; I've even replaced a SASLP with another brand new one.  It may appear to resolve issues specific to that drive/HBA, but overall, the sluggishness continues.

 

[...]

 

When WD was absent in the 4TB market, I started adding Hitachi Deskstar 5K4000 drives; all drives <4TB are WD Greens, which were a mixture of 2 and 3TBs.  These same drives I did not experience performance issues when they were installed on the Atom setup, so I don't believe the types of drives are the culprit.  I expanded the new server with WD 2TB Greens that I had plenty on-hand from when I had gradually upgraded the Atom to larger 3 then 4 TB drives.  And I'm now slowly replacing those 2TBs with WD 4TB Greens, delaying as long as I can for the WD 5TBs to be released.

 

OK, looks like this path might be a dead end...as you are already investigating the change of HBA make/model, I'd wait what result come up from this activity.

I was told that unRAID will only recognize one of multiple mobo network interface ports and it's non-negotiable.  How unRAID determines which port is unclear (for mine it's port #2), but it will always be that same port.  Regarding switches and cabling, when I replaced the Atom server, I used the same cables and gigabit switches as I physically placed the new server in the same place; as I stated, I had no issues with the Atom server with the exact same network equipment while the new server had performance issues from the start.  I even have the Atom server now attached to the same switch with no issues.  However, I will replace the cable to the new server just in case...

True, that unRAID will only support one NIC but inside the config-files, from a CLI terminal, you can change the selection of it, AFAIR.

As far as cables are concerned...re-using a previously working equipment does not mean it will work forever...even a single port on a switch can go bad.

 

I will try this.  I assume unRAID has sufficient power management features to negate the need for BIOS managed power management?  I would prefer to be as power-efficient as possible, but of course, will disable APM if need be.

APM, AFAIR, is specifically for devices on the bus, like your HBAs and the NICs are. You will gain little power savings from it, if any, in a single server.

This is different from CPU frequency scaling and C-State management...you should enable this in BIOS in order to able to use these when idle.

Just disable APM...I have a previous model, a X8SIL-F, where enabling APM would render the NIC useless under load.

 

I have been running the new SAS2LP for a week now, ensuring that the parity drive and the most accessed data drive (containing the YAMJ media library data files) is attached, and there is definitely a remarkable speed improvement of drives attached to it.  I still see the sluggish performance episodes, but so far, they appear to be on drives not attached to the SAS2LP (I access the data drives directly when moving/transferring media files).  I will continue observing the performance and if they continue to be as I've described, will get a second SAS2LP.

This sounds like you are on the right path.

[Auggie]

The SASLP-MV8 is a PCIe 1.0 card but the SAS2LP-MV8 is PCIe 2.0 so even though the x8 card would be running at half speed in a x4 slot it will be twice as fast with the 2.0 PCIe.

 

I just found out about SAS expanders and am now looking at a Chenbro CK22803 28-port expander.  I'm thinking this will be much more efficient and faster alongside a single SAS2LP, versus going to three SAS2LP's where one will be in a x4 slot, yes?  I can even plug-in the Chenbro into the x4 slot since the card only uses the PCI bus for power, correct?

 

I'm really loving the noticeable speed improvements with the one SAS2LP and I'm about to pull the trigger on two more SAS2LP's but for $100 more for the Chenbro and if it will be much faster than having an HBA in the x4 slot, I'd rather go the Chenbro route.

[BobPhoenix]

The SASLP-MV8 is a PCIe 1.0 card but the SAS2LP-MV8 is PCIe 2.0 so even though the x8 card would be running at half speed in a x4 slot it will be twice as fast with the 2.0 PCIe.

 

I just found out about SAS expanders and am now looking at a Chenbro CK22803 28-port expander.  I'm thinking this will be much more efficient and faster alongside a single SAS2LP, versus going to three SAS2LP's where one will be in a x4 slot, yes?  I can even plug-in the Chenbro into the x4 slot since the card only uses the PCI bus for power, correct?

 

I'm really loving the noticeable speed improvements with the one SAS2LP and I'm about to pull the trigger on two more SAS2LP's but for $100 more for the Chenbro and if it will be much faster than having an HBA in the x4 slot, I'd rather go the Chenbro route.

I like SAS expanders but my reasons were different.  I wanted to save PCIe slots and a SAS expander worked great for that.  I get 60-130MB/s parity checks off it and I still have all but one PCIe slot available for other cards passed through to other VMs.

 

 

Will it be faster using a SAS expander - possibly.  There is a little lost speed with an expander but I don't know if it will match the loss on a x4 slot.  I would guestimate it at 10% if you had 8 drives on a SAS2LP in an x4 slot you may still be able to achieve full speed on all 8 spinners - you wouldn't with SSDs.

 

 

Not familiar with the Chenbro SAS expander but my Intel RES2SV240 expanders don't need to be plugged into a slot I am using a molex plug to power them so that I can use the PCIe slots for other PCIe cards.

[Auggie]

Not familiar with the Chenbro SAS expander but my Intel RES2SV240 expanders don't need to be plugged into a slot I am using a molex plug to power them so that I can use the PCIe slots for other PCIe cards.

 

The Chenbro's also don't necessarily need to be plugged into a PCI slot, but unlike the Intel's, it doesn't appear to come with a Molex power connector.  The Chenbro's do have external ports but I don't need that feature.  Since the Intel's appear to be cheaper, I think I will go with RES2V240 expander.  Does it come with a backplane bracket?  I don't foresee needing to add any other PCIe cards so mounting the SAS expander in the x4 slot will ensure it's securely mounted.

[BobPhoenix]

Not familiar with the Chenbro SAS expander but my Intel RES2SV240 expanders don't need to be plugged into a slot I am using a molex plug to power them so that I can use the PCIe slots for other PCIe cards.

 

The Chenbro's also don't necessarily need to be plugged into a PCI slot, but unlike the Intel's, it doesn't appear to come with a Molex power connector.  The Chenbro's do have external ports but I don't need that feature.  Since the Intel's appear to be cheaper, I think I will go with RES2V240 expander.  Does it come with a backplane bracket?  I don't foresee needing to add any other PCIe cards so mounting the SAS expander in the x4 slot will ensure it's securely mounted.

I'm fairly sure it does because I had mine mounted that way beyond my X9SCM MB - but now I have all of them sitting - with self adhesive rubber feet stuck to them - to the bottom of the case.  It also comes with long SFF-8087 to SFF-8087 cables to connect the expander to drive backplanes like you find in a Norco 4224 and two shorter ones to connect to your SAS2LP-MV8.

[UhClem]

Auggie,

 

Keep in mind that the total actual throughput is limited by the controller/HBA, and using an expander just spreads that limited throughput over more drives, resulting in a lower max speed (for a parity check).

 

I believe that the m1015 controller(s) that BobPhoenix uses have better throughput than the SAS2LP-MV8 (I don't want to hear how they are both PCIe x8v2, so they both have 4GB/s--crap!!), so be careful in extrapolating his results to your situation. I don't know how many drives BobP is pushing through his m1015+expander set-up(s), but it is possible/likely that even his better throughput is still a bottleneck (for him)--but, since he is slot-limited, he has no choice.

 

You don't have that limitation--so, why constrain yourself with that "solution"? Unless you are OK with slower-than-need-be parity-checks ...

 

--UhClem

 

[BobPhoenix]

Auggie,

 

Keep in mind that the total actual throughput is limited by the controller/HBA, and using an expander just spreads that limited throughput over more drives, resulting in a lower max speed (for a parity check).

 

I believe that the m1015 controller(s) that BobPhoenix uses have better throughput than the SAS2LP-MV8 (I don't want to hear how they are both PCIe x8v2, so they both have 4GB/s--crap!!), so be careful in extrapolating his results to your situation. I don't know how many drives BobP is pushing through his m1015+expander set-up(s), but it is possible/likely that even his better throughput is still a bottleneck (for him)--but, since he is slot-limited, he has no choice.

 

You don't have that limitation--so, why constrain yourself with that "solution"? Unless you are OK with slower-than-need-be parity-checks ...

 

--UhClem

Maybe you can answer the question he had.  How much would a fully loaded x8 card plugged into a x4 electrical PCIe slot loose in speed?  I suspect it won't be as much as the SAS expander if it has ANY loss but for me even with 13-15 3TB drives I got 130MB/s MAX on the outside dropping to about 60 on the inner.  I now have 19 drives with 8 4TB drives and I just kicked off a check since I don't have current stats.  Actually I found out why I hadn't taken a current stat.  I have a WD Black as parity and the 800GB platters really slow down the checks < 100MB/s at the current position (6GB into check).  Will have to see what it is when I replace the Black with 1TB platter Red - hoping it will be closer to my record of 130MB/s on the outer tracks.

 

 

OK a found a more concrete example of my times with an expander.  I have a 14 Data 1 Parity array of ST3000DM001 drives (3TB) that completed a NOCORRECT parity check in 29888sec (Apr 21 04:34:26 Media1 kernel: md: sync done. time=29888sec (unRAID engine)).  So is that really 100MB/s average on the whole thing?  Because my math seems off to me - I was expecting an average less than that.  Anyway when someone else specifies the correct number this is what I got with my SAS Expander and an M1015 controller to give you a better idea of speeds with SAS expander.

[Auggie]

Keep in mind that the total actual throughput is limited by the controller/HBA, and using an expander just spreads that limited throughput over more drives, resulting in a lower max speed (for a parity check).

 

Yes, that has always been in the back of mind: 24 drives going through one PCIe x8 HBA/slot versus spread across two PCIe x8 HBAs/slots (16 drives) and one PCIe x4 HBA/slot (8 drives).

 

An RES2SV240 can be had for $245+tax (free shipping) from Staples versus, a pair of SAS2LP's for $210 shipped from SuperBiiz.

 

Decisions, decisions...

 

EDIT: Found a 15% SuperBiiz coupon bringing to the total shipped to $196 so I went the three SAS2LP route.  I still will eventually get the RES2SV240 but for my other unRAID Atom-based X7SPA-HF mobo since it only has once PCIe slot.  I may try a parity check speed test between the three SAS2LPs and a single SAS2LP with RES2SV240 to get the definitive answer to this question of performance hits between the x4 slot versus SAS expander...

[UhClem]

...

OK a found a more concrete example of my times with an expander.  I have a 14 Data 1 Parity array of ST3000DM001 drives (3TB) that completed a NOCORRECT parity check in 29888sec (Apr 21 04:34:26 Media1 kernel: md: sync done. time=29888sec (unRAID engine)).  So is that really 100MB/s average on the whole thing?  Because my math seems off to me - I was expecting an average less than that.

Well, you are definitely saturating the throughput limits of your M1015. Consider: Those drives (ST3000DM001) do an average of 150 MB/s (200max/100min). If they were not hampered by controller limits, you would have completed in 20000-21000 secs.

 

A simple, but effective, improvement you can make is to move 4-6 drives onto your onboard SATA ports. It shouldn't be necessary to run a complete parity-check--just start it, and note the speed for the first 5 minutes. Do that before and after the above drive migration.

 

--UhClem

 

[UhClem]

Decisions, decisions...

Auggie,

 

Using 3 x SAS2LP is overkill. You can get the same performance [with less added expense] (on your mobo) by using 2xSAS2LP + 1xSASLP + 4xSATA2 mobo ports; only put 4 drives on the SASLP. You might also get additional improvement by only putting 7 drives on each SAS2LP, and using the other 2 (SATA3) mobo ports.

 

As I mentioned to BobP above, comparing the observed speeds of the initial 5 minutes of a parity-check is sufficient to choose an optimal confuguration.

 

--UhClem

 

[BobPhoenix]

Well, you are definitely saturating the throughput limits of your M1015. Consider: Those drives (ST3000DM001) do an average of 150 MB/s (200max/100min). If they were not hampered by controller limits, you would have completed in 20000-21000 secs.

 

A simple, but effective, improvement you can make is to move 4-6 drives onto your onboard SATA ports. It shouldn't be necessary to run a complete parity-check--just start it, and note the speed for the first 5 minutes. Do that before and after the above drive migration.

 

--UhClem

I could but that box is currently running ESXi with an unRAID VM so my onboard ports are limited to ESXi datastores.  I could do that if I booted with unRAID and not ESXi but the Windows VM on that ESXi server is recording something (Satellite/OTA) every day so I would have to rearrange the recording schedule to block out an hour or two to give me a cushion to reconnect back like I currently have it.  May do that in the future when I have my current expansion project done on my other ESXi boxes however.

[Auggie]

Auggie,

 

Using 3 x SAS2LP is overkill. You can get the same performance [with less added expense] (on your mobo) by using 2xSAS2LP + 1xSASLP + 4xSATA2 mobo ports; only put 4 drives on the SASLP. You might also get additional improvement by only putting 7 drives on each SAS2LP, and using the other 2 (SATA3) mobo ports.

 

The SAS2LP in the 4x slot perhaps may be overkill, but at only $85 it's not much more of an expense over having to get a single Norco SFF-8087 Reverse Breakout Cable at $30 shipped, or two for $60 if I wanted to utilize all six mobo SATA ports.

 

I can always do the SASLP versus SAS2LP in the x4 slot speed test to see if there really is no appreciable speed increase.

[UhClem]

The SAS2LP in the 4x slot perhaps may be overkill, but at only $85 it's not much more of an expense over having to get a single Norco SFF-8087 Reverse Breakout Cable at $30 shipped, or two for $60 if I wanted to utilize all six mobo SATA ports.

Now that I've familiarized myself with your rack's drive connectivity, I agree.

I can always do the SASLP versus SAS2LP in the x4 slot speed test to see if there really is no appreciable speed increase.

Given the infeasibility of using the mobo ports, don't waste your time testing with the SASLP (with 8 drives connected). The operative shortcoming of the SASLP is not that it is (only) x4, but that its x4 is only PCIe V1.

 

In fact, it would not surprise me if the SAS2LP (PCIe x8 v2) in the x4 slot  (phys x8) performed just as fast as in a x8 slot.

 

--UhClem