Ive recently upgraded my server to use 2 9206-16e drives, flashed with the latest firmware.  Ive got 2 systems that have this setup and both are experiencing the same issues. One system has large disks, most around 8TB, the one has only 2TB disks.  When a parity check is issued, the parity disk will throw read errors.  I am really at a loss.  Attached are diagnostics.

 

Previously I had the 9201-16E card in without issues. I would have kept the card but I needed to upgrade to a half-height card due to server only having 1 full height and 1 half height PCI-E slot.

unraid-backup-diagnostics-20200101-0823.zip

Edited January 1, 20206 yr by icemansid

Jan  1 04:42:52 UnRAID-Backup kernel: mpt2sas_cm0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
### [PREVIOUS LINE REPEATED 1 TIMES] ###
Jan  1 04:42:52 UnRAID-Backup kernel: sd 4:0:1:0: Power-on or device reset occurred
Jan  1 04:42:52 UnRAID-Backup kernel: mpt2sas_cm0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
### [PREVIOUS LINE REPEATED 2 TIMES] ###
Jan  1 04:42:52 UnRAID-Backup kernel: sd 4:0:1:0: Power-on or device reset occurred
Jan  1 04:42:53 UnRAID-Backup kernel: mpt2sas_cm0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
### [PREVIOUS LINE REPEATED 3 TIMES] ###
Jan  1 04:42:53 UnRAID-Backup kernel: sd 4:0:1:0: Power-on or device reset occurred
Jan  1 04:42:53 UnRAID-Backup kernel: mpt2sas_cm0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
### [PREVIOUS LINE REPEATED 3 TIMES] ###
Jan  1 04:42:53 UnRAID-Backup kernel: sd 4:0:1:0: Power-on or device reset occurred
Jan  1 04:42:54 UnRAID-Backup kernel: mpt2sas_cm0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)

 

 

This looks more a like a power/connection issue, try replacing/swapping cables/backplanes/PSU

 

 

So you connect disks direct to HBA external port ? SATA link should limit to as short as possible (how long of the cable ?) and you have a new set cable for SFF8644.

Edited January 2, 20206 yr by Benson

The external cables are .5 meter and the internal cables are .5 meter as well - about as short as i can get it.  HBA external to another chassis with a  Dual Mini SAS 26pin SFF-8088 to 36pin SFF-8087 Adapter.  All of the SFF8644 cables are new as well.

Ordered replacement cables. After a bit of digging, i found a number of negative reviews on the specific cables I ordered.

Seems you run in SATA protocol whole path in those cable ( 8088, 8087, 8644 ), if true, you must change the design.

 

I connect HBA (2 port) to external chassis by two 2 meter cable,  then input to a expander and fan out to 12 disks.

 

So HBA to Expander was SAS protocol and last mile was SATA protocol. Never make SATA run too long.

Edited January 2, 20206 yr by Benson

Benson is correct, unless you're using a SAS expander max total cable length from HBA to SATA disks is 1 meter, any longer and you'll have issues.

 

 

21 minutes ago, johnnie.black said:

Benson is correct, unless you're using a SAS expander max total cable length from HBA to SATA disks is 1 meter, any longer and you'll have issues.

 

 

Interesting.  I'm using 3 x SFF8088 -> 4 Sata from a 9201-16e at 1.5m with no problems.  I guess YMMV

5 minutes ago, Squid said:

I guess YMMV

Yes, 1m is the max supported length, and you should avoid going beyond that, but it will depend on cable quality, etc.

3 hours ago, Squid said:

Interesting.  I'm using 3 x SFF8088 -> 4 Sata from a 9201-16e at 1.5m with no problems.  I guess YMMV

My old setup was similar - no issues.  Will see how the new cables far.  Right now they are as short as they can be.

3 hours ago, Benson said:

Seems you run in SATA protocol whole path in those cable ( 8088, 8087, 8644 ), if true, you must change the design.

 

I connect HBA (2 port) to external chassis by two 2 meter cable,  then input to a expander and fan out to 12 disks.

 

So HBA to Expander was SAS protocol and last mile was SATA protocol. Never make SATA run too long.

Can you expand on what SAS expander you are using and is there any IO limitation by accessing 12 disks over a single SAS cable?  Also what HBA are you using for this?

11 minutes ago, icemansid said:

Can you expand on what SAS expander you are using and is there any IO limitation by accessing 12 disks over a single SAS cable?  Also what HBA are you using for this?

You may ref below post

 

One link may have some performance hit for 12disks, SAS2, 6Gx4 ~ 2GB/s bandwidth, so each disk got 166MB/s

 

HBA was 9211-8i, the 2 meter cable was a 8087 to 8087.

Edited January 2, 20206 yr by Benson

Many thanks!

Quick update - the replacement cables are in and so far, issues seem to have been resolved, even though I am over the 1M total length spec. Currently at 1.5M but running a parity check and no errors. I also ordered a SAS expander so will be implementing that later this week as well.

Update #2 - for some reason, single disk speeds are normal, but when reading from all disks, speeds are slow.  This was apparent with a parity check and using the DiskSpeed container and benchmarking all disks on the controller. By all accounts, the new SAS card should be much faster than the old SAS card.

Check if you have write cache enabled on your drives:

hdparm -W 1 /dev/sdX   -> enables write cache

hdparm -W 0 /dev/sdX  -> disables write cache

 

Do this for all your drives, maybe that solves that problem, could be related to their issues:

https://forums.unraid.net/topic/79966-enable-write-cache/

https://forums.unraid.net/topic/80074-sata-parity-write-cache-disabled/page/2/

 

Edited January 7, 20206 yr by RedReddington

23 minutes ago, icemansid said:

but when reading from all disks, speeds are slow. 

Check HBA link speed/width, I believe diskspeed shows current one, if not you can check with lspci -vv

37 minutes ago, RedReddington said:

Check if you have write cache enabled on your drives:

hdparm -W 1 /dev/sdX   -> enables write cache

hdparm -W 0 /dev/sdX  -> disables write cache

 

Do this for all your drives, maybe that solves that problem, could be related to their issues:

https://forums.unraid.net/topic/79966-enable-write-cache/

https://forums.unraid.net/topic/80074-sata-parity-write-cache-disabled/page/2/

 

No real change in multi-drive reads.  It does appear to have increased the single drive speed reads though.  Also I want to mention - the ONLY thing that I changed was the SAS card and cables. 

40 minutes ago, johnnie.black said:

Check HBA link speed/width, I believe diskspeed shows current one, if not you can check with lspci -vv

Ive added that to the image.  Comparing that to my other system with H310 cards installed, this one should be faster.

 

I am not family with disk speed docker, but seems abnormal of simultaneous bandwidth only 347MB/s.

5 minutes ago, icemansid said:

Comparing that to my other system with H310 cards installed, this one should be faster.

It wounld't be faster since 8 disks won't use all the available bandwidth on a Perc H310, but it shouldn't be slower, strange combined speed being so low, if you want post the diags, might be something visible there.

5 minutes ago, johnnie.black said:

It wounld't be faster since 8 disks won't use all the available bandwidth on a Perc H310, but it shouldn't be slower, strange combined speed being so low, if you want post the diags, might be something visible there.

diags are at the top of this post - haven't rebooted since but i can pull a new one if you like.  There are a ton of drive errors which have been resolved due to bad SAS cables but speeds are still way off from previous SAS card.  The H310 shows max throughput of only 4 GB/s compared to the 6GB/s on the 2308.

11 minutes ago, Benson said:

I am not family with disk speed docker, but seems abnormal of simultaneous bandwidth only 347MB/s.

Here is the same benchmark from the H310 - technically slower card - though it has much better/faster drives attached to it. Single disk reads are nearly identical to multi-disk reads. This is what I would expect the 2308 card to perform like.

12 minutes ago, icemansid said:

Here is the same benchmark from the H310 - technically slower card - though it has much better/faster drives attached to it. Single disk reads are nearly identical to multi-disk reads. This is what I would expect the 2308 card to perform like.

I know, I haven't speed bottleneck for 2008 and 2308 chip to connect 12 and 16 disks which have same result as above.

 

9206-16e have two 2308 chips, I suppose it should have a PCIe switch on HBA in order to connect two chips, there may be some compatibility issue with mother board.

 

Or could you check relate overheat on HBA ?? But I am not sure HBA would or wont throttle because I never see that.

Edited January 7, 20206 yr by Benson

28 minutes ago, Benson said:

I know, I haven't speed bottleneck for 2008 and 2308 chip to connect 12 and 16 disks which have same result as above.

 

9206-16e have two 2308 chips, I suppose it should have a PCIe switch on HBA in order to connect two chips, there may be some compatibility issue with mother board.

 

Or could you check relate overheat on HBA ?? But I am not sure HBA would or wont throttle because I never see that.

It has 4 ports on it - 2 connect to 1 chip, two connect to the other.

 

Also on the over-heating - its unlikely as this card is in a 1U HP enterprise server. It has enough fan power to take flight.

Edited January 7, 20206 yr by icemansid
more detials

SAS expander in and no change.  What I did come across is someones comment about is that old drives connected to this controller will cause drive speed issues across all of the drives. I think i will do some more testing as I am using a bunch of very old disks in this setup.