New Replacement Drive = SLOWER speeds

[KermitJr]

Howdy all! I had:

8tb Parity

8tb Disk1

8tb Disk2

4tb Disk3

4tb Disk4

All running on same LSI SAS2308. EDIT: Balanced at 3/2 for the two SAS-to-SATA splitter ports.

Speed: 205MB/s+

 

Upgraded Parity and moved to data:

14tb Parity

8tb Disk1

8tb Disk2

8tb Disk3 (old parity drive)

4tb Disk4

Now, I did put the Parity is in a new slot, but still on same HBA, I think. EDIT: Now split at 4/1 for the two ports.

New Speed: 110MB/s consistently.

 

Attaching diag if anyone can help or has any ideas.  Mobo is an MSI K9A2 Platinum with SataII 3Gbps. The LSI SAS2308 is a SataIII 6Gbps.

 

The reason I'm confused - both configs have 5 drives on the HBA - so why a drastic speed difference?

 

Appreciate any help in diagnosing. I'll have physical access tomorrow - so what would you move/check first? Cable?

 

EDIT: Okay, asked family member to get some close up pics of cabling. The two 4TB drives are on cables 1&4 of the HBA controller (single side) the other disks (14,8,8,8) are ALL on cable 5-8. So the shift is instead of a 3/2 balance, it's now 4/1 for the two ports on the HBA. Could this really account for a near 50% drop in speed? that's going to be the first think I adjust.

 

 

Only SMART issue I see with 8TB moved from Parity to Disk3 is "199    UDMA CRC error count    021" but that hasn't come back since clearing yesterday and I've been running parity for 12hrs.

 

Thanks!

KermitJr

 

 

 

[0:0:0:0]    disk    Sony     Storage Media    0100  /dev/sda   /dev/sg0 
  state=running queue_depth=1 scsi_level=0 type=0 device_blocked=0 timeout=30
  dir: /sys/bus/scsi/devices/0:0:0:0  [/sys/devices/pci0000:00/0000:00:13.5/usb1/1-4/1-4:1.0/host0/target0:0:0/0:0:0:0]
[6:0:0:0]    disk    ATA      ST4000VX007-2DT1 CV11  /dev/sdb   /dev/sg1 
  state=running queue_depth=1 scsi_level=7 type=0 device_blocked=0 timeout=30
  dir: /sys/bus/scsi/devices/6:0:0:0  [/sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/host6/port-6:0/end_device-6:0/target6:0:0/6:0:0:0]
[6:0:1:0]    disk    ATA      Samsung SSD 860  1B6Q  /dev/sdc   /dev/sg2 
  state=running queue_depth=32 scsi_level=7 type=0 device_blocked=0 timeout=30
  dir: /sys/bus/scsi/devices/6:0:1:0  [/sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/host6/port-6:1/end_device-6:1/target6:0:1/6:0:1:0]
[6:0:2:0]    disk    ATA      ST4000VX007-2DT1 CV11  /dev/sdd   /dev/sg3 
  state=running queue_depth=1 scsi_level=7 type=0 device_blocked=0 timeout=30
  dir: /sys/bus/scsi/devices/6:0:2:0  [/sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/host6/port-6:2/end_device-6:2/target6:0:2/6:0:2:0]
[6:0:3:0]    disk    ATA      WDC WD80EMAZ-00W 0A83  /dev/sde   /dev/sg4 
  state=running queue_depth=1 scsi_level=7 type=0 device_blocked=0 timeout=30
  dir: /sys/bus/scsi/devices/6:0:3:0  [/sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/host6/port-6:3/end_device-6:3/target6:0:3/6:0:3:0]
[6:0:4:0]    disk    ATA      WDC WD80EMAZ-00W 0A83  /dev/sdf   /dev/sg5 
  state=running queue_depth=1 scsi_level=7 type=0 device_blocked=0 timeout=30
  dir: /sys/bus/scsi/devices/6:0:4:0  [/sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/host6/port-6:4/end_device-6:4/target6:0:4/6:0:4:0]
[6:0:5:0]    disk    ATA      WDC WD140EDGZ-11 0A85  /dev/sdg   /dev/sg6 
  state=running queue_depth=1 scsi_level=7 type=0 device_blocked=0 timeout=30
  dir: /sys/bus/scsi/devices/6:0:5:0  [/sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/host6/port-6:5/end_device-6:5/target6:0:5/6:0:5:0]
[6:0:6:0]    disk    ATA      WDC WD80EMAZ-00W 0A83  /dev/sdh   /dev/sg7 
  state=running queue_depth=1 scsi_level=7 type=0 device_blocked=0 timeout=30
  dir: /sys/bus/scsi/devices/6:0:6:0  [/sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/host6/port-6:6/end_device-6:6/target6:0:6/6:0:6:0]
NVMe module may not be loaded

 

skynet.local-diagnostics-20221201-1357.zip

Edited December 1, 2022 by KermitJr

[Vr2Io]

7 hours ago, KermitJr said:

New Speed: 110MB/s consistently

You mean parity check speed ?

 

7 hours ago, KermitJr said:

Could this really account for a near 50% drop in speed? that's going to be the first think I adjust.

It doesn't matter how disk are connect in LSI HBA ports.

 

7 hours ago, KermitJr said:

nly SMART issue I see with 8TB moved from Parity to Disk3 is "199    UDMA CRC error count    021" but that hasn't come back since clearing yesterday and I've been running parity for 12hrs.

This was happen in long time ago.

 

Error 21 [0] occurred at disk power-on lifetime: 224 hours (9 days + 8 hours)
  When the command that caused the error occurred, the device was active or idle.

 

  9 Power_On_Hours          -O--C-   097   097   000    -    24564

 

Anyway no cause found.

 

[KermitJr]

@Vr2Io, are you the only one who loves me? 🙂 (thank you for engaging!)

 

Yes, I mean parity check speed. Literally cut in half with the above changes.

 

I just logged in to check status of the thread, but first thing this morning, I put the new 8tb disk3 (old parity) on the "other side" of the HBA to rebalance it at 3/2 and parity check speeds are consistently at 190MB/s (which is normal until it gets past that remaining 4TB disk4). I suspect that it will then increase to the 210+ range.

 

So while I understand "how disks are connected to the LSI HBA ports" shouldn't matter - it appears that it actually might in this circumstance?  Again, I'm simply reporting realtime. Speeds might tank later into the parity check. 

 

Regardless, I'll follow up.  And I thank you for your continued engagement!

 

EDIT: Okay, it's at 3.75GB on parity check (still hitting that 4GB) and down to 100MB/s. Ugh. Temps are running 46 but that's shouldn't be an issue. I paused check to let the cool down and resuming put it at same slow speed. Will update soon.

 

EDIT2: So once it made it past the 4TB drives, the speed is still around 140-150 MB/s, which is a 25% reduction in speed. Really odd but I'll set up the new server soon and troubleshoot from there.

Edited December 3, 2022 by KermitJr

[Vr2Io]

Problem usually cause by age disk or some disk at specific region got abnormally slow. Diskspeed docker may help you identify that.

Edited December 3, 2022 by Vr2Io