(SOLVED) SAS Writes slowing down, but SATA is wire-speed

[JudyZ]

This is my third Unraid server, triggered by some affordable Seagate Constellation ES.3 4TB drives.

Will be 9x ES.3 SAS 4TB drives + WD40EZRZ SATA 4TB.

 

I have 30TB to load so, for speed, am adding and filling one drive at a time.

I'll add and calculate parity at the end (the source files are backed up so ok with that).

No cache in use.

 

Copying to any SAS drive starts at 110MB/s, then after 20 seconds, drops to 40MB/s

Copying to a SATA drive runs continuously at 113MB/s

What's causing the drop in speed on the SAS drives?

 

All drives preclear fine at 170MB/s.

The SAS drives are connected to an LSI 9220-8i HBA just flashed to IT mode in a Pci-e 2.0 x16 slot, straight to the cpu.

Cables are two SFF-8087 to 4x SFF-8482 shielded cables

SATA drive is connected to a SATA 6Gb/s port on the chipset.

Mobo is Gigabyte GA-Z68AP-D3 Rev 2, cpu is Pentium dual core 2.6Ghz, 8GB RAM

Fresh install of Unraid Plus 6.7.2

Source machine for copies is Windows 8.1 Pro - I imagine using SMB shares

Copies are just Windows explorer copy/paste

I attach full diags (SAS drive is unassigned at present)

 

Many thanks, Judy

 

tower-diagnostics-20190916-2135.zip

[Vr2Io]

Should be due to SAS disk writeback cache in disable

 

• === START OF INFORMATION SECTION ===
• Vendor:               SEAGATE
• Product:              ST4000NM0023
• Revision:             0004
• Compliance:           SPC-4
• User Capacity:        4,000,787,030,016 bytes [4.00 TB]
• Logical block size:   512 bytes
• LU is fully provisioned
• Rotation Rate:        7200 rpm
• Form Factor:          3.5 inches
• Logical Unit id:      0x5000c5008405bcdf
• Serial number:        Z1Z99X490000C604BMPF
• Device type:          disk
• Transport protocol:   SAS (SPL-3)
• Local Time is:        Mon Sep 16 14:35:53 2019 PDT
• SMART support is:     Available - device has SMART capability.
• SMART support is:     Enabled
• Temperature Warning:  Enabled
• Read Cache is:        Enabled
• Writeback Cache is:   Disabled

[dalben]

Is it worth trying to load up Unraid 6.6.7 to see if there is a difference. 6.7 seems to have some weird performance issues. 

[JudyZ]

Thank you. So the SAS disk write cache is disabled.

I checked/turned on/checked the SAS disk write cache using:

sdparm /dev/sdc --get=WCE

sdparm /dev/sdc --set=WCE

sdparm /dev/sdc --get=WCE

 

Copies to SAS are now running at 80MB/s [Windows Explorer, from a ZFS machine  to Unraid, so will pribably go to 110MB/s for pure Windows to Unraid.

 

Some questions please:

Does preclear switch on disk write cache to  achieve 170MB/s on the SSA drive?

What is turning off SAS disk write cache? Is it the LSI 9220-8i flashed to IT mode or just Unraid SAS default?

Are all unraid users with flashed SAS HBAs getting just 40MB/s - and that's without parity writing?

 

SATA drives appear to default to disk write caching enabled in Unraid, so am I safe to leave SAS disk write caching on?
 

Edited September 17, 2019 by JudyZ
sdparm - doh!

[JorgeB]

3 minutes ago, JudyZ said:

hdparm

You need to use sdparm for SAS devices, see here.

[Vr2Io]

1 hour ago, JudyZ said:

What is turning off SAS disk write cache?

Disk default

 

1 hour ago, JudyZ said:

Are all unraid users with flashed SAS HBAs getting just 40MB/s - and that's without parity writing?

In general, not HBA related

 

1 hour ago, JudyZ said:

SATA drives appear to default to disk write caching enabled in Unraid, so am I safe to leave SAS disk write caching on?

Some SATA disk also disable in default, this just against suddenly poweroff and data still in disk cache.

Edited September 17, 2019 by Benson

[Vr2Io]

1 hour ago, JudyZ said:

Does preclear switch on disk write cache to  achieve 170MB/s on the SSA drive?

Interesting, may be due to the data pattern was zero.

Edited September 17, 2019 by Benson

[JudyZ]

Thank you all for the replies and help. 

A big factor in using unraid for the last 8 years has been the exceptional support on this forum, and 6.7.2 is fab

 

Just to recap and mark this thread as solved (how?)

 

1. This has sped up initial load to SAS drives *without* parity from 40MB/s to 100MB/s 

(ZFS source, large films, using Windows 8.1 Pro Explorer drag and drop on a third machine)

2. if your drives are writing unusually slowly, try checking if the disk cache is enabled.

For SAS drives, type   sdparm /dev/sdc --get=WCE 

For SATA drives, type    hdparm -W /dev/sdc

 (your device will differ - sdb, sdd etc.)

3. drives may (or not) have disk write cache enabled by default

4. the disk cache is completely distinct from the write cache on a host bus or RAID adapter

5. the commands to enable write caching are:

For SAS, type    sdparm /dev/sdc --set=WCE

For SATA, type    hdparm -W1 /dev/sdc

6. the disk write cache setting on my drives (Seagate Constellation ES.3 SAS) persists across reboots and power downs, so there is no need to script it at boot.

 

***** NB My knowledge of Unraid is small, so I can't say if this is a good idea. It has helped me during initial load of 30TB.

If the power drops before the disk has flushed its cache, the data will be lost - but I'd just rewrite the entire file if that happened.

I might change it back after adding a parity drive.

 

I'd be grateful for any opinions on this.

 

Thanks again, Judy