Hi folks,

I'm trying to get to the bottom of a write performance issue on a new SAS drive.  I've exhausted all of the troubleshooting steps I can find on the internet.  I hoping that someone here might be able to give me some additional ideas.

 

The problem: I added a Seagate 10 TB 4Kn SAS drive (ST10000NM0226) to my system and initially ran a pre-clear using the Preclear Plugin to stress test the drive.  The drive ran well with write speeds when zeroing of 115-250 MB/s.  I then added the drive as an additional data drive in my array and Unraid wanted to run a disk-clear which it started, but the write speed doesn't get above about 50 MB/s.

 

Interestingly, if I cancel Unraid's disk clear and try to run another pre-clear zeroing pass, the Preclear Plugin will only get the same 50 MB/s.  If I reboot my Unraid box, the Preclear Plugin will once again run at ~250 MB/s until Unraid's disk clear accesses the drive - and then write performance maxes out at ~50 MB/s.

 

Things I've tried so far:

- Upgraded LSI SAS2008 SAS controller firmware from 19.00.00 to 20.00.07.

- Shut down docker and VMs.

- Booted in safe mode.

- Moved the drive in question to a different SAS connector.

- Confirmed the SAS controller is connected to the PCIe bus at x8.

- Confirmed no obvious errors appearing in syslog

 

This is the first SAS drive and first 4K native logical sector size drive in my array.  The only thing that I can think of now is that the 4K sector size of the new SAS drive mixed with my other (SATA) drives which are 512 logical sector size might be causing the issue.  To test this, I'm thinking about making a new Unraid flash and bring it up with just the one SAS drive (assuming it will come up with one drive) and see if Unraid's disk clear is similarly slow without the 512 byte logical sector size drives.

 

Additional troubleshooting: I connected an old 3TB SATA drive with a 512 sector size to the SAS controller along with the 10 TB SAS disk.  When I added only the 3 TB disk to the array and Unraid cleared the disk, the write speed was a health ~200 MB/s.  I then cancelled the disk clear and added both the 3 TB and 10 TB disk to the array.  When the disk clear started, both disks started clearing at 50 MB/s.  It is looking more and more that joining a 4Kn sector drive to an array with 512 sectors causes the clearing speed to be very slow.  Time will tell whether once the clearing finishes, if the drive will then allow writes at full speed.

 

Any thoughts would be greatly appreciated!

 

John

 

filer-diagnostics-20230218-1634.zip

Edited February 19, 20233 yr by John K
Added additional troubleshooting.

13 hours ago, John K said:

The only thing that I can think of now is that the 4K sector size of the new SAS drive mixed with my other (SATA) drives which are 512 logical sector size might be causing the issue.

IIRC the md driver uses 512K for all, so having 512K and 4KN together should not be an issue, what could be an issue is the drive performing much worse with 512K sector size, post the output of

fdisk -l /dev/md4

 

Thanks very much, @JorgeB.  It looks like I need to wait for the disk clear to finish in a couple of days before I can give the output of fdisk.  Below is the output while the disk clear is in progress.  (BTW, I moved the drive to slot 5 during other troubleshooting attempts).  I will follow-up once it is done.

root@Filer:/dev# fdisk -l /dev/md5
fdisk: cannot open /dev/md5: Input/output error

 

In case it helps, as another test, I added the 4Kn drive as my second parity drive to see what the parity build speed would be.  It similarly ran at the degraded ~50 MB/s.

The drive finished clearing and formatting.   Below is the requested fdisk -l.

 

root@Filer:/mnt/disk5# fdisk -l /dev/md5
Disk /dev/md5: 9.1 TiB, 10000831295488 bytes, 19532873624 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

 

To test if performance is normal after the format is finished, I ran dd if=/dev/zero of=/mnt/disk5/test1.img bs=1G count=2 oflag=dsync.  As shown in the screenshot below, the 4Kn SAS drive seems to be still maxing out at 50 MB/s.   Any thoughts are much appreciated.

 

 

Try with turbo write instead.

The disk write speed is unchanged.  But one odd thing (I think) is that the disk write speed of my new disk (Disk 5) causes all of my other data disks to read at the same speed.  Perhaps this is part of the parity calculation process when parity write is enabled?  I don't think I've seen that before.

 

root@Filer:/# dd if=/dev/zero of=/mnt/disk5/test1.img bs=1G count=2 oflag=dsync
2+0 records in
2+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 40.7871 s, 52.7 MB/s

 

10 minutes ago, John K said:

I don't think I've seen that before.

That's from turbo write, that disk appears to really be limited to 50MB/s, if it's not with preclear the issue might be the md driver using 512K sectors, unfortunately not much you can do for now.

21 minutes ago, JorgeB said:

the issue might be the md driver using 512K sectors

Are there steps that you can recommend to get confident that the issue is caused by the 512K sectors on a 4Kn drive?  If this is causing the significant write performance decrease, I think the community would benefit from this being broadly known.  With 4Kn drives becoming more and more common, people my want to avoid them.  (I know I would have.)  Thanks very much for your continued help on this.

9 minutes ago, John K said:

Are there steps that you can recommend to get confident that the issue is caused by the 512K sectors on a 4Kn drive?

Assign the drive to a cache pool and re-test, pools don't use the md driver.

10 minutes ago, John K said:

With 4Kn drives becoming more and more common, people my want to avoid them.

It's not an issue with 4Kn drives in general, I've seen other models being used without issues, it might be a problem with that model.

2 hours ago, JorgeB said:

Assign the drive to a cache pool

I fear that I didn't understand how to do this and have now hosed up my cache disk.  I tried to add the disk to my cache pool, but when I did this and started the array, both disks showed "Unmountable: No pool uuid".  Worried that I did something wrong, I tried to reverse my steps, removed the second disk from my cache pool, and set it down to a size of 1 drive.  My one original cache disk continued to show No pool uuid.  I then stopped the array and rebooted.  After the reboot, the one original drive now shows "Unmountable: Wrong or no file system" as shown below.

 

 

 

Syslog shows the following:

 

Feb 23 12:12:02 Filer  emhttpd: shcmd (718): mount -t btrfs -o noatime,space_cache=v2 /dev/nvme0n1p1 /mnt/cache
Feb 23 12:12:02 Filer root: mount: /mnt/cache: wrong fs type, bad option, bad superblock on /dev/nvme0n1p1, missing codepage or helper program, or other error.

 

After searching other posts, I tried to restore my superblock from the backup.

 

root@Filer:/dev# btrfs-select-super -s 1 /dev/nvme0n1p1
No valid Btrfs found on /dev/nvme0n1p1
ERROR: open ctree failed

 

Is there anything else I can try as I would really like to recover the data if I can.  Note that I never issued a format on the cache disk, so I'm hoping that there is still some hope.

 

Thanks.

Disk should be assigned as a new pool, possibly your previous pool was xfs, post the output of

blkid

 

4 minutes ago, JorgeB said:

Disk should be assigned as a new pool

A hard lesson that I won't soon forget!  🙂   You may be on to something with xfs.

 

root@Filer:/dev# blkid
/dev/sda1: LABEL_FATBOOT="UNRAID" LABEL="UNRAID" UUID="272B-4CE1" BLOCK_SIZE="512" TYPE="vfat"
/dev/loop1: TYPE="squashfs"
/dev/sdf1: UUID="0913c306-1a66-45c8-b8c5-9bd222024830" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="6af48176-ab7d-4256-a4c3-0cc7b2e1571a"
/dev/nvme0n1p1: UUID="6aab72e1-6b1e-4d71-841b-198c562e1d64" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="a363d4e2-01"
/dev/sdd1: UUID="58cb496c-f9dd-481f-9759-4bb2253c004a" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="71ca7a66-f8cb-4a1c-ae4e-d081cbf4fdb6"
/dev/sdb1: UUID="bad831f6-443f-47d7-acd7-0312135edcd7" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="b9ee617a-18c6-4901-b7f5-5d0b77ab2c33"
/dev/loop0: TYPE="squashfs"
/dev/sdc1: UUID="a853f64b-4308-435f-ab54-6b7f6da06779" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="407d5887-5172-4e57-969b-a8d93c729f09"
/dev/sdg1: PARTUUID="4080ed17-b0b3-4fb6-990e-162d6605431f"
/dev/sde1: PARTUUID="92515382-768b-4460-b640-626af0ecc063"

 

Should I simply change the file system under the cache disk settings to xfs and try to restart?  Thanks again.

 

1 minute ago, John K said:

Should I simply change the file system under the cache disk settings to xfs and try to restart?  Thanks again.

Correct, pool slots must be set to 1.

3 hours ago, JorgeB said:

Assign the drive to a cache pool and re-test, pools don't use the md driver.

OK!  Back on track with troubleshooting of the original issue...  I assigned the 4Kn SAS disk to a new cache pool and retested the write speed.   It performed at the speed that I would expect from the drive.  So it definitely seems to be md related.  Can you think of any further tests to try to narrow down why?  Do you see any value in booting from a freshly made Unraid USB stick with just the 4Kn SAS drive connected?  Would a single drive Unraid "array" use the md driver?  Thanks for your ongoing help, JorgeB.

 

root@Filer:/mnt/temp-sas# dd if=/dev/zero of=/mnt/temp-sas/test1.img bs=1G count=2 oflag=dsync
2+0 records in
2+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 9.59568 s, 224 MB/s

 

 

 

 

20 minutes ago, John K said:

So it definitely seems to be md related. 

It does look related, possibly because it's using 512K sectors and that disk performs very bad with those.

 

21 minutes ago, John K said:

Do you see any value in booting from a freshly made Unraid USB stick with just the 4Kn SAS drive connected?  Would a single drive Unraid "array" use the md driver?

You can try, and yes, a single drive assign to the array will still use the md driver, and it would be a good test without parity, though I suspect the results will be the same.

Interestingly, booting on just the single drive with a fresh Unraid USB stick, I see full-speed write performance.   In case helpful, I've attached diags from the test system.  Other thoughts on how to determine the root cause?  Thanks.

 

 

root@Tower:/mnt/disk1# dd if=/dev/zero of=test2.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.85953 s, 221 MB/s

tower-diagnostics-20230223-1535.zip

Interesting, was there anything else going on with the array during the disk clear? Parity sync and turbo write can be limited by a controller bottleneck but a disk clear would not, assuming nothing else was going on.

 

Please post the output of

lspci -d 1000: -vv

 

When I was doing the disk clear through Unraid, the system was very quiet.  I had Docker containers and VMs stopped.  The write performance would never get higher than 50-60 MB/s, even for brief periods. 

 

One odd behavior, which I mentioned in my original post, was if I booted the system with the 4Kn drive unassigned, a pre-clear (using the preclear plugin) would run at 200-250 MB/s.  If I then assigned the drive, started an Unraid disk clear (which ran at ~50 MB/s), stopped the disk clear, unassigned the drive, and then ran the pre-clear again, the zeroing performance would max out at ~50 MB/s instead of 200-250 MB/s.  It seems as if once Unraid accessed the drive, the performance was capped until the system was rebooted.

 

I should also note that Unraid didn't recognize the drive as zero'ed after running the pre-clear (I tried both the pre-clear plugin and the Docker container).  Unraid always wants to clear the disk itself.

 

root@Filer:~# lspci -d 1000: -vv
01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
        Subsystem: Dell 6Gbps SAS HBA Adapter
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        IOMMU group: 15
        Region 0: I/O ports at 6000 [disabled] [size=256]
        Region 1: Memory at 85540000 (64-bit, non-prefetchable) [size=64K]
        Region 3: Memory at 85500000 (64-bit, non-prefetchable) [size=256K]
        Expansion ROM at 85400000 [disabled] [size=1M]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [68] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x8
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [d0] Vital Product Data
pcilib: sysfs_read_vpd: read failed: No such device
                Not readable
        Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
                Vector table: BAR=1 offset=0000e000
                PBA: BAR=1 offset=0000f800
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [138 v1] Power Budgeting <?>
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas

 

This is very strange, controller is not downgraded, single array drive performs normally, so it cannot just be the md driver by itself, kind of stumped for now...

Unless anyone else has troubleshooting suggestions, I may start over, low level formatting the drive and then letting Unraid zero it again.  I very much appreciate all of the help, Jorge.

I'm happy to report that my 4Kn drive is now performing as expected.  Originally, I low level formatted the drive using sg_format to disable Protection Type 2 which appeared to work normally.  This time around, I re-formatted the drive using openSeaChest_FormatUnit.  I re-ran all of the commands above to check sector sizes and such and cannot find any differences, so I really don't know why the drive is behaving differently after formatting with openSeaChest versus sg_format.  It could just be coincidence.  Nonetheless, I'm happy to declare victory.  Thanks very much for all the help.

 

On 2/26/2023 at 9:48 AM, John K said:

I'm happy to report that my 4Kn drive is now performing as expected.  Originally, I low level formatted the drive using sg_format to disable Protection Type 2 which appeared to work normally.  This time around, I re-formatted the drive using openSeaChest_FormatUnit.  I re-ran all of the commands above to check sector sizes and such and cannot find any differences, so I really don't know why the drive is behaving differently after formatting with openSeaChest versus sg_format.  It could just be coincidence.  Nonetheless, I'm happy to declare victory.  Thanks very much for all the help.

 

I’m about to purchase the same drive.  Can you tell me the exact command you ran with openSeaChest?  Is this a Seagate only tool?
Thanks