Drive write speeds really slow (SOLVED)


Go to solution Solved by Vexorg,

Recommended Posts

Hello all,

 

I've just build a new unraid server and after playing around with a bit before I really start using it I was finding it the drive writes seemed to be a bit slow.

 

I first noticed the slowness while I was doing a badblock run on the new drives and after 50 hours it had just finished the first pass. For 4TB drives I understand it should take a total of 40hours or so for this size of drive. I did not see any errors and killed the process and proceeded to build the array with the drives and when it was doing the parrity build the stated write speed to the drive with 47MB/sec which seemed to me a bit slow for the drives. I checked the spec's from the manufacturer and it said it should be closer to 200MB/sec so I'm getting about 25% of the speed.

 

Next thing I did was boot into windows and ran a few benchmarking tools from there and they all reported read/write speeds about 200MB/sec... I did find that one of the LSI cards was in a PCIe 4x slot instead of an 8X one so I moved it to a new slot. 

 

Next try was a live linux USB to see if the problem was with unix after things seem to be fine in windows. Used knoppix and ran a few tests and the drives speeds where in the 200MB/sec again?!? 

 

My conclusion is the Unraid distro is the problem and not with hardware.

 

Now for some hardware details of my system:

Supermicro x10DRi-ln4+

2x intel e5-2630's

3x LSI 9211 HBA's connected to a supermicro BPN-SAS-846A backplane

6x HGST 4tb 3.5" HDD HUS726040ALS214 SAS drives

 

root@Thor:~# hdparm -Tt /dev/sdh

/dev/sdh:
 Timing cached reads:   21860 MB in  1.99 seconds = 10970.43 MB/sec
 Timing buffered disk reads: 570 MB in  3.00 seconds = 189.82 MB/sec

root@Thor:~# dd if=/dev/zero of=/dev/sdh bs=128k count=10k
10240+0 records in
10240+0 records out
1342177280 bytes (1.3 GB, 1.2 GiB) copied, 28.5249 s, 47.1 MB/s

I'm not really sure where to look at next.

thor-diagnostics-20180716-1149.zip

  • Like 1
Link to comment
  • Vexorg changed the title to Drive write speeds really slow (SOLVED)
  • Solution

After a ton of Google-fu I was able to resolve my problem.

 

TL;DR Write cache on drive was disabled

 

found an page called How to Disable or Enable Write Caching in Linux. The artical covers both ata and scsi drives which i needed as SAS drive are scsi and are a total different beast.

 

root@Thor:/etc# sdparm -g WCE /dev/sdd
    /dev/sdd: HGST      HUS726040ALS214   MS00
WCE           0  [cha: y, def:  0, sav:  0]

This shows that the write cache disabled

 

root@Thor:/etc# sdparm --set=WCE /dev/sdd
    /dev/sdd: HGST      HUS726040ALS214   MS00

This enables it and my writes returned to the expected speeds

 

root@Thor:/etc# sdparm -g WCE /dev/sdd
    /dev/sdd: HGST      HUS726040ALS214   MS00
WCE           1  [cha: y, def:  0, sav:  0]

confirms the write cache has been set

 

Now I'm not total sure why the write cache was disabled under unraid, bug or feature? While doing my googling there was a mention of a kernel bug a few years ago that if system ram was more then 8G it disables the write cache. My current system has a little more then 8G so maybe?

  • Like 5
  • Thanks 1
  • Upvote 6
Link to comment
3 minutes ago, Vexorg said:

Now I'm not total sure why the write cache was disabled under unraid, bug or feature?

First time I hear that, maybe if it's limited to SAS drives since they are not used by many, still would think other people would have complained by now it it's a general problem.

Link to comment

Thanks for this Vexorg. I was searching for a different issue (SAS spindown issues) and saw your post. I had been building a new server and had poor performance but hadn't had time to research. What I found was interesting - all my HP branded SAS drives had write cache disabled, while all my Seagate drives had it enabled. These drives were split across two servers so my assumption is that this was set on the drives themselves at some point and that this has nothing to do with unRAID. 

 

Enabled write caching on the HP drives and performance is back to where I'd expect, so you saved me a ton of time. Thanks again.

Link to comment
  • 5 months later...
  • 1 month later...
  • 2 months later...

Awesome post. I saw that my write cache was disabled and wasn't sure about enabling it. Thanks for posting your experiences and the commands ran to change the states. I was also experiencing the sub 50MB/s speeds prior to enabling write cache and now get over 200MB/s.

GIGABYTE GA-7PESH2 board with onboard sas controller flashed to IT mode with Seagate ST6000NM0034 drives. 

Link to comment

johnnie.black redirected me to this post

As the writing speed of Parity Clearing time was showing 12 hrs.

sdparm command not found was the message at first but as per AnnabellaRenee87 post, installed nerdpack GUI plugin.

after that everything was magic.

12 hrs parity work reduced to 3 hrs.

This is very helpful post.

Thanks

  • Like 1
Link to comment
  • 3 weeks later...
On 7/17/2018 at 11:25 AM, Vexorg said:

After a ton of Google-fu I was able to resolve my problem.

 

TL;DR Write cache on drive was disabled

 

found an page called How to Disable or Enable Write Caching in Linux. The artical covers both ata and scsi drives which i needed as SAS drive are scsi and are a total different beast.

 

Thanks a lot, I spent a ton of time spinning my wheels on this problem!  I added a Dell PERC H310 SAS card to my setup along with 6x 3TB SAS drives.  The parity sync was crawling along around 40 MB/s.  Turns out all the write cache's were disabled for the SAS drives (enabled for my SATA drives on the separate on-board SATA controller).  Enabled the write cache on the SAS drives and I'm at 140 MB/s now which is perfectly acceptable.

 

Some other stuff if anyone else bought an H310 and stumbles onto this.  The H310 requires some work to get going (apparently well known).  Had to tape pins B5 and B6 on the card itself so my PC would boot.  Initially after doing that unRaid noticed all the SAS drives but parity sync was around 10 MB/s.  That's when I stumbled onto flashing the H310 firmware with the IT firmware (I had the RAID firmware).  That's what brought me up to 40 MB/s.  The write caching was the last piece of the puzzle here.  Link to the firmware bit.

 

 

  • Like 1
Link to comment
  • 2 weeks later...
  • 1 month later...
  • 1 month later...
  • 1 month later...
43 minutes ago, Endda said:

@Squid @limetech any word on this fix being added to unRAID Disk Settings or a popular plugin like Turbowrite?

It's not a problem with the OS, but rather how the manufacturers choose to configure their hard drives.  Which is why Fix Common Problems is telling you about it.  Not sure if I'd want unRaid (or any OS) automatically doing this for me.  On the other hand though, a setting in Disk Settings would make life easier.

  • Like 2
Link to comment
  • 1 month later...

I'm apparently having the same problem.  My write speeds are in the sub-30MB/s range.  I checked the WCE flag and it's enabled on all my drives.  This has been consistent across two different builds (other was an HP 8000 Elite SFF C2Q Q9650, 16GB ram, native SATA connectors.)

 

I'm still looking for a solution.

Link to comment
2 hours ago, sota said:

I'm apparently having the same problem.  My write speeds are in the sub-30MB/s range.  I checked the WCE flag and it's enabled on all my drives.  This has been consistent across two different builds (other was an HP 8000 Elite SFF C2Q Q9650, 16GB ram, native SATA connectors.)

 

I'm still looking for a solution.

Tools - Diagnostics, attach complete diagnostics zip file to your NEXT post.

Link to comment
2 hours ago, trurl said:

Tools - Diagnostics, attach complete diagnostics zip file to your NEXT post.

Here you go.

 

I have no idea what changed, but all of the sudden i'm now getting 100MB/s transfer speeds into unRAID.

I'm trying to think of what it is I messed with during this time:

Hot added a 4th drive to the machine, which showed up as /dev/sde.

Mounted sde as /mnt/ntfs1 (it's an NTFS formatted drive.)

Did some file copy tests from it to one of the array disks (/mnt/disk2.)

Got a computed 112MB/s data transfer rate, which to me is appropriate.

Deleted the copied files and unmounted /mnt/ntfs1.

After all that, I went back and looked at the network copy activity, and it was running near 100MB/s.  i could see on the graph (Windows Server 2012R2 source) the big hump and jump up.  I guess whatever Fat Electron cleared out of the wire?

 

cube-diagnostics-20191019-0015.zip

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.