Drive write speeds really slow (SOLVED)


Go to solution Solved by Vexorg,

Recommended Posts

Hello, just signed up to say thanks and share my config.

 

I was having similar issues with slow write speeds, around 25 MB/s but on benchmarks it peaks around 200 MB/s

 

image.thumb.png.73c1b0cc2c5595080bd250c95f0b8b1c.png

 

Now with disk cache enabled peaks around 190 MB/s both read and write, without using the cache drive and with turbo write enabled.

 

sdparm --set=WCE /dev/sdb

From NetData running in a docker container, before the read and write was at 100% utilization.

image.thumb.png.6430e2a485c0f26b6724b284de2736bc.png

 

Im using a DELL PERC H330 controller (even updated the firmware from unraid), tested both RAID (Physical disk mode) and HBA mode without any difference.

 

With 4x Dell / Seagate model: ST2000NM0018

 

Edited by Robin AK
Added HDD model.
Link to comment

Back to it being slow.
Had the machine off for a couple days, waiting on some parts to come in.

Moving more data on to it, and write speeds are down again to 30MB/s.

Did the WCE thing on all drives, no effect.

Will try mounting an NTFS disk in a bit, do a test transfer, and see if things change.

 

Local disk-to-disk transfer of a 4GB file resulted in a 168MB/s data rate.

 

cube-diagnostics-20191026-2353.zip

Edited by sota
Link to comment

Looks like you're having ACPI Errors too.

Oct 26 16:12:33 Cube kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20180810/exfield-393)
Oct 26 16:12:33 Cube kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20180810/psparse-516)
Oct 26 16:12:33 Cube kernel: ACPI Error: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20180810/power_meter-338)
Oct 26 16:18:02 Cube kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20180810/exfield-393)
Oct 26 16:18:02 Cube kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20180810/psparse-516)
Oct 26 16:18:02 Cube kernel: ACPI Error: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20180810/power_meter-338)

 

 

Looks like you don't have write cache enabled on some of the drives.

Oct 26 16:02:19 Cube kernel: sd 1:0:6:0: [sdg] Write cache: disabled, read cache: enabled, supports DPO and FUA
Oct 26 16:02:29 Cube kernel: sdg: sdg1
Oct 26 16:02:35 Cube kernel: sd 1:0:5:0: [sdf] Write cache: disabled, read cache: enabled, supports DPO and FUA
Oct 26 16:02:45 Cube kernel: sdf: sdf1
Oct 26 16:02:48 Cube kernel: sd 1:0:4:0: [sde] Write cache: disabled, read cache: enabled, supports DPO and FUA
Oct 26 16:02:48 Cube kernel: sde: sde1
Oct 26 16:02:53 Cube kernel: sd 1:0:4:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
Oct 26 16:02:53 Cube kernel: sde: sde1
Oct 26 16:04:12 Cube kernel: sd 1:0:3:0: [sdd] Write cache: disabled, read cache: enabled, supports DPO and FUA
Oct 26 16:04:12 Cube kernel: sdd: sdd1
Oct 26 16:04:16 Cube kernel: sd 1:0:3:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
Oct 26 16:04:16 Cube kernel: sdd: sdd1
Oct 26 16:04:17 Cube kernel: sd 1:0:2:0: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA
Oct 26 16:04:27 Cube kernel: sdc: sdc1
Oct 26 16:04:31 Cube kernel: sd 1:0:1:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
Oct 26 16:04:41 Cube kernel: sdb: sdb1
Oct 26 16:08:20 Cube kernel: sd 1:0:1:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
Oct 26 16:08:20 Cube kernel: sdb: sdb1
Oct 26 16:08:46 Cube kernel: sd 1:0:1:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
Oct 26 16:08:46 Cube kernel: sdb: sdb1

 

Link to comment

I'm not sure how to interpret the ACPI errors; keep in mind there's 3 controllers on this machine: CD-ROM SATA, HP B210i (not used), HP P420 (used).  And of course the USB slot for the unRAID stick (inside the case.)

 

I re-checked the cache settings and --get says they're all set to enabled.  However when I did force it for sdf and sdg and refreshed the log I see an entry each for sdf and sdg saying it's now enabled.  Weird.

 

eta: dug more on the ACPI errors... they're nothing to worry about.  They're related to HP being HP; see https://www.serveradminblog.com/2015/05/kernel-acpi-error-smbusipmigenericserialbus/

I'll worry about dealing with that later.

Edited by sota
Link to comment
  • 11 months later...
  • 4 weeks later...

Could someone help me diagnose slow write speeds on my server? 

I'm getting about 15-20MB/S, and when copying over multiple small files it takes an extremely long time. 

I've checked the write cache as per post two, and all seem to be enabled with the exception of the boot flash drive. 

 

Diagnostics attached! 

TIA!

 

 

 

edit: UPDATE

So I did some more testing, and it appears that the slow write speeds (and read speeds) were due to MacOS - I have two Macs as my main production machines, and a windows box as a fail-over. 

I tested the windows box and was consistently getting 105ish MB/s read over the network and ~70-80MB write, compared to ~50MB/s read on both Macs and 15-20MB/s write. 

After some hunting and troubleshooting, I found that MACos SMB implementation is apparently pretty crap, but turning off delayed ACK seemed to fix the issue. 

 

To test it (non permanent) in terminal: 

sudo sysctl -w net.inet.tcp.delayed_ack=0

 

All of a sudden, I was getting read and write speeds the same as the windows box. This is a MASSIVE improvement!!

 

For anyone that is on MACos and needs a permanent fix, this is what I did: 

 

 

To make the change permanent (requires reboot):

 

Create/edit the file /etc/sysctl.conf

Terminal:

sudo vim /etc/sysctl.conf

 

Add this line to the configuration file:

net.inet.tcp.delayed_ack=0

 

Save and close file (type “:wq” without quotes, then hit enter)

reboot computer

 

 

Edited by BenW
Link to comment
  • 7 months later...
  • 9 months later...
On 7/17/2018 at 6:25 PM, Vexorg said:

After a ton of Google-fu I was able to resolve my problem.

 

TL;DR Write cache on drive was disabled

 

found an page called How to Disable or Enable Write Caching in Linux. The artical covers both ata and scsi drives which i needed as SAS drive are scsi and are a total different beast.

 

root@Thor:/etc# sdparm -g WCE /dev/sdd
    /dev/sdd: HGST      HUS726040ALS214   MS00
WCE           0  [cha: y, def:  0, sav:  0]

This shows that the write cache disabled

 

root@Thor:/etc# sdparm --set=WCE /dev/sdd
    /dev/sdd: HGST      HUS726040ALS214   MS00

This enables it and my writes returned to the expected speeds

 

root@Thor:/etc# sdparm -g WCE /dev/sdd
    /dev/sdd: HGST      HUS726040ALS214   MS00
WCE           1  [cha: y, def:  0, sav:  0]

confirms the write cache has been set

 

Now I'm not total sure why the write cache was disabled under unraid, bug or feature? While doing my googling there was a mention of a kernel bug a few years ago that if system ram was more then 8G it disables the write cache. My current system has a little more then 8G so maybe?

 

this post show how everyone is really excited about this feature.

Currently im getting around 40MB/S write speed and i would like to enable this feature either.

could you write down all steps?

I do not understand where and how i can write this commands down, and how i handle my own HDD/SSD to this command.

Link to comment
  • 10 months later...

Grand! 

As i dont have space for a cache drive in my DL60 and use SAS disks this really helped out! 

Was getting the 40-50mb write over network smb and now its 105mb ish. 

What i had tho was to add the save prefix as well for it to kick in

sdparm --set=WCE --save /dev/sda

 

 

from https://linux.die.net/man/8/sdparm

To set the "Writeback Cache Enable" bit in the current values page:

sdparm --set=WCE /dev/sda

To set the "Writeback Cache Enable" bit in the current and saved values page:

sdparm --set=WCE --save /dev/sda

 

 

 

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.