I'm using 10x 8TB Seagates Archive drives and nothing else. System specs are below. Each drive tests at 200MB/s individually, however I am seeing 150MB/s at the start of a parity sync.

 

Is it possible my Norco 4224 Hotswap bays are limiting me? It's 1 SAS connection per 4 drives and they say SATA6 hotswap bays, so 4x 150MB/s is exactly the limit of Sata6. The SAS2LP cards are rated at 600MB/s per channel (I assume that means per drive).

I moved 4 of the drives down to different hotswap bays (so 2 drives per SAS connection and only 4 drives connected to each SAS2LP), and it's still 150MB/s.

 

Any ideas what could be limiting me?

V6.1 or V6.2-beta? What are your tunnable settings?

V6.1 or V6.2-beta? What are your tunnable settings?

 

6.1.9

 

Tunable (poll_attributes):

1800 (Default)

 

Tunable (md_num_stripes):

5632

 

Tunable (md_sync_window):

2560

 

I tried doubling the last 2 and get same result.

Maintaining the other settings as above, type in the console:

 

mdcmd set md_sync_thresh 1256

 

for i in {b..z}; do echo 8 > /sys/block/sd$i/queue/nr_requests; done

 

Either one should give a little boost, no problem using the two at the same time.

 

Also make sure the SAS2LP you're using is on one of the top two slots, the other two are x4.

 

 

Maintaining the other settings as above, type in the console:

 

mdcmd set md_sync_thresh 1256

 

for i in {b..z}; do echo 8 > /sys/block/sd$i/queue/nr_requests; done

 

Either one should give a little boost, no problem using the two at the same time.

 

Also make sure the SAS2LP you're using is on one of the top two slots, the other two are x4.

 

I did those commands but the second one you linked returned "-bash: /sys/block/sdp/queue/nr_requests: No such file or directory".

 

It did not fix the problem, and it's even worse now. i'm at 56% and only going 90-120MB/s (going back and forth each refresh). The drives should be going roughly 140MB/s at this point according to individual tests. 9 hours in, estimated 10 hours to finish.

I did those commands but the second one you linked returned "-bash: /sys/block/sdp/queue/nr_requests: No such file or directory".

 

That's normal and ok, it sets the value from sdb to sdz, error is from the missing disks.

 

It did not fix the problem, and it's even worse now. i'm at 56% and only going 90MB/s. The drives should be going roughly 140MB/s at this point according to individual tests.

 

I'm surprised it didn't help, use this to get back to default:

 

for i in {b..z}; do echo 128 > /sys/block/sd$i/queue/nr_requests; done

 

mdcmd set md_sync_thresh 192

 

 

I'm going only 20-40MB/s now, with spikes into 90-120MB/s every now and then. I don't think it's related to those commands, I was randomly seeing this before running them. The 2 SAS2LP cards in use are on the 3.0 x8 slots, should be more than enough bandwidth.

 

EDIT: It started sustaining 135MB/s again after letting it sit for a bit. However, it will dip randomly to ~70MB/s. I put in your custom settings again, and reduced my tunables to Tunable (md_num_stripes): 2560, Tunable (md_sync_window): 1024. I let it sit for about 5 minutes, and it went up to 140-150MB/s sustained at 58% in. The random dips are still there though. Perhaps the issue is somewhat fixed, no way to know until this finishes and I do a fresh parity check. Can I just put your commands in my go file?

 

EDIT2: Watching the HDD activity lights on my server, I'm thinking this is due to initial parity sync. The parity drive (writing) HDD activity light is always on, while all the other drives are flashing. Sometimes the other 9 HDD activity lights will turn completely off for 3-5 seconds, while the parity light stays completely on. Explains the drops, seems these drives write slower than they read.

May I ask what model of Seagate Archive drive are you using - the older ST8000AS0002 or the newer ST8000AS0022?

I've always had constant speed (following the disks performance curve) on my server that uses 8TB archive disks, be it parity sync, check or disk rebuilds.

May I ask what model of Seagate Archive drive are you using - the older ST8000AS0002 or the newer ST8000AS0022?

 

I just bought them from Amazon and they look to be the old ST8000AS0002 models. Only different I see is slightly less power consumption?

 

Still weird that a company like Amazon would be shipping old versions. Didn't v2 come out in March?

 

I've always had constant speed (following the disks performance curve) on my server that uses 8TB archive disks, be it parity sync, check or disk rebuilds.

 

I dunno what to do then. I have parity drive and 7 data drives are all on a single SAS2LP, but it's running on PCI-E 3.0 x8 so it should have more than enough bandwidth (unless supermicro is lying about 600MB/s per channel).  I tried spreading all 10 drives across all 3 SAS2LP controllers, and had the same results. The HDD activity lights on my Norco 4224 are showing a very clear bottleneck on the parity drive. It's on completely, while the data disks are flickering and sometimes turning completely off for a couple seconds - which explains the drop in speeds.

 

Since you only do an initial parity sync once, I figure i'll let this complete and see how fast my parity check goes when it only has to read from every drive. The parity drive's smart is fine to my eyes:

 

1 Raw read error rate 0x000f 119 099 006 Pre-fail Always Never 220717904

3 Spin up time 0x0003 094 094 000 Pre-fail Always Never 0

4 Start stop count 0x0032 100 100 020 Old age Always Never 6

5 Reallocated sector count 0x0033 100 100 010 Pre-fail Always Never 0

7 Seek error rate 0x000f 072 060 030 Pre-fail Always Never 20514796

9 Power on hours 0x0032 100 100 000 Old age Always Never 241 (10d, 1h)

10 Spin retry count 0x0013 100 100 097 Pre-fail Always Never 0

12 Power cycle count 0x0032 100 100 020 Old age Always Never 6

183 Runtime bad block 0x0032 100 100 000 Old age Always Never 0

184 End-to-end error 0x0032 100 100 099 Old age Always Never 0

187 Reported uncorrect 0x0032 100 100 000 Old age Always Never 0

188 Command timeout 0x0032 100 100 000 Old age Always Never 0

189 High fly writes 0x003a 100 100 000 Old age Always Never 0

190 Airflow temperature cel 0x0022 058 057 045 Old age Always Never 42 (min/max 30/43)

191 G-sense error rate 0x0032 100 100 000 Old age Always Never 0

192 Power-off retract count 0x0032 100 100 000 Old age Always Never 81

193 Load cycle count 0x0032 100 100 000 Old age Always Never 176

194 Temperature celsius 0x0022 042 043 000 Old age Always Never 42 (0 23 0 0 0)

195 Hardware ECC recovered 0x001a 119 099 000 Old age Always Never 220717904

197 Current pending sector 0x0012 100 100 000 Old age Always Never 0

198 Offline uncorrectable 0x0010 100 100 000 Old age Offline Never 0

199 UDMA CRC error count 0x003e 200 200 000 Old age Always Never 0

240 Head flying hours 0x0000 100 253 000 Old age Offline Never 265991619608657

241 Total lbas written 0x0000 100 253 000 Old age Offline Never 28798451088

242 Total lbas read 0x0000 100 253 000 Old age Offline Never 32173147128

May I ask what model of Seagate Archive drive are you using - the older ST8000AS0002 or the newer ST8000AS0022?

 

I just bought them from Amazon and they look to be the old ST8000AS0002 models. Only different I see is slightly less power consumption?

 

No, they are very different. Firmware rewritten, new command set implemented. https://lime-technology.com/forum/index.php?topic=47509.msg455104#msg455104

 

Total size: 8 TB

Elapsed time: 13 hours, 25 minutes

Current position: 6.39 TB (79.9 %)

Estimated speed: 120.5 MB/sec

Estimated finish: 3 hours, 43 minutes

 

This is what I'm seeing right now. So probably about ~18 hour parity sync. If I remember right John, you get about 14 hours? I added your commands half way into this, which seemed to hurt initially, but waiting a few minutes seem to help. Perhaps if I run it again it'll be faster.

 

May I ask what model of Seagate Archive drive are you using - the older ST8000AS0002 or the newer ST8000AS0022?

 

I just bought them from Amazon and they look to be the old ST8000AS0002 models. Only different I see is slightly less power consumption?

 

No, they are very different. Firmware rewritten, new command set implemented. https://lime-technology.com/forum/index.php?topic=47509.msg455104#msg455104

 

So basically I just dropped $2200 on drives from Amazon, when Newegg sells them for the same price and guarantees v2?

 

EDIT: I'm not even sure Newegg is selling that model http://www.newegg.com/Product/Product.aspx?Item=N82E16822178748. They advertise v2 but it's the same model # as mine. I can't find a single store selling a "ST8000AS0022".

I can't seem to get consistent speeds even with a parity check (all reading). I was getting 200MB/s, restarted, and it was slow again.

 

The only way I seem to get consistent 190+ MB/s speeds is:

- Restart server

- Telnet in and manually apply your tweaks

- Start parity check

 

Anyway to apply these settings on startup? Adding to the go file doesn't seem to work. I'm still seeing random drops to ~170MB/s for unknown reasons, but it's not a huge deal.

md_sync_thresh only works after the array has started, so to have it on the go file you need to insert a delay, like this:

 

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp &

sleep 30
mdcmd set md_sync_thresh 1256

 

If you stop and the re-start the array, like when upgrading a disk, the setting goes back to default.

 

Both these settings are tunnable in v6.2-beta.

 

md_sync_thresh only works after the array has started, so to have it on the go file you need to insert a delay, like this:

 

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp &

sleep 30
mdcmd set md_sync_thresh 1256

 

If you stop and the re-start the array, like when upgrading a disk, the setting goes back to default.

 

Both these settings are tunnable in v6.2-beta.

 

Ah thank you. I won't worry about it then, i'll probably update to 6.2 as soon as the RCs start rolling out.

 

I'm doing a parity check right now and it's at 48% with 6 hours in and 7 hours estimated finish. Much better.