Very unstable read/write speads during parity sync

[Fredrick]

Hi,

 

I've added a parity drive, and it is currently doing the sync for the new drive. The overall speeds seems to me to be decent at about 70MB/s (should I be expecting more?), but with the system stats plugin I get this view:

 

 

Is this normal? It seems the overall speed is going down as I'm nearing the inner part of the drives, which I understand is expected. I'll be running some tests on the drives and the tunables tester after the parity sync to see if anything seems wierd.

 

Here is also a view of my drives:

 

 

 

[JorgeB]

31 minutes ago, Fredrick said:

The overall speeds seems to me to be decent at about 70MB/s (should I be expecting more?)

 

It really depends on the hardware used, we can see the disks but not the rest, CPU, board, HBA, etc, tunables can also have an effect.

 

Without other bottlenecks, and just looking at your disks, it should start at or above 150MB/s

[Vr2Io]

I suspect you have something parallel writing during sync and worse with those SMR drive in array.

Sync complete 50%+ ? Pls try "Clear Statistics" and post the screen capture after 3 min.

[Fredrick]

24 minutes ago, johnnie.black said:

 

It really depends on the hardware used, we can see the disks but not the rest, CPU, board, HBA, etc, tunables can also have an effect.-

 

Without other bottlenecks, and just looking at your disks, it should start at or above 150MB/s

 

HP Proliant ML350 G6, 2x Xeon X5650. LSI SAS 9201-16e. Tunables are default. All drives are run through a single SAS cable to my EXP3000 as we discussed in another thread.

 

18 minutes ago, Benson said:

Are you sure haven't any job writing in same time ? I suspect you have something parallel writing during sync and worse with those SMR drive in array.

Sync complete 50%+ ? Pls try "Clear Statistics" and post the screen capture after 3 min.

 

The server has been very quiet during this sync, and no larger operations to suggest what we are seeing. There are some reads here and there, but not a lot, and not for the whole day as the stats have showed. No writes in the time i pasted stats for. 

 

"Clear statistics"? As in the reset button in the stats page?

 

Here is the past 5 minutes after clearing stats.

 

[Vr2Io]

4 minutes ago, Fredrick said:

"Clear statistics"? As in the reset button in the stats page?

 

"Main" page which near bottom "Clear statistics", thanks

[Fredrick]

This is after a couple minutes (do you need exact time?)

 

 

Currently one stream going to Plex, apparently from Disk 1. Also Parity sync is at 97,6%, so the 4tb are done.

[Vr2Io]

I suppose there should be no any write in parity/disk1/disk6, but I may be wrong. I think you should focus on this.

 

If you don't mind stop current parity2 sync, you may stop it and start array in maintenance mode and check speed uniform or not. ( In this mode, STATS plugin won't work )

[Fredrick]

The write numbers are the same 10 minutes later if that matters, the system stats graph still looks just as spikey

[Vr2Io]

Some write in other disks should be normal during parity2 sync, I think problem was those SMR disks (ST8000AS0002), nothing you can do.

[Fredrick]

The Seagate Archive drives?

 

I know they are not ideal, but they were so much cheaper here in Norway when I bought them  Anything I can do to make things better?

 

BTW, I'm replacing the archive parity with a Red now, but letting it build parity to second drive first. (going from 1 Seagate Archive Parity drive to two Reds)

[JorgeB]

4 minutes ago, Fredrick said:

The Seagate Archive drives?

 

I know they are not ideal, but they were so much cheaper here in Norway when I bought them  Anything I can do to make things better?

 

BTW, I'm replacing the archive parity with a Red now, but letting it build parity to second drive first. (going from 1 Seagate Archive Parity drive to two Reds)

 

The Seagate archive drives works perfectly well with unRAID, I have a server with only those disks and average speed during parity check is ~150MB/s.

 

If the check is now near the end the much slower speed is normal, please post the stats graph that shows from the beginning of the sync until now.

[Fredrick]

1 minute ago, johnnie.black said:

The Seagate archive drives works perfectly well with unRAID, I have a server with only those disks and average speed during parity check is ~150MB/s.

 

Glad to hear it!

 

1 minute ago, johnnie.black said:

If the check is now near the end the much slower speed is normal, please post the stats graph that shows from the beginning of the sync until now.

 

[JorgeB]

That is all over the place, strange pattern, possibly a disk with slow sectors, but they usually look more like this when that's the problem:

 

[Vr2Io]

Would you open another browser page ( main and stats ), try to check does those drop happen in each write ( exclude parity2 )

[Vr2Io]

Trying simulate parity2 sync, no any write on other disks

 

 

[Fredrick]

12 minutes ago, johnnie.black said:

That is all over the place, strange pattern, possibly a disk with slow sectors, but they usually look more like this when that's the problem:

 

Yes, as I said its fairly unstable. I should probably have opened with that graph instead.

 

12 minutes ago, Benson said:

Would you open another browser page ( main and stats ), try to check does those drop happen in each write ( exclude parity2 )

Does not seem to be the problem. I'm still seeing a lot of spikes between about 450MB/s total read and 150MB/s total read.

 

There has been no other writes to the array/parity since i cleared statistics almost an hour ago.

 

 

[Vr2Io]

Sound good those write stop.

Agree, it may slow sector in some disk.

[Fredrick]

How can I troubleshoot? Repeated read speed tests on each drive?

 

Short/Extended self test would not reveal this?

 

Oh, and btw. During preclear of my Parity 2 drive (and the other Red sitting as unassigned currently) the speeds were steady all the through. Both pre/post-read and during write.

 

EDIT2: Finished!

 

Sep 28 22:14:12 Tower kernel: md: sync done. time=114278sec
Sep 28 22:14:12 Tower kernel: md: recovery thread: completion status: 0

So thats pretty spot on 70MB/s

[Vr2Io]

Short/Extended self test would not help.

 

https://forums.lime-technology.com/topic/29435-drive-performance-testing-version-264-for-unraid-5-thru-63/

[Fredrick]

Thanks, I've got the plugin installed, and that is what I was planning on doing

 

I'll keep the 4tb outside the testing for now as its clear the problem persisted throughout the whole 8tb.

[JorgeB]

Diskspeed plugin is the best way, make sure you select a lot of sample points, example of a disk with slow sectors:

 

[Fredrick]

I'll update later with the results of the Diskspeed test. Do I need more iterations aswell? I feel 1 iteration is very vulnerable to anything going on and can give false results. 

[JorgeB]

I feel 1 iteration is very vulnerable to anything going on and can give false results. 

 

The array shouldn't be in any kind of use during the test or it will be pointless.

 

 

[Fredrick]

Well, here are some more odd results. Please note, this is with the array started, I'm running further tests now with it stopped. Currently running 2% slices on all drives in normal mode with array stopped. 

 

Cache drive and the unassigned "LOGICAL VOLUME" are attached through the internal raid controller in my Proliant (no S.M.A.R.T. available). All other drives are via the LSI card and in the same EXP3000.

 

This is with Fast Mode enabled:

Drive Identification Parity:  ST8000AS0002-1NA17Z Z840FDHA   8 TB   159 MB/sec avg Parity 2:  WDC WD80EFZX-68UW8N0 VK1BX6DY   8 TB   152 MB/sec avg Disk 1:  ST8000AS0002-1NA17Z Z840YYKD   8 TB   157 MB/sec avg Disk 2:  ST8000AS0002-1NA17Z Z840SGX9   8 TB   155 MB/sec avg Disk 3:  ST4000VN008-2DR166 ZDH0SQDZ   4 TB   163 MB/sec avg Disk 4:  ST4000VN000-1H4168 Z300Q4G4   4 TB   143 MB/sec avg Disk 5:  WDC WD80EFAX-68LHPN0 7SGEPSXC   8 TB   159 MB/sec avg Disk 6:  WDC WD80EFAX-68LHPN0 7SGE58EC   8 TB   160 MB/sec avg Cache:  LOGICAL_VOLUME_50014380236B8AE0_3600508b1001ce64db334f5dc2199a3d0       229 MB/sec avg sdb:  Unable to determine       228 MB/sec avg sdh:  WDC WD80EFZX-68UW8N0 VK1BEXEY   8 TB   151 MB/sec avg Generated on Tower at Fri Sep 29 07:04:42 CEST 2017 Drives scanned 1 time every 1.0%

 

This is normal mode, and with 2% slices. 

 

Drive Identification
Parity: 	ST8000AS0002-1NA17Z Z840FDHA  	8 TB	  160 MB/sec avg
Parity 2: 	WDC WD80EFZX-68UW8N0 VK1BX6DY  	8 TB	  150 MB/sec avg
Disk 1: 	ST8000AS0002-1NA17Z Z840YYKD  	8 TB	  156 MB/sec avg
Disk 2: 	ST8000AS0002-1NA17Z Z840SGX9  	8 TB	  155 MB/sec avg
Disk 3: 	ST4000VN008-2DR166 ZDH0SQDZ  	4 TB	  163 MB/sec avg
Disk 4: 	ST4000VN000-1H4168 Z300Q4G4  	4 TB	  143 MB/sec avg
Disk 5: 	WDC WD80EFAX-68LHPN0 7SGEPSXC  	8 TB	  157 MB/sec avg
Disk 6: 	WDC WD80EFAX-68LHPN0 7SGE58EC  	8 TB	  158 MB/sec avg
Cache: 	LOGICAL_VOLUME_50014380236B8AE0_3600508b1001ce64db334f5dc2199a3d0  		  n/a
sdb: 	Unable to determine  		  n/a
sdh: 	WDC WD80EFZX-68UW8N0 VK1BEXEY  	8 TB	  149 MB/sec avg

Generated on Tower at Thu Sep 28 23:40:35 CEST 2017
Drives scanned 1 time every 2.0%

 

Now first of all I feel these lines are all "jagged", but no clear drive with bad/slow sectors. The average speeds seems decent overall with disk 4 lagging behind a bit.

 

Secondly. What is going on with my SSDs? The Cache and Logical Volume (both Kingston SSD Now, 240gb and 120gb respectively) show 228/229MB/s which is very low. Furthermore they seem to start and stop at 0gb. I'll test this some more aswell.

 

And third, my parity 2 drive seems to be slower than my other Reds, and has a much more jagged look than the other drives. 

 

[Fredrick]

One more bit of information. Several of my drives are very full. Could this be a problem? I read conflicting reports, and until recently my array was pretty full overall. I could move some data around if that can help these speeds.