Hi guys,

 

Any idea why my native disk to disk copies would be painfully slow? I've tested using MC via SSH and windows SMB copy on a 30gb+ single file copies and i burst around 200MB/s and then continue to fall until i reach 25/30MB/s I was really expecting to be able to hold 70-90MB/s disk to disk across the HBA's and MB! Nothing else happening accross VM's/Docker at the time of the copy. CPU <10% as well.

 

1tb Samsung Evo (CACHE DRIVE) [Plugged into 6GB/s port on MB]  -> Hitachi 2TB (Disk14) [Plugged into 6GB/s HBA]

 

My unraid setup runs dual parity on the following hardware spec:

 

Supermicro X9SCM-F  (E3-1240)

3 x Dell H200 crossflashed to IT mode. (using 2 8087 cables per card to midplane) 

16 x 2tb disks + 1tb samsung evo cache

 

Anyone experienced anything similar?

Do you have the 'Turbo Write' option enabled?   That might be required to get anything like the speeds you are looking for (at the expense of having all drives spinning). When copying to drives in the parity protected array with default settings then speeds of around 30-40 MBs are quite typical.   

 

The architecture of unRAID means that each 'write' operation actually involves 4 I/O operations - a read of the parity and target drives to get the current contents of a sector, followed by writes to both of those drives to output the updated sector contents.   This means that there is a minimum of one additional disk rotation per write.   The 'turbo' write option often speeds things up by eliminating the need to read a sector of the parity drive as it reconstructs it using the data from all the other drives (which can all be read in parallel as long as your disk controller can handle it) and thus saves the time for a disk rotation.

FYI, Turbo Write is in my signature.  Also FYI, your array is composed of 2TB drives.  They're pretty slow - you'd get better speeds with high density 6+ TB drives.

Do you have the 'Turbo Write' option enabled? 

No i don't. I try and keep the array on as low TDP as much as possible. I didn't really think i'd have to enable turbo mode to get a disk transfer speed of more than 25MB/s on a drive capable of around 140-150MB/s

 

The architecture of unRAID means that each 'write' operation actually involves 4 I/O operations 

Totally. Seems a bit excessive though on a large file transfer right (1 file at 30gb+ 25MB/s)  Especially coming from an SSD Cache.

 

Also FYI, your array is composed of 2TB drives.  They're pretty slow - you'd get better speeds with high density 6+ TB drives.

Totally agree, however that would involve buying 6 x 6TB Disks (cost) simply for the sake of a few extra MB/s for disk to disk copies.... Mine were all free (with a handful of spares ready for disk failure!) and should be more than capable of archiving data. What sort of disk to disk speeds could i expect going from SSD -> 6tb disk without turbo mode?

 

I suppose i'm trying to evaluate whether my config has an issue and whether the copy i detailed should be sitting more around 50/60/70MB/s  or whether a copy of a 30GB single file from  SSD -> disk is perfectly acceptable to be 25MB/s it just dosnt seem right to me. What speeds do you guys see doing these sort of copies?

30 minutes ago, thestraycat said:

What speeds do you guys see doing these sort of copies?

 

You shout get between 50 and 75MB/s with modern disks and normal write mode, up to 200MB/s with turbo write.

Hi Johnnie - That's what i thought. the 2tb disks (7200rpm | 6gb/s -> 6gb/s HBA's) Should be capable of 50-75MB's on single large file copies from SSD surely...

 

Any idea where to start on why i might be seeing tragic copy times? The destination disk is a 2TB BLANK, the SSD cache has 600gb free on it . (does it need to be pre-cleared at all, not added additional disks since initial installation of unraid)

 

I'll try the same copy to another disk in the array to see if i see similar disk speeds to out rule a dodgy disk.... Disk's SMART info shows as a pass for what it's worth...

 

 

I mentioned modern disks, older disks will be slower, what is the model of your disks?

58 minutes ago, thestraycat said:

That's what i thought. the 2tb disks (7200rpm | 6gb/s -> 6gb/s HBA's) Should be capable of 50-75MB's on single large file copies from SSD surely...

My 3 year old 3TB WD Reds can't sustain 75MB/s so I don't know why you'd expect your 2TB drives to be able to do that.  It takes the higher areal density of the high capacity drives to get above 50MB/s.  I could probably sustain 40-50MB/s when I was running just 3TB drives.  25-30MB/s is within the reasonable range if those are a bunch of older 2TB drives.  What does your RAM/CPU situation look like?

Johnnie.Black - These are mine... https://www.cnet.com/products/hgst-deskstar-7k3000-hds723020bla642-hard-drive-2-tb-sata-600-series/specs/

Quoted at 200mb/s+ internal speeds.. Understand there's going to be penalty's but 8 times? 

 

tdallen - CPU = 8% RAM = 29%

 

Copy from the cache -> Disk 4 = same speed

Move from disk4 -> disk14 (all spinners) = Averaged at 45MB/s

 

 

 

 

 

The 207 MB/s is internal transfer rate.

 

Average transfer speed of the drive is 121.3 MB/s http://www.silentpcreview.com/article1243-page4.html

 

Once you add the slowdowns because of reading & updating parity, you're not that far off the mark of where you should be.

With my Green system I've pretty much come to the conclusion that things will be done when things are done. 

1 hour ago, thestraycat said:

Averaged at 45MB/s

 

That sounds about right, it's and older model with 600GB platters, 40 to 50MB/s it's the speed I would expect for normal writes, don't know why it's slower when copying from your cache but's something else is going on, in normal write mode speed should be the same regardless if the source is cache or another array disk,

Agree. Any idea of where to start with this one?

Not really, it's an unusual problem, post your diagnostics, maybe something visible there.

Generating it now. Will upload when it completes.

Here you go! 

 

mediaserver-diagnostics-20171108-2221.zip

Nothing jumps out, you'll need to do some testing to try and rule things out, e.g., using a different cache device, controller, etc.

Thanks for looking. 

OK. So i restarted unraid (38 days uptime) and copied the same file and averaged 58MB/s throughout a 45gb single file transfer (which was what i was expecting.) Any idea why a reboot seemed to speed things up?