My setup:

HDD: 6x WD Red Pro 22TB (XFS - 1x drive parity)

SDD: 2x WD Black 4TB (ZFS - Raid 1)

 

I have turbo write enabled but seem to get write speeds of around 110MB/s when transferring from SSD cache pool but weirdly during parity check I get sustained speeds of 250MB/s. Am I doing something wrong?

 

 

look nomral. need diag file to further assist.
 

 

I have seen posts where users state that their write speeds during turbo write transfers match the max write speeds of their drives. Is this not the case?

 

Also I have attached the diagnostics ZIP file.

server-diagnostics-20241127-1816.zip

2 minutes ago, unraid_user11 said:

 

I have seen posts where users state that their write speeds during turbo write transfers match the max write speeds of their drives. Is this not the case?

 

Not that I know of.    ~100MB is a typical speed when using Turbo Write.

This is one of the posts I am referring to from Reddit:

 

 

According to user Nick2Smith he is getting 250MB/s turbo write speeds.

 

review:

 

Edited November 28, 20241 yr by bmartino1
data

With turbo write, you should get approximately the same speed you get during a parity check/disk rebuild at that point, since disks are faster or slower depending on if you are writing to the outer or inner sectors.

2 hours ago, JorgeB said:

With turbo write, you should get approximately the same speed you get during a parity check/disk rebuild

 

It is my experience that this is only true when writing large files (say, >4GB).  Otherwise, the file overhead (and the required disk head movements) slow things down.   Large numbers of very small files (Say, 100K) can slow things to a crawl (say, ~5MB/s)!  SMR drives are another issue to be considered but performance with them is dependent more on how the disk manufacturer has set things to address the block rewrite problem.

2 hours ago, JorgeB said:

With turbo write, you should get approximately the same speed you get during a parity check/disk rebuild at that point, since disks are faster or slower depending on if you are writing to the outer or inner sectors.

I think that is only true if all drives are the same model and/or have the same packing density.   In turbo write the system has to read a sector from all drives before anything can be written so unless the drives are perfectly synchronised in read speed there are additional delays introduced.

56 minutes ago, Frank1940 said:

It is my experience that this is only true when writing large files (say, >4GB)

True, that's a good point.

 

15 minutes ago, itimpi said:

I think that is only true if all drives are the same model and/or have the same packing density.

Parity check/rebuild will also be slower if you sue different drives, turbo write should still perform very similar, to what the sync/rebuild speed is at that point.

4 hours ago, JorgeB said:

Parity check/rebuild will also be slower if you sue different drives, turbo write should still perform very similar, to what the sync/rebuild speed is at that point.

 

I noticed that myself. I had 3 IronWolf 12TB drive and would easily have a 200MB/s average parity check speed. I added a couple of WD Red Plus drives of same capacity 12TB drives and it easily slowed the process quite a bit. I will say that I also went from single parity to dual parity as well.

 

This diskspeed graph posted by another user can give a better idea, say you are writing to disk4 just before the 1TB mark with turbo write, max speed will be around 55MB/s, limited by disk6 read speed:

 

 

If you write to the same disk4 just after the 1TB mark, max speed will be around 160MB/s:

 

13 hours ago, itimpi said:

I think that is only true if all drives are the same model and/or have the same packing density.

All the 6x HDD drives are all the same. Even both NVME cache drives in RAID1 are the same.

 

16 hours ago, JorgeB said:

With turbo write, you should get approximately the same speed you get during a parity check/disk rebuild at that point, since disks are faster or slower depending on if you are writing to the outer or inner sectors.

This was my assumption as well, trying to figure out what is preventing my transfers from matching my parity check speeds.

 

14 hours ago, Frank1940 said:

 

It is my experience that this is only true when writing large files (say, >4GB).

All files were greater than 4GB.

7 hours ago, unraid_user11 said:

trying to figure out what is preventing my transfers from matching my parity check speeds.

Post diags during a large file transfer.

On 11/29/2024 at 4:07 AM, JorgeB said:

Post diags during a large file transfer.

Sorry for the late reply. Please see the attached diagnostics.

server-diagnostics-20241203-2330.zip

Is this over LAN or from a pool?

4 hours ago, JorgeB said:

Is this over LAN or from a pool?

NVME cache pool to HDD array using Unbalanced plugin. 

Try transferring a large file directly with pv:

 

pv /path/to/large/file/on/cache > /path/to/dest/in/the/array

 

Post the results

35 minutes ago, JorgeB said:

Try transferring a large file directly with pv:

 

pv /path/to/large/file/on/cache > /path/to/dest/in/the/array

 

Post the results

 

I moved a 20TB file using the following command in terminal:

 

pv "/mnt/cache_nvme/media/Movies 4K/Blade Runner 2049 (2017)/Blade Runner 2049 (2017) {imdb-tt1856101} [Bluray-2160p][DV HDR10][EAC3 7.1][x265]-hallowed.mkv" > "/mnt/disk2/media/Movies 4K/Blade Runner 2049 (2017)/Blade Runner 2049 (2017) {imdb-tt1856101} [Bluray-2160p][DV HDR10][EAC3 7.1][x265]-hallowed.mkv"

 

I would get write speeds of around 120MiB/s each on disk 2 and parity drive. Attached diagnostics below.

server-diagnostics-20241204-1130.zip

Run the diskspeed docker tests, both for the disks and the controllers tests, and post the results.

Did I do it correctly?

The disks yes, but do the controller test as well.

7 minutes ago, JorgeB said:

The disks yes, but do the controller test as well.

Is this it?

Yep, and everything looks normal, not sure what the issue could be, when writing with all disk involved looks like there's a hard limit at around 110MB/s, do you already have data in the array, or it could be blown up?

5 hours ago, JorgeB said:

when writing with all disk involved looks like there's a hard limit at around 110MB/s

So that I can learn, where can you see the hard limit of 110MB/s in the above data?

 

 

5 hours ago, JorgeB said:

do you already have data in the array, or it could be blown up?

Here is a screenshot of the main array and the usage of each drive.