The total copying speed is very slow when using commands to copy data simultaneously on multiple disks in Unraid, and cannot find the reason

[LJQ]

Unraid version: 6.11.5/6.12.0-rc2

Background:

I've set up an array with 24 disks.
Among them, there are 8 old hard drives, disk1-disk8, each 8TB, xfs system, filled with old data.
The other disks, disk10-disk17, are new hard drives, each 12TB, xfs/zfs system, with no data.
I want to copy all the data from disk1-disk8 to disk10-disk17 simultaneously.

Problems encountered:

No matter what method I use to copy data, the total copy speed for multiple disks is 100-300MB, far below the disk read and write speed.
Copying from a single old hard drive to a new hard drive can reach 250MB+, but the speed cannot be increased when copying from multiple hard drives simultaneously, and it may even decrease.

Tried methods (ineffective):

Running multiple rsync background processes to sync data simultaneously, such as:
nohup rsync -avP --append /mnt/disk1/ /mnt/disk10 > rsync1.log 2>&1 &
nohup rsync -avP --append /mnt/disk2/ /mnt/disk11 > rsync1.log 2>&1 &
Running multiple cp background processes to sync data simultaneously, such as:
nohup cp -R /mnt/disk1/* /mnt/disk10/ > cp1.log 2>&1 &
nohup cp -R /mnt/disk2/* /mnt/disk11/ > cp2.log 2>&1 &
Formatting disk10-disk17 as xfs system, then performing a full data copy
Formatting disk-disk17 as xfs system, then performing a full data copy
Downgrading unraid to version 6.11.5 or upgrading to 6.12.0-rc2;
Adjusting the process priority of rsync/cp commands using nice/renice.
Adjusting zfs_arc_max from the default 32GB to 128GB.

Main host configuration:
LSI2308 12GB direct-attached card, 258GB memory, AMD Epyc 7551P CPU
It shouldn't be a CPU, memory, or disk bottleneck

The only effective solution (effective):
Parallel dd commands can achieve speeds of 1.5GB-1.7GB

nohup dd if=/dev/sdh of=/dev/sdx bs=128K status=progress > dd1.log 2>&1 &
nohup dd if=/dev/sdi of=/dev/sdy bs=128K status=progress > dd2.log 2>&1 &
nohup dd if=/dev/sdj of=/dev/sdr bs=128K status=progress > dd3.log 2>&1 &
nohup dd if=/dev/sdg of=/dev/sds bs=128K status=progress > dd4.log 2>&1 &
nohup dd if=/dev/sdc of=/dev/sdt bs=128K status=progress > dd5.log 2>&1 &
nohup dd if=/dev/sdd of=/dev/sdu bs=128K status=progress > dd6.log 2>&1 &
nohup dd if=/dev/sde of=/dev/sdv bs=128K status=progress > dd7.log 2>&1 &
nohup dd if=/dev/sdf of=/dev/sdw bs=128K status=progress > dd8.log 2>&1 &

I will upload some screenshots and logs.

Supplement;
1.Previously, I started multiple rsync processes on a TrueNAS machine, and copied data to another Unraid machine's disk1-disk4 through a 10Gb network. The speed could reach 600MB+ without any bottleneck observed.
2.Also, I started multiple rsync processes on an Unraid machine, and copied disk1-8 data to another TrueNAS machine through a 10Gb network. The speed could reach 700MB+ without any bottleneck observed.

ljq-unraid-diagnostics-20230403-1751.zip

[JorgeB]

49 minutes ago, LJQ said:

Parallel dd commands can achieve speeds of 1.5GB-1.7GB

nohup dd if=/dev/sdh of=/dev/sdx bs=128K status=progress > dd1.log 2>&1 &
nohup dd if=/dev/sdi of=/dev/sdy bs=128K status=progress > dd2.log 2>&1 &
nohup dd if=/dev/sdj of=/dev/sdr bs=128K status=progress > dd3.log 2>&1 &
nohup dd if=/dev/sdg of=/dev/sds bs=128K status=progress > dd4.log 2>&1 &
nohup dd if=/dev/sdc of=/dev/sdt bs=128K status=progress > dd5.log 2>&1 &
nohup dd if=/dev/sdd of=/dev/sdu bs=128K status=progress > dd6.log 2>&1 &
nohup dd if=/dev/sde of=/dev/sdv bs=128K status=progress > dd7.log 2>&1 &
nohup dd if=/dev/sdf of=/dev/sdw bs=128K status=progress > dd8.log 2>&1 &

This is likely because you are bypassing the md driver, try the same with /dev/md1 to /dev/md10, /dev/md2 to /dev/md11, etc

 

 

 

 

[LJQ]

1 hour ago, JorgeB said:

This is likely because you are bypassing the md driver, try the same with /dev/md1 to /dev/md10, /dev/md2 to /dev/md11, etc

 

 

 

 

Your meaning is——Using the dd command bypasses the md driver, allowing for maximum read and write speeds across multiple disks, while rsync and cp commands still operate disks through the md driver, resulting in significant performance limitations？Based on this, Can we draw the following conclusions:
1.Unraid's use of the md driver may cause a bottleneck when reading and writing across multiple disks simultaneously, with total write speeds only reaching 200-300MB/s;
2.When transferring data via a 10Gb network, the speed of reading from multiple Unraid disks and writing to TrueNAS, or reading from TrueNAS and writing to multiple Unraid disks, is faster than direct read and write speeds between Unraid disks. This may be due to partially bypassing the md driver through the network?
3.IF the array is stopped and XFS file system disks are manually mounted, followed by using rsync to transfer data across multiple disks, this may also bypass the md driver, resulting in higher transfer speeds? owever, will this operation have any hidden risks?

[JorgeB]

18 minutes ago, LJQ said:

Your meaning is——Using the dd command bypasses the md driver

No, using /dev/sdX bypasses the md driver, try dd the same but with /dev/md1 for disk1, /dev/md2 for disk2, etc

[LJQ]

17 minutes ago, JorgeB said:

No, using /dev/sdX bypasses the md driver, try dd the same but with /dev/md1 for disk1, /dev/md2 for disk2, etc

Thank you for your patience. In my previous post, I mentioned using the dd command to directly copy disk sdh (8TB) to disk sdx (12TB) and then expand disk sdx, which resulted in Unraid not recognizing sdx. You responded to that as well. 
If I perform the following operations:
nohup rsync -avP --append /dev/md1p1 /mnt/md10p1 > rsync1.log 2>&1 &
nohup rsync -avP --append /dev/md2p1 /mnt/md11p1 > rsync2.log 2>&1 &
.....
I will not bypass the md driver, and there should be no speed bottleneck for copying multiple disks? However, after expanding the sdx (12TB) disk using fdisk and xfs_growfs, Will this cause Unraid to still not recognize it?

[JorgeB]

Using dd with /dev/mdX was just to confirm if the issue really is the md driver, like I suspect, but the destinations devices would not be usable in the array, if they are different capacities, you are reading and writing at speeds much higher then usual, since there's no parity, only reading, or only writing should be much faster, like the result you had when transferring over LAN.

 

Another way you can test, and this way the copy will be valid, is to create multiple single device pools and temporarily assign the destinations disks as a single pool members, then copy from disk1 to poolone, disk2 to pooltwo, etc, if the md driver was the problem performance should be much better, when the copy is done you can reassign the pool disks to the array.

[LJQ]

35 minutes ago, JorgeB said:

Using dd with /dev/mdX was just to confirm if the issue really is the md driver, like I suspect, but the destinations devices would not be usable in the array, if they are different capacities, you are reading and writing at speeds much higher then usual, since there's no parity, only reading, or only writing should be much faster, like the result you had when transferring over LAN.

 

Another way you can test, and this way the copy will be valid, is to create multiple single device pools and temporarily assign the destinations disks as a single pool members, then copy from disk1 to poolone, disk2 to pooltwo, etc, if the md driver was the problem performance should be much better, when the copy is done you can reassign the pool disks to the array.

Alright, in order to copy the data from the 8 old disks to the 8 new disks, I have tried too many methods and spent a lot of time. 
I are now put the 8 old disks back to another host and transferring the data to the new disks in Unraid via a 10-gigabit local area network. 
Once I have finished transferring all the data, I will find time to test the read and write issues of the Unraid array disks using the methods You suggested, and will update the thread with any new findings. 
Thank you again for your patient guidance.

[LJQ]

12 hours ago, JorgeB said:

Using dd with /dev/mdX was just to confirm if the issue really is the md driver, like I suspect, but the destinations devices would not be usable in the array, if they are different capacities, you are reading and writing at speeds much higher then usual, since there's no parity, only reading, or only writing should be much faster, like the result you had when transferring over LAN.

 

Another way you can test, and this way the copy will be valid, is to create multiple single device pools and temporarily assign the destinations disks as a single pool members, then copy from disk1 to poolone, disk2 to pooltwo, etc, if the md driver was the problem performance should be much better, when the copy is done you can reassign the pool disks to the array.

After testing various scenarios, the following results were observed:

1.Unraid A array (disk1-disk4) reading simultaneously == parallel rsync over 10G LAN ==> Unraid B (disk10-disk13) writing simultaneously; no noticeable bottlenecks for both reading and writing.
2.Unraid B cache pool (disk1-disk4) reading simultaneously == local ==> Unraid B cache pool (disk10-disk13) writing simultaneously; no noticeable bottlenecks for both reading and writing.
3.Unraid B array (disk1-disk4) reading simultaneously == local ==> Unraid B (disk10-disk13) writing simultaneously; no noticeable bottleneck for reading, but a significant bottleneck for writing.

From these results, we can maybe draw the following conclusions:

1.When multiple drives in an Unraid array are reading and writing simultaneously, there may be a bottleneck for writing, resulting in a total writing speed of 200-300MB/s.
2.When multiple drives are writing to or reading from Unraid through the network, no noticeable bottleneck is observed.
3.When multiple cache pools are set up in Unraid and read and write operations are performed simultaneously between them, no noticeable bottleneck is observed.
4.When using the dd command to directly operate on dev/sd* and bypass the array's md driver, there is no speed bottleneck.

This may reveal that the implementation of Unraid arrays can potentially lead to bottlenecks when a large number of disks are writing simultaneously within the array.

[JorgeB]

5 hours ago, LJQ said:

This may reveal that the implementation of Unraid arrays can potentially lead to bottlenecks when a large number of disks are writing simultaneously within the array.

Yes, something to look at, but for normal usage, i.e., with parity you are not able to hit those bottlenecks, so not something that most users will ever run into, changing these settings might also make a difference (Settings -> Disk Settings):

 

 

Thanks for the report, I'll do some tests myself when I have some time.