TheSkaz Posted September 21, 2021 Share Posted September 21, 2021 Please dont kill me.... Is there anything that would allow (in the works, or currently exists) the mover to multithread the movement of documents to the spinning rust? Scenario: I have a 1TB NVME cache drive and 10ish 16TB drives in an array. The cache drive is 100% full and the mover is active. it seems to grab files one at a time and move them. So my overall speed to move the files are reduced to the write speed of a single drive. since my drives are set as "most free" it will copy the next file to the next drive. if it could do them in parallel, that would be tons faster than it currently is. is this something that is possible or feasible? Quote Link to comment
JonathanM Posted September 21, 2021 Share Posted September 21, 2021 Just now, TheSkaz said: So my overall speed to move the files are reduced to the write speed of a single drive. since my drives are set as "most free" it will copy the next file to the next drive. if it could do them in parallel, that would be tons faster than it currently is. Writes to the parity array will always be limited by the speed of the parity drive(s). Not much point in trying to write to multiple data drives simultaneously when the parity drive is the bottleneck. Quote Link to comment
TheSkaz Posted September 21, 2021 Author Share Posted September 21, 2021 did not consider that... ok then. thanks! Quote Link to comment
IronBeardKnight Posted August 23, 2023 Share Posted August 23, 2023 On 9/22/2021 at 7:05 AM, JonathanM said: Writes to the parity array will always be limited by the speed of the parity drive(s). Not much point in trying to write to multiple data drives simultaneously when the parity drive is the bottleneck. While this may be true for cache > array but what about array > cache would it still be the same limitation if the share is spread over multiple drives? Quote Link to comment
JonathanM Posted August 23, 2023 Share Posted August 23, 2023 1 hour ago, IronBeardKnight said: While this may be true for cache > array but what about array > cache would it still be the same limitation if the share is spread over multiple drives? Not sure what you are asking. If the write destination is to the parity array drives, it's even worse if the share is spread out. Each write to the individual data drives in the parity array is also calculated and written to the parity drive. So if you write to disk1 and disk2 at the same time, the parity drive is thrashing back and forth to the 2 physical locations that map to the data written to the 2 different disks. Pools have no such limitation, the parity array parity disk is not touched for pool writes. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.