Hello!

 

I have an array set up and I'm working on moving files from my other drives onto the array, then wiping the drive, then installing the drive to the array. I have about 100TB I need to get onto the array. So the process I'm following is:

1. Plug the drive into my DAS that is connected via sata to SF8088 to an HBA on my server
2. Use unassigned devices to mount the drive.

3. use Krusader to move the files to my share.

4. clean the drive and add it to the array

5. repeat for next drive.

I have two questions. the first is that my speeds started around 140MB/s which I think from what I read is typical. They are now around 30MB. It is a bunch of smaller files so maybe thats fairly normal? Moving them across the network via a smb share my speeds are about the same, although I haven't tested hardwire yet. I plan to soon.

the second question is can I do both at the same time to maximize efficiency? Or will moving via one slow down the other in a proportional amount? I could move one drive over the network and another via krusader but I just didn't know if that worked or not.
 

Thanks for the help!

Edited July 16, 20241 yr by JakeW
spelling

Do you have parity installed?

I do. Just finished a 36 hour parity sync. 

3 hours ago, JakeW said:

now around 30MB

This is typical for parity writes unless Turbo Write is enabled.

 

https://docs.unraid.net/unraid-os/manual/storage-management/#array-write-modes

 

And trying to write from multiple streams will actually make things worse since each write has to also write parity.

 

 

Another possibility is to unassign parity until initial load is completed. We often recommend just this. Then parity will rebuild when you reassign it.

 

3 hours ago, JakeW said:

clean the drive and add it to the array

Do you have another copy of anything important and irreplaceable? Parity is not a substitute for backups.

Hey thanks for all the replies! I'll turn on turbo write and turn off parity while I do the initial load. I think what you said there was that I'll just have to rerun the parity sync after loading, but that's not a big deal since I can use it while its running. Although with 100TB more on there I assume the parity sync will take much longer than 36 hours next time. 

I don't have other backups, but this information isn't SUPER important. I don't want to have to find and collect it all again, which is why I have a little redundancy going in place, but I haven't justified the cost of long term backups for data I can find again should I absolutely have to. 

Thanks again! 

Just now, JakeW said:

Although with 100TB more on there I assume the parity sync will take much longer than 36 hours next time. 

The time for the parity sync is determined primarily by the size of the largest parity drive and not the amount of data on the array so it should not change.  Parity works at the raw sector level and has no idea what the content of any sector means.

That's great to hear. I asked chatgpt that same question when I couldn't find the right google search to give a result, and it told me otherwise, which is why I did the parity sync first originally. THANK YOU!

1 hour ago, JakeW said:

turn on turbo write and turn off parity

turbo write has no effect when you have no parity since turbo write is only about how parity is updated. See the link I posted earlier.

 

1 hour ago, JakeW said:

this information isn't SUPER important

You get to decide what is important and irreplaceable. Many of us don't backup everything, including me.

Hey one more follow up. If I turn off the parity, would doing one drive over LAN, and then another directly plugged into the server using krusader to move the files speed things up at all? I feel like I hit drive speed and LAN speed bottlenecks before I hit theoretical write speed issues so maybe it would be faster?

1 hour ago, JakeW said:

Hey one more follow up. If I turn off the parity, would doing one drive over LAN, and then another directly plugged into the server using krusader to move the files speed things up at all? I feel like I hit drive speed and LAN speed bottlenecks before I hit theoretical write speed issues so maybe it would be faster?

 

If you have no parity then writing to different drives can happen in parallel without affecting speed as there is then no contention for a shared drive.