Hello All,

 

i am new to unraid but been a NAS fanatic for many years, jumping from windows server to freenas, then openmediavault which i have been using for the past two years or so. i have decided to go with unraid and just bought a pro licence, now i have quite a lot of data on disks, around 70tb of it on 6tb reds and SEs, all disks are just ext4 formatted with no RAID etc as anything important is backed up elsewhere.

 

i have 4 new 6tb drives which i have loaded into my unraid box, precleared them and created a couple of shares and let the parity drive complete. i also have 2 240gb SSD drives in a cache pool. I have then loaded my drives into another machine with ubuntu on it one proceeded to move the data drive by drive to the new unraid server via the network, however i am only getting 15-25mbps transfer rate so its giving me a transfer time of around 11 hours per TB, ive read about disabling the cache pool to try and troubleshoot but the transfer rate seems to be the same. both machines have gigabit ETH single connections to my router.

 

my question to everyone is,

 

can i do this any quicker, i understand i have lost some performance but it will take me weeks to move the data onto the new unraid server. can i mount the drives locally and use terminal or midnight commander to move them? will unraid read EXT4 partitions? or can i set up a VM of linux and mount the drives via USB to move, would i get any performance increase on this. i am using SMB to transfer from linux as i have always had permission issues with NFS.

 

i dont have any plugins yes and am using 6.2 rc4

 

thanks for your help in advance.

I would suggest that you install the Unassigned Devices plugin to handle drives connected via USB.  I am reasonably certain that ext4 will be supported.

 

You might also want to first install the Community Applications plugin as that tends to be the best way to handle installing and managing other plugins.

If the speed is the same with or without cache the issue may not be the unRAID server, but you can also try turning turbo write on for the initial copy, with just 4 recent disks write speed should be at or close to gigabit.

 

Settings -> Disk Settings -> Tunable (md_write_method): reconstruct write

The quickest way would be to mount the drives in the unRAID box or use USB 3.0 and mount them with Unassigned Devices (plugin). Turn on turbo write, turn off the caching of shares (to avoid the SSD capacity) and begin the copies via rsync (research).

wow thanks for the quick responses on this. i have seen the communities app for the plugins and dockers, i must say unraid is very user friendly

 

i have tried copying files to a share on a windows desktop and its a lot quicker so i know its not the source machine. i will get the unassigned devices plug in later on and turn on the turbo write as suggested, happy to create an rsync job to copy the data then verify and trash the original drive and add to pool rinse and repeat

 

i take it the parity build wont take 11 hours each time i add a disk?! 

 

will come back and let you know how i get on,

 

thanks

 

 

 

 

 

 

i take it the parity build wont take 11 hours each time i add a disk?!

It will if you take the approach of building parity every time.

 

You can add drives to unraid without rebuilding parity if the drive is all zeroes, so it doesn't effect the parity calculation. Two ways to do this. Either use the third party preclear utility, which is used on a drive not assigned to the array but connected to the unraid box, or simply add the drive and allow unraid to clear it before it is added. The drive will not be available to write new data until the process is done, but you can still use the rest of the array.

 

 

i take it the parity build wont take 11 hours each time i add a disk?!

It will if you take the approach of building parity every time.

 

You can add drives to unraid without rebuilding parity if the drive is all zeroes, so it doesn't effect the parity calculation. Two ways to do this. Either use the third party preclear utility, which is used on a drive not assigned to the array but connected to the unraid box, or simply add the drive and allow unraid to clear it before it is added. The drive will not be available to write new data until the process is done, but you can still use the rest of the array.

Just to elaborate on this a bit, preclear is available on this forum. It is the usual way to both test and clear disks to add.

 

The time required to build parity is pretty much dependent on the size of the parity disk, so you could expect it to take about the same amount of time.

 

But as already said, rebuilding parity is not the way to add a disk. Clearing the disk (setting it to all zeros) allows it to be added without affecting parity.

i take it the parity build wont take 11 hours each time i add a disk?! 

When you add a disk to an array with active parity protection, then the parity is not rebuilt.    Instead the new disk is 'cleared' by unRAID so that it can be added without affecting parity. 

 

You can also use the pre-clear plugin to carry out this clear action before adding the drive into the array so that unRAID does not need to do the clear.  This takes longer than the unRAID clear but this is because it also does  read tests which has the added advantage of carrying out a confidence check on the drive, but maybe with existing trusted drives that is not so important.

i read about preclearing and used this plugin when i put the new drives in, it took a long time for one pass but read its a good indicator. as this is a new system i did not have a parity disk or any in the array, once ive copied the data from the drives, they are free to be wiped by unraid / using the pre-clear tool when added so this should be fine.

 

 

i read about preclearing and used this plugin when i put the new drives in, it took a long time for one pass but read its a good indicator. as this is a new system i did not have a parity disk or any in the array, once ive copied the data from the drives, they are free to be wiped by unraid / using the pre-clear tool when added so this should be fine.

If you are setting up a new array then it does not matter if the disks are already cleared or not.  UnRAID will add them 'as is' and then do a simple format to the desired file system.

i read about preclearing and used this plugin when i put the new drives in, it took a long time for one pass but read its a good indicator. as this is a new system i did not have a parity disk or any in the array, once ive copied the data from the drives, they are free to be wiped by unraid / using the pre-clear tool when added so this should be fine.

If you are setting up a new array then it does not matter if the disks are already cleared or not.  UnRAID will add them 'as is' and then do a simple format to the desired file system.

And to elaborate on this a bit, a disk only needs to be clear when it is added to a new data slot in an array that already has parity, so parity will remain valid. Any other scenario such as adding a bunch of disks before parity is built, or replacing a disk for rebuild, does not require a clear disk.

ok so in this system so far i now have 4 disks x 6tb giving me 18tb usable space and 1 6tb for parity.  these are all new disks never been used, i have used the preclear tool to wipe the disks, it took about a day to do each one.

 

i am aiming to clear the data on the used disks one at a time (so the first disk will go onto disk 1 of my array in a user share set up to fill disk 1 first) then add them to my array to grow the array

 

example would be disk 4 (or disk 4 of my array)  so does this disk need preclearing or wiping before adding to slot 5 or can i just add it and tell unraid to format it as xfs and it will add to the array automatically or will i need to do something first like i can just run DBAN and zero the disk on another system if needbe then add it to the array.

 

my aim is to have 17 x 6tb disks in the array 15+2 parity but will want to increase this when the time comes,

 

i only have 1 parity disk at the minute, should i add a second as soon as possible or does it not matter?

 

ok so in this system so far i now have 4 disks x 6tb giving me 18tb usable space and 1 6tb for parity.  these are all new disks never been used, i have used the preclear tool to wipe the disks, it took about a day to do each one.

 

i am aiming to clear the data on the used disks one at a time (so the first disk will go onto disk 1 of my array in a user share set up to fill disk 1 first) then add them to my array to grow the array

 

example would be disk 4 (or disk 4 of my array)  so does this disk need preclearing or wiping before adding to slot 5 or can i just add it and tell unraid to format it as xfs and it will add to the array automatically or will i need to do something first like i can just run DBAN and zero the disk on another system if needbe then add it to the array.

 

my aim is to have 17 x 6tb disks in the array 15+2 parity but will want to increase this when the time comes,

 

i only have 1 parity disk at the minute, should i add a second as soon as possible or does it not matter?

It would probably be less confusing to refer to disk4 as slot4, and refer to parity as disk0 (syslog calls it that) or slot0. Parity2 actually gets called disk29 in syslog.

 

If you already have parity built, unRAID will require a clear disk to add to a new data slot; i.e., any slot that doesn't already have a data disk in it. If you don't preclear it unRAID will clear it when you add it. And unRAID will not recognize a clear disk if you use anything else to zero the disk. It is expecting a "clear signature" to be present on the disk when it is added or it will begin clearing it. The signature is necessary else how would it know the disk was already zeroed without reading the whole thing?

hi thanks for the response, yes apologies i was looking at the array after i sent that on the webgui and it shows

parity

disk1

disk2

disk3

 

so disk4 would be the first new disk that once i have copied the data to the array via rsync.

 

i understand that unraid does something to the drives hence why you can not just take xfs drives from say a linux distro and mount them.

 

i had a similar thing with openmediavault, if i didnt put an unformatted drive in the system and allow the system to create it, i had issues with things

 

and you are saying that even if i used something else to wipe the disks first, unraid will still want to do this first

 

i have the parity built so i will add a test 2tb disk i have that has old unwanted data on, you are saying unraid will then clear it, and format it to xfs (as thats my chosen FS) and add it to the array without the parity having to be rebuilt?

 

i understand that unraid does something to the drives hence why you can not just take xfs drives from say a linux distro and mount them.

 

i had a similar thing with openmediavault, if i didnt put an unformatted drive in the system and allow the system to create it, i had issues with things

 

and you are saying that even if i used something else to wipe the disks first, unraid will still want to do this first

 

i have the parity built so i will add a test 2tb disk i have that has old unwanted data on, you are saying unraid will then clear it, and format it to xfs (as thats my chosen FS) and add it to the array without the parity having to be rebuilt?

If the partition structure matches what unraid expects, then you can simply mount the drive in an array and use it. The issue is that once parity is generated, the disk must be zeroed to keep parity valid and add the disk. So, if wanted to add a disk that was properly partitioned and formatted, you would set a new config, telling unraid to ignore the parity information and rebuild it from the current data disks.

 

Wiping the disk with DBAN or simply writing zeroes to the disk is not enough to add the drive, unraid would still have to verify that the disk was blank, it won't blindly trust you. The preclear utility adds a signature to the disk after it is done zeroing that unraid trusts.

 

Yes, if you add a disk to a current parity protected array, it will be cleared, added to the array, then formatted with whichever filesystem you chose.

thats great thanks for clarifying