Importing existing disks into unRAID

[Omnicrash]

I'm currently in the process of moving my data to unRAID. The hardware that is currently running my NAS setup will also be used for my unRAID setup.

 

It contains:

2x 4TB in RAID1

1x 3TB

 

I also bought 2 new 4TB HDD's to add to my new setup and before that, to import my old data.

 

Now my current plan is:

 

1. Set up VM using VMware Workstation, so I can still use my PC as well as my NAS during the transfer period

2. Pass-through unRAID USB drive, and attach new HDDs

3. Pre-clear the new drives

4. Import the existing data over network

5. Once that's done, shut down the old server

6. Install the new disks along with one of the old RAID1 disks in the server

7. Once everything is settled in, add the other RAID1 disk as parity, and add the 3TB as a cache

 

This way, there should still be a copy of all data at all times.

 

Now, currently using the VMWare Workstation pass-through, the disks aren't identified with their proper serial numbers, but rather as VMware_Virtual_SATA_Hard_Drive_000...1 etc. Will this be a problem when moving the disks to bare-metal? Or will they be recognised and imported?

[itimpi]

Now, currently using the VMWare Workstation pass-through, the disks aren't identified with their proper serial numbers, but rather as VMware_Virtual_SATA_Hard_Drive_000...1 etc. Will this be a problem when moving the disks to bare-metal? Or will they be recognised and imported?

It all depends on whether the VMWare installation correctly presents the drives or not.  When I last checked VMWare workstation had problems passing through physical disks larger than 2TB.  One way to test it would be:

• Connect one of the new 4TB disks to the VM running unRAID, and then copy some data onto it
  
• Put the drive plus the unRAID USB stick into the NAS and boot it off the USB stick
  
• Do a Tools->New Config and assign the disk you have just added with some data (ignoring the current NAS drives)
  
• Start the array and see if the data is still there.  If the disk had been correctly set up under VMWare workstation then it will be.
  

This could be done without running preclear just to see if VMWare workstation is handling the disks correctly so should not take long to do.

 

If the above does not work, then another possibility is:

• Plug the new 4TB disks into the PC and carry out any confidence checks. The manufacturers test software is probably OK for this and will run under Windows.
  
• Copy the data from the NAS onto these new disks using Windows Explorer.
  
• You can now boot the NAS under unRAID and repurpose the existing disks for unRAID purposes.  At this point you could probably use one of the 4TB disks as parity although you may prefer to only add the parity to unRAID later as copying is faster without parity assigned.
  
• Copy the data back from the new disks onto the unRAID managed disks.  This could be done over the network from the Windows system, or by plugging them into the unRAID server and using the Unassigned Devices plugin to mount them and then using a Linux command line tool (e.g. 'mc' or 'rsync') to copy the data locally within the server.
  
• If you have not yet set up a parity disk on the unRAID server, then as the disks are emptied they can simply be added to the unRAID server.
  
• If you have set up parity then they should be precleared before adding them to the server to minimise downtime.  As they are tested disks at this point you could skip the pre-read and post-read phases of the preclear to speed it up.
  

[Omnicrash]

When I last checked VMWare workstation had problems passing through physical disks larger than 2TB

 

Looks like that's working fine at least. Been pre-clearing the first disk since yesterday.

 

One way to test it would be:

• Connect one of the new 4TB disks to the VM running unRAID, and then copy some data onto it
  
• Put the drive plus the unRAID USB stick into the NAS and boot it off the USB stick
  
• Do a Tools->New Config and assign the disk you have just added with some data (ignoring the current NAS drives)
  
• Start the array and see if the data is still there.  If the disk had been correctly set up under VMWare workstation then it will be.
  

 

Good suggestion, I'll do that with a portion of the data to make sure.

 

If the above does not work, then another possibility is:

• Plug the new 4TB disks into the PC and carry out any confidence checks. The manufacturers test software is probably OK for this and will run under Windows.
  
• Copy the data from the NAS onto these new disks using Windows Explorer.
  
• You can now boot the NAS under unRAID and repurpose the existing disks for unRAID purposes.  At this point you could probably use one of the 4TB disks as parity although you may prefer to only add the parity to unRAID later as copying is faster without parity assigned.
  
• Copy the data back from the new disks onto the unRAID managed disks.  This could be done over the network from the Windows system, or by plugging them into the unRAID server and using the Unassigned Devices plugin to mount them and then using a Linux command line tool (e.g. 'mc' or 'rsync') to copy the data locally within the server.
  
• If you have not yet set up a parity disk on the unRAID server, then as the disks are emptied they can simply be added to the unRAID server.
  
• If you have set up parity then they should be precleared before adding them to the server to minimise downtime.  As they are tested disks at this point you could skip the pre-read and post-read phases of the preclear to speed it up.
  

 

This is also a great option because it means I can spread data over more drives (using high-water) instead of filling a single drive completely with existing data.

 

I have some followup questions to this approach:

1. Does unRAID (using either the Unassigned Devices plugin or a manual mount operation) support NTFS file systems (read-only will be fine)?

2. When copying the files, if I set up a parity, I should always copy from '/mnt/NTFSDISK' to '/mnt/SHARE' rather than directly to an unRAID disk, correct?

 

Anyway thanks for the input, this will help me out immensely!

[itimpi]

I have some followup questions to this approach:

1. Does unRAID (using either the Unassigned Devices plugin or a manual mount operation) support NTFS file systems (read-only will be fine)?

The unassigned devices plugin adds the ntfs-3g driver that provides full read/write access to NTFS disks.    Having said that I think there may have been a mention of Windows 10 formatted NTFS disks only providing read access for some reason, although I could be wrong about that.

2. When copying the files, if I set up a parity, I should always copy from '/mnt/NTFSDISK' to '/mnt/SHARE' rather than directly to an unRAID disk, correct?

You can actually do either.  Copying directly to an unRAID disk allows to control placement directly bypassing split level and allocation method on user shares.    Copying to the user share bring the split level and allocation method for the user share into play.

 

What you should not do is copy unRAID disk to User share as this can potentially lead to data loss if you are not careful.  This is because as Disks and user shares provide two different views of the same data you can end up trying to copy a file to itself which ends up truncating it to zero length (this is the so-called User share copy bug) and thus lead to data loss.  However as your NTFSdisk is outside the array it is not affected by this potential issue.

[Omnicrash]

You can actually do either.  Copying directly to an unRAID disk allows to control placement directly bypassing split level and allocation method on user shares.    Copying to the user share bring the split level and allocation method for the user share into play.

 

What you should not do is copy unRAID disk to User share as this can potentially lead to data loss if you are not careful.  This is because as Disks and user shares provide two different views of the same data you can end up trying to copy a file to itself which ends up truncating it to zero length (this is the so-called User share copy bug) and thus lead to data loss.  However as your NTFSdisk is outside the array it is not affected by this potential issue.

 

Thanks for clearing that up, the ability to manage files transparently on a disk level is really cool!

 

34% into step 2 of the pre-clear of disk 1 now, so it'll be a while before I can actually start testing stuff.

[Omnicrash]

Just a quick update, everything worked perfectly!

 

• Pre-cleared the disks using VMware Workstation using Physical Disk
  
• Transferred all my files to the virtual unRAID setup
  
• Put them in real hardware
  
• New Config
  
• Assigned the correct filled/cleared drives, and a parity
  

 

Now in the process of using an Ubuntu image to transfer other old files from a single drive that's now running in the VM (it's a mdadm drive so it won't mount using Unassigned Devices).

 

Now I just need to buy a decent USB flash drive first before buying an unRAID license and adding the rest of the drives. This one keeps crapping out on 'Loading bzroot...' about 50% of the time. And yes, it happens both in VM and on bare-metal.