Migrating 64 TB's of material from Windows NTFS to Unraid


RAP2

Recommended Posts

Gotcha... Lesson learned.

 

This particular drive had three primary folders that were each there separate Windows shares.  So, I did each copy one after the other - so they overlapped. 

 

The rest of the 7 drives will be one drive at a time - one copy at a time - followed by re-activating Parity after each one.  Those other  7 drives do not have back-ups, but they are somewhat less critical - but frankly its why I'm moving to UNRAID.  My plan now is to resume parity after each drive is added.  If I get read errors on the source disks, not much I can do about that.

 

Except for the parity drive (WDC Gold 10TB), all my drives are Ironwolf NAS 8TB drives; which I believe are CMR.

 

Thanks again Frank1940 - and itimpi - for your guidance.

Link to comment
24 minutes ago, RAP2 said:

Gotcha... Lesson learned.

 

This particular drive had three primary folders that were each there separate Windows shares.  So, I did each copy one after the other - so they overlapped. 

 

The rest of the 7 drives will be one drive at a time - one copy at a time - followed by re-activating Parity after each one.  Those other  7 drives do not have back-ups, but they are somewhat less critical - but frankly its why I'm moving to UNRAID.  My plan now is to resume parity after each drive is added.  If I get read errors on the source disks, not much I can do about that.

 

Except for the parity drive (WDC Gold 10TB), all my drives are Ironwolf NAS 8TB drives; which I believe are CMR.

 

Thanks again Frank1940 - and itimpi - for your guidance.

Parity in Unraid is real time so once you have a parity drive present it will start using it.   You do not gain anything much by pausing a build of parity while copying.   You should either let the parity build complete before doing any further copying (so your array is protected), or remove the parity drive from the array until copying is complete and add it afterwards (if you want to maximise copy speed).

Link to comment

@ itimpi:  Gotcha.  So instead I turned parity on.  It completed this evening:  

 

Duration: 8 hours, 3 minutes, 54 seconds. Average speed: 344.5 MB/sec - for a 10TB WDC Gold.

 

Still 3 Days for the copies to complete.  Once that is done, I will test shares on my Network.

 

Then I want to install Windows 10 as a VM on the same machine - to eventually replace the original Windows 10 installation.  Once that is all working then I will start migrating one drive at a time, the other 8TB drives.   I'll probably be back here to continue my education... 🙂

 

 

  • Like 1
Link to comment

The speed + time quoted for the parity build will be optimistic as Unraid will give the wrong answer when you are manually pausing and resuming the parity build.  It will assume you did the whole drive during the last increment when calculating the speed and time.

 

You may want to consider installing the Parity Check Tuning plugin so that parity checks only run outside prime time.  As a side effect it will give the correct duration + speed of the check as it takes into account pauses and resumes.

Link to comment

I will install the Parity Check Tuning plug-in.  Thanks!

 

I note - and to your comment about multiple copy processes - that as soon as I was down to one copy, the process was highly accelerated.  It went down from 2 days to 10 hours!  I won't make that mistake again.

 

I do wish there was a way to queue copy processes (not just here, but in Windows) - where for example, holding a modifier down and click drag to copy/move, will queue subsequent processes - rather then doing them in parallel... but I digress. 🙂

 

So this evening that first copy should be complete; and I will test my shares on my Windows network. 

 

I have a question:  One of the folders that I copied had an exclamation mark which has been replaced by  an underscore.  Is it a problem to change the share and folder name by removing the underscore?

Link to comment
48 minutes ago, RAP2 said:

 

I have a question:  One of the folders that I copied had an exclamation mark which has been replaced by  an underscore.  Is it a problem to change the share and folder name by removing the underscore?

 

The Explanation Point---   !   --- was removed because it is an operator in the Linux OS.

 

You can remove the Underscore if you want to.  Now a quick word of explanation about share names.  There are two ways to create a share a in Unraid.

 

First, you can use the Shares tab in the GUI.

 

Second, you can create a directory in the root of any hard drive that is on the array.

 

Either of these actions will create a share.  Now, a quick word about naming shares, directory/folder, or files that will be stored on Unraid.  Avoid symbols (including spaces--- I use an underscore to replace spaces) if you can.  If you don't, you will have issues if you have to manipulate these files later using the Linux OS.  There is an 'escape' character---   \   --- that will address the issue but it a kludge.

 

Another  point, Win10 does not like the default security setup that Unraid SMB starts out with.  You would be best served by setting up Unraid and Windows to use what Win10 expects and wants.  See here for how to do that:

 

    https://forums.unraid.net/topic/110580-security-is-not-a-dirty-word-unraid-windows-10-smb-setup/?tab=comments#comment-1009109

 

Link to comment

Thanks Frank1940! 🙂 

 

I prefer to not spaces at all with any shares or volume names - removing that symbol/underscore is perfect.  Thanks for that link - I've been already reading security issues with SMB1.  So I knew I was going to drill down on that once I started shares.

 

My Windows Server had been hacked in the past (cryptocurrency miner) - so I take security pretty seriously... its why I asked about the root account above.

Link to comment

OK - so only accessed via my intranet?  

 

I'm not sure how that is not a security risk.  If someone somehow hacks into WiFi (not likely given my access policies - but I'm speaking in general now), UNRAID access has a well-known user:  root.

 

Perhaps no access to stored files but certainly to wreak havoc. Am I missing something?

Link to comment
12 minutes ago, RAP2 said:

I'm not sure how that is not a security risk.  If someone somehow hacks into WiFi (not likely given my access policies - but I'm speaking in general now), UNRAID access has a well-known user:  root.

 

 

The risk is there.  You only protection is to use a long secure password (> 15 characters) to make gaining access very , very difficult. 

 

IF you install the Fix Common Problems plugin, you will find this setting:

Quote

Number of allowed invalid logins per day - This is the number of "grace" invalid logins allowed per day either via the local console or through SSH / Telnet (ie: you typed your password wrong) This is used to determine if any hacking attempts are being made on your server

 

You can also turn off Telnet and SSH logins via the   Settings >>  Management Access 

 

I know of no way (at this time) to block access to the standard GUI after a number of unsuccessful attempts.  There is a thread (there may be more) in the 'Feature Request' forum about this topic.  You may want to add your opinion on this topic there:

 

    https://forums.unraid.net/topic/84628-sane-security-defaults/?tab=comments#comment-784646

 

 

Link to comment

OK... my shares are working fine on my network.  A bit concerned on performance - but I'll drill down in that later.

 

What I'd like to do is install a Windows 10 VM on this machine.

 

Currently, I have an 500GB NVMe that is partitioned into 3 sections - one is the working Windows 10 installation that I will try to replace as a VM.  The other are 2 similar sized NTFS partitions.  I've mounted this drive and so I can see it under Unassigned Devices.  Its strange but each of the partitions (4 of them for Windows 10 - reserved, back-up, etc.) are all showing as 0 Bytes.  I'm guess because UNRAID really does not deal with NTFS...

 

Anyway - looking at this, I'm thinking I can't use this drive at all; unless the entire drive is formatted as XFS - and my entire approach to converting this machine hits a wall.

 

If this is the case, can I install a Windows 10 VM on a spinning disk (that is currently getting pre-cleared for my next data drive), and then once I get it set-up and working, move the VM to the NVMe?

Link to comment
5 hours ago, RAP2 said:

Anyway - looking at this, I'm thinking I can't use this drive at all; unless the entire drive is formatted as XFS - and my entire approach to converting this machine hits a wall.

You could simply pass the entire drive through to the VM and it should boot into Windows as long as you set the connection type to SATA.  If you add the virtio drivers to the windows system you can then use that connection type as well.

 

The only obvious reason to reformat it to XFS would be because you want to use vdisk image files for the VM rather than use the raw drive directly.

 

5 hours ago, RAP2 said:

Currently, I have an 500GB NVMe that is partitioned into 3 sections - one is the working Windows 10 installation that I will try to replace as a VM.  The other are 2 similar sized NTFS partitions.  I've mounted this drive and so I can see it under Unassigned Devices.  Its strange but each of the partitions (4 of them for Windows 10 - reserved, back-up, etc.) are all showing as 0 Bytes.  I'm guess because UNRAID really does not deal with NTFS..

Not sure why this should be happening as the UD plugin should handle standard NTFS drives fine.

Note however if passing the entire drive through to the VM you want to set the "PassThru" setting f/r the drive in UD so it does not try to mount it.

Link to comment

@ itimpi:  Thanks-you.  That's a relief.

 

I don't a need vdisk.  However, I cannot find anything in the docs that talk about "pass the entire drive through to the VM" - pass through talks about various things, graphics, USB, PCE, etc.  But no hard drives.  I also cannot see the "PassThru settings" either for the drive it self - or in settings for Unassigned Devices.  The drive is currently not mounted.  Do I need to mount first?

 

In my VM template:  Both HVM and IOMMU are  enabled.  I currently have a Windows 10 template using Q35-5.1 and OVMF.  The VirtIO drivers are downloaded and available - as is a Windows ISO.  (But as I've read your message, I can use my existing installation; which would be awesome!)  Network bridge is set to public (br0) I also passed a local USB mouse.

 

I just don't see anything in the template that passes the existing NTFS drive...

 

A link to some prerequisite learning would be cool - I can't seem to find references in the manual on this.

 

BTW:  I tried to start this VM to see how far I could go - but had some strange problems going into some old school BIOS looking screen - and never managed to get to the Windows 10 installation.  I had to force stop the VM - should I delete this one?

 

 

 

 

Edited by RAP2
typo
Link to comment

OK - been doing some reading of other having similar problems and made some changes in my VM Template:

image.thumb.png.5b359847ee343da085d6cabfcb9adea4.png

 

First I mounted the NVMe drive in UD.  Then wetn to the template and manually set the location to the Windows partition that is now available via UD.  I left the vDisk blank, the type to the default raw and changed the Primary vDisk Bus to SATA.

 

Then I click on UPDATE.  The page currently says "updating" - but I'm afraid its been stuck like that for some time.  Its either crashed or really taking its time.

 

Rob

Edited by RAP2
typo/clarification
Link to comment

Well 15 minutes later - still saying "Updating..." -  so I clicked on the VMs menu and then hit edit on the Windows 10 template to review.  Looks like it took the revisions; not sure why it hung in that state.  So next, I tried to start the VM.

 

First, I got the UEFI Interactive Shell window... ESC or other keys did not really do anything - but I was able to type exit; it then went to an old school BIOS setting window as before (Standard PC (Q35 + CH9, 2009) 

 

Reset dump me back to the UEFI IS v2.2... and I am in now "living in a mobius strip" - I think that should be the name of a song!

 

Ideas?

 

 

Link to comment

Full error message here:
 

VM creation error

error: Failed to attach device from /tmp/hotattach.tmp error: internal error: cannot parse product 0x1939#remove , error: Failed to attach device from /tmp/hotattach.tmp error: internal error: cannot parse product 0x1939#remove error: Failed to attach device from /tmp/hotattach.tmp error: internal error: cannot parse product 0x0029#remove

Link to comment

@ itimpi:  That has been changed... see directly above your post:  My understanding is that is only for spinning disks and I am using an NVMe.  It was an early error I made watching Space Invader One's video.

 

Currently, the VM has been started - but I can't seem to RDP to it.  There is supposed to be a place to "enable RDP" for the VM - but I can't seem to locate this.  Also, I'm no even sure the IP address of the VM!  How do I determine this?

Link to comment

OK - After some reading, I've removed the VM.  I've also removed the manual stubs for the NVMe drive that I wanted to pass-through; but even after removal, System Devices showed it was still bound - I had to stop the array and then reboot UNRAID.  Back on-line, SysDev no longer had the NVMe bound - and it now magically shows up again under Unassigned Devices.  Yea!

 

However: does any one have instruction on who to do the dual-boot Windows 10 pass-thru under 6.9.2?   I've realized that as a newbie, following these videos is risky, unless one is working with  the exact same version of software.  Setting up a VM in UNRAID is not straightforward - lots of gotcha's.  Docs do not explain process and videos can be problematic unless you are using one that matches your version of UNRAID.

 

I had no problem setting up Windows 10 and Ubuntu on Virtual Box a couple of years ago.  Maybe the solution is to do that and then run UNRAID on a VB VM? Thoughts?

Link to comment

Well - I gave it another shot:  Reset my NVMe as "Pass Through".  Created the VM.  Set the SysDev binding for that NVMe.  Rebooted UNRAID.

 

Logged in, started the array, started the VM and then tried to RDP to it - nada.  No joy.

 

So then I decide to stop the VM and change my Video card to VNC so I can see what is going on.  It shows me the UEFI Interactive Shell with a mapping table to floppy's and an ATA???  The shell command allows me to "exit" - but it enter some kind of VM BIOS and continue or reset does the same thing - just loops back to the EUFI Shell ... exit to the VM Bios - etc.  Stuck in a loop.

 

Seem the VM does not real get started after all.  I cannot RDP because Windows has not booted in the VM.

 

What could be wrong?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.