Getting Goin Posted October 31, 2020 Share Posted October 31, 2020 Hi All, I know my Subject title is repetitive, its my hope that it gets index well to help others find this thread to help answer this question as basic as it may seem to many and maybe most. I have been for days trying to figure out how to do this without any satisfactory result. Yes i have searched these forums, read tutorials, manual and even youtube and nowhere could i find a clear cut answer to how to properly transfer my data to new unraid setup. Things i have come across are hooking my computers together via Ethernet, connecting via USB, unassigned devices, krusader, rsync, etc... I am pretty sure that i have read enough that i could hobble my way through this at this point. But none of it inspires data confidence with me being completely new to the game. Most forum scenarios i have read are more complex than mine and usual take on another conversation due to the nature of their previous setup. Mine is simple. I am coming from a regular computer with standard drives formatted to NTFS. Some drives may have few partitions (three or less). The Scenario --------------------------------------------- So to keep this basic and not get side track to how many gigs or time this will take, etc.. I just want all to assume i have just two NTFS formatted drives FULL with 8tb of data in three partitions. I have a brand new trial unraid with default settings other than a password has been assigned to root/admin account. I have installed squids community applications and can see apps listed as a heading in unraid, but have not installed any plugins. I have purchased one brand new 10tb hard drive and inserted it into unraid and it passed a Smart Short and Extended tests. I have not started a Array as not quite sure i can or should at this point. I have not installed a parity drive or a cache drive although i do have one of each. I would like to encrypt my data from this point on. I would like to start off learning the correct way to properly / securely transfer my data one disk at a time to unraid. My beliefs, knowledge and questions at this point. --------------------------------------------------- I have read about preclear. It seems to be suggested alot in past guides and tutorials. It seems this third party function has been built into unraid now and is no longer needed. I am not sure as i don't see it listed in the web GUI. I have a new drive and it would seem that it would not be needed for it, but will i need it or something similar to clear the old 8tb drive after i transfer it to the new fresh 10tb drive? How to setup the old 8tb drive to get ready for the next transfer? Parity and Cache. Its my understanding that it would be best to put these in after i have transferred all my data as this will help expedite / speed things up. Keeping in mind i am used to no parity or raid. I have been gambling with no redundancy to this point on most data thats not super valuable. So i am ok being somewhat unprotected in the interim, but look forward to the new protection unraid will give me. Encryption. I would like to encrypt all data moving forward, but would like to know if this sacrifices productivity much or at all? Also can this drives be unlocked without unraid in the future if i need to with the key and some other software? Don't what to get to sidetracked on this as its really another topic, but quick push in right direction at this point would be nice. Thanks --------------------------- Much appreciate any help you can give. I know this topic has probably been covered and most likely seem basic to most. But i could not find a cut and dry answer to this to be 100% positive i was doing the right thing with my new unraid setup. I did look and i am sure somebody can and probably point a link directly to simple answer, but i truly couldn't find it. So thanks to all that help and i hope it may help others as time goes on. Cheers to all and have a great day! Few keywords to help index -------------------------------------------------- Transfer old hard drive to unraid, transfer existing data to unraid, transfer full hard drive to unraid, migrate old data to unraid, migrating data to unraid from windows, data to unraid, move data to unraid, moving data to unraid, transfer to unraid, moving hard drive to unraid. Again thanks to all! Quote Link to comment
itimpi Posted October 31, 2020 Share Posted October 31, 2020 You mention having a ‘parity’ disk and separately mention a 10TB disk. Are these the same or do you have another 10TB disk to use as parity? It is a requirement that any parity disks be at least as large as the largest data disk in the array. all drives need to be formatted by Unraid before they can be used in the array. Prior to formatting a drive (which wipes any existing content) you will need to set whether it is going to use an encrypted file system or not. Therefore the NTFS disks cannot be used ‘as-is’, although they CAN be mounted as Unassigned Devices for purposes of copying the data off them. I prefer unencrypted drives as encrypted drives can be hard to recover their contents if anything happens to go wrong, but that is a personal preference. You have to decide yourself on how to balance privacy v convenience. parity and cache can be added at any point. It is commonly recommended to not use cache during initial data load. The recommendation to not use parity during this stage is a trade-off of speed v protection. Copying without parity is faster but your data is not protected against drive failure until parity has been built. regarding pre-clear, you may find this section of the online documentation useful in clarifying its use. Quote Link to comment
Getting Goin Posted October 31, 2020 Author Share Posted October 31, 2020 Thanks for the quick response itimpi. I have highlighted your questions in BOLD ITALIC as i could not figure out how to do separate quotes and tags [] [/] such as these did not seem to work. You mention having a ‘parity’ disk and separately mention a 10TB disk. Are these the same or do you have another 10TB disk to use as parity? It is a requirement that any parity disks be at least as large as the largest data disk in the array. Yes, I mention them later on as having them, but not utilizing as of yet to hopefully gain transfer speed. They will be separate disk and parity will be 12TB, Cache will be SSD. So to answer it simply i have more disks for those purposes, the 10tb is specifically for starting to transfer my data over to/on. all drives need to be formatted by Unraid before they can be used in the array. Prior to formatting a drive (which wipes any existing content) you will need to set whether it is going to use an encrypted file system or not. Therefore the NTFS disks cannot be used ‘as-is’, although they CAN be mounted as Unassigned Devices for purposes of copying the data off them. I prefer unencrypted drives as encrypted drives can be hard to recover their contents if anything happens to go wrong, but that is a personal preference. You have to decide yourself on how to balance privacy v convenience. So i have to format the old 8tb drive after i transfer the data to the new 10tb drive. You mention in as though it can be done via unraid, will have to look into that process. I will need to install unassigned devices because they are NTFS and will not be part of array. Once I have unassigned devices install, i can put old drive in and it will be obvious how to copy? Just select all and drag and drop sort of scenario? Does it check the process to make sure %100 copied? What if it gets interrupted like a power outage, will it know how to resume? Will it have to start over and will i have to override each prior copy file with a question like duplicate file do you want to keep or copy over, etc.. This seems less than ideal, i sort of envision a better process for unraid, although maybe there is not. If this is the way, i guess i can hobble through as i said prior. I really thought there would be a process for the copy/sync and maybe a recheck of the data to ensure successful completion. I am on the fence about encryption, I like the idea though. I would probably try it on a less risky disk first to see how the process goes and decide from there based on experience. Although i would hate to lose any data. I hate to get sidetracked on this. But what could go wrong? parity and cache can be added at any point. It is commonly recommended to not use cache during initial data load. The recommendation to not use parity during this stage is a trade-off of speed v protection. Copying without parity is faster but your data is not protected against drive failure until parity has been built. Yes, this was my belief prior to your response, thanks for confirming. I will leave parity and cache until the very end. regarding pre-clear, you may find this section of the online documentation useful in clarifying its use. Thank you i will read this documentation. Many thanks for response and help. Quote Link to comment
Getting Goin Posted October 31, 2020 Author Share Posted October 31, 2020 The writing may be on the wall for drive encryption. I can't see to get it to work on top of the data transfer issues. I have figured out how to format via XFS. Click on the disk and in the properties change file type to XFS. Then start array and it will tell you not valid partition and at some point within the Gui it will tell you to format the unassigned drives and list them. I was able to do this and start the array. However when i tried to do the same but with Encrypted - XFS I cannot start array. On the main page it tells me i need a passphrase or keyfile. If i choose passphrase and type in basic word or even complex pass i cannot get it to allow start array. Start array button is greyed out. I can only click on maintenance check box, but it does nothing either. I cannot save Passphrase or start array. Maybe this is a sign, i dunno. Nothing ever seems to come easy. Still unsure how to properly transfer existing data to the array. Thanks Quote Link to comment
Getting Goin Posted October 31, 2020 Author Share Posted October 31, 2020 Ok, persistence pays i guess . I have managed to get the drive encrypted. The steps i took are listed. I post this to not only show what i have done at this point, but in hopes it may help somebody in the future. Steps to encrypt drives in Unraid for new setup. (original topic sidetracked) -------------------------------------------------------------------------------------------------------- Clicked on the disk and selected XFS - Encrypted and then went into SETTINGS TAB (main unraid setting tab) and click on disk settings. In disk settings had to change default file system to XFS - Encrypted. Then back to MAIN TAB and next to the greyed out start array button i typed in my password twice, but it would still not show proper start button until i clicked on show password and it registered and the start array button became available. Then I started array and it rebooted with Unmountable: Volume not encrypted showing under the disk. On same MAIN TAB under array operation it will show the unmountable disk present with a greyed out FORMAT button. Next to it will say "Format will create a file system in all Unmountable disks." with a box that can be checked saying "YES I WANT TO DO THIS". When you click that box it will pop up a WARNING, click ok and the format button will be available. Click the FORMAT Button and it will start the process. Original topic back on track ----------------------------------------------------- Any more tips on moving data to unraid? Should i just insert hard drive, install unassigned devices and just Copy and paste the files? Hoping for better way.. Thanks Quote Link to comment
trurl Posted October 31, 2020 Share Posted October 31, 2020 5 hours ago, Getting Goin said: could not figure out how to do separate quotes Select the text you want to quote and "Quote selection" popup will appear for you to click on. Quote Link to comment
Getting Goin Posted October 31, 2020 Author Share Posted October 31, 2020 8 minutes ago, trurl said: Select the text you want to quote Awesome, never saw that before, i like it alot. Thanks! I now have installed unassigned devices and have mounted a drive via that section. I can navigate the drive and its partitions right down to single file level via the unraid web GUI. I cannot however copy folders or any file, it allows me to open the files but cannot copy them at this point. Quote Link to comment
Getting Goin Posted November 1, 2020 Author Share Posted November 1, 2020 (edited) Few hours later and really no further along here, would love a push from somebody. I am starting to believe rsync may be the best solution, but the commands are confusing and unsure weather it can be properly run from the GUI. I am reading about the commands and have opened it via terminal in the GUI. But some guides suggest SSH, unsure why as seems terminal in GUI should be suitable. Seems rsync may be more advance than i can handle without proper guidance. I really don't want to corrupt data or delete something unnecessary. Many guides refer to identifying the hard drive via disk #. But the unassigned drives do not have disk #. They are labelled by device . The Wiki speaks as if moving and coping within the array using commands such as MV or CP -R are valid solution leaning towards copy as the best considering it doesn't delete old file like move does. Which is how i would like to do it. But it doesn't give commands from unassigned to array. It also doesn't list commands to show status of transfer or success as seems Rsync has for at least transfer status. Really hoping somebody can chime in here, love a good push it right direction. Thanks Edited November 1, 2020 by Getting Goin Quote Link to comment
Getting Goin Posted November 1, 2020 Author Share Posted November 1, 2020 Ok, give up for another day. Last thing i tried was installing Krusader, which can only be done with docker. In order to install it i had to start array, went into apps and installed krusader. I accepted all defaults. To get into the docker web gui i clicked on the docker image of krusader in Unraid web gui. It loaded the krusader docker which looks similar to windows file explorer. I thought i would move one directory as a test and i did. Then i realized i should have copied, instead of move as original drive now missing that directory and i am unsure this will work correctly. So i try to copy it back and it won't let me. Error of some sort, can't recall now. I look around on the drive and even search for the file to see locations and it seems as it split the file in few folders, under app data and home, also within krusader folder structure. Weird to beginner newbie. So I figured i will copy the data over instead of move, but leery of this as unlikely to give me status and proper verification that all files transferred properly. So I see the synchronization option. Decide to try that. It seems to work, but rather than go from unassigned drive to array share disk it did the opposite. Maybe had right to left and left to right setup wrong. Could have since late and been trying to do this for days now. Frustration has set in a little and why i need to break away for now. I now have my original disk a little messed up as i have folders from the disk array on it and the array has a few files and folders scattered around it. Nothing to crazy, but less than ideal. Just want to figure out how to get it all moved correctly, will fix that folder structure later. I need a break away though, got a little stir crazy with it. Hope some direction comes soon. thanks Quote Link to comment
Vr2Io Posted November 1, 2020 Share Posted November 1, 2020 (edited) 2 hours ago, Getting Goin said: Then i realized i should have copied, instead of move as original drive now missing that directory Yes, don't use move, because it means source will be modify, always should keep source without change. 19 hours ago, Getting Goin said: hooking my computers together via Ethernet, connecting via USB For reliability and speed ( time consume ), all disk should connect in local. 1) You need well planning on how those 2 disk data ( source ) allocate to how many disk ( Unraid ), no reason to copy all then relocate file again. 2) Suggest you use "MC" via remote terminal, because Unraid web terminal with MC won't work well. 3) Due to source are NTFS and mulit-partition. I think Linux won't work well, so I suggest you share it thr SMB or NFS as READ-ONLY, then use UD to access it. This would be slow, but consider it is safe and seems you not family with command base operation. 4) Use MC to perform main copy task, then use rsync with "-n" ( dryrun ) to check does file match from destination with source. 5) Do you understand /mnt/user or /mnt/diskX their different, it not just use either one, don't mix use. Edited November 1, 2020 by Vr2Io Quote Link to comment
Getting Goin Posted November 1, 2020 Author Share Posted November 1, 2020 11 hours ago, Vr2Io said: For reliability and speed ( time consume ), all disk should connect in local. Ok, so maybe I am doing this wrong. But I have put the disk physically in the Unraid machine, then mounted it via Unassigned Devices (UD). With your post its making me believe i should have connected to it differently. But unsure as in the actual machine is still local I believe. Which way do you recommend for me to connect the drive the best? Physically in machine, Ethernet or USB? USB would seem less ideal to me than physically in machine, but maybe Unraid reacts differently to external drives, unsure?.. 11 hours ago, Vr2Io said: 1) You need well planning on how those 2 disk data ( source ) allocate to how many disk ( Unraid ), no reason to copy all then relocate file again. I was planning on copying them exactly like they were 1 disk for 1 disk sort of speak. I wish i could just plug my existing drives into unraid and be done with it. I like how they are organized and see no need to change it at this point. I do believe i would like to setup things differently going forward, but not sure if that will matter for my prior data arrangement. I will give example. ------------------------------------- Say I have one drive full of old movies and they are sorted via genre folders. Like Horror, comedy, drama, etc.. lets say 10 genre folders for this example and 20 movies. I have two movies in each genre folder. disk is full. lol yes they much be 10k to fill up the 8tb. Its just example. Ok. I would like to move that whole disk and keep it that way on Unraid. Why, because they are old movies. I am unlikely to be accessing them regularly or at the same time on different devices. But on the new drives I install, maybe I would like to spread new movies around on different disk. They are new and maybe in demand/popular throughout my devices. So for speed maybe mix them up across the drives, but keeping each movie, episode or music cd together in one folder ( not scattered pieces of it around, movie one place, thumbnail another, etc..). Does this sound logical? Can this be done? Or if i want to mix up the new directories does this have to start from the beginning with the old drives? To be truthful i haven't given layout strategies much thought at this point. Been focusing on how to transfer. If I have to move this all around again after completion it will suck for time wise, but it will be a learning experience. Something i am ok with if i have to be. To get it right the first time would be strange for me as it never seems to happen like that. I would be very pleasantly surprised though if it did... 12 hours ago, Vr2Io said: 2) Suggest you use "MC" via remote terminal, because Unraid web terminal with MC won't work well. Ok, this will be new to me but i will look into learning how to do this right away. 12 hours ago, Vr2Io said: 3) Due to source are NTFS and mulit-partition. I think Linux won't work well, so I suggest you share it thr SMB or NFS as READ-ONLY, then use UD to access it. This would be slow, but consider it is safe and seems you not family with command base operation. I now have two drives in my Unraid. The semi empty 10tb one as i have managed to copy some scatter files on it with krusader. The other one being the 8tb data NTFS disk. But based on your comment above it sounds as it best left in original machine and then shared? When you say use UD (Unassigned Devices) to access it, is this done via remote terminal as well? Cause when I use UD to access the current mount drive in the unraid web gui within the MAIN TAB, it really doesn't allow me to do anything other than load/read file. There are no copy commands of any sort. I do have options prior to mount, such as read only, pass thru and something else can't recall. Slow and safe is preferred. I am limited on commands. The other drives that are NTFS are currently in a linux (Ubuntu) machine. I have been playing with Ubuntu for about a year, would love to get better with it and wish i jumped onboard years ago. But my learning ability is not as quick these days and most commands have been cut and paste without any realization into learning what they are actually doing or how the commands break down. 12 hours ago, Vr2Io said: 4) Use MC to perform main copy task, then use rsync with "-n" ( dryrun ) to check does file match from destination with source. I will try and give it a go, but will have to research it prior. I am not exactly sure the proper command structure for my scenario. I will have to do some reading to try and figure out Midnight Commander (MC) and Rsync. I have been trying to learn rsync prior as i though it may have been the answer, but haven't had to much luck with these commands. 12 hours ago, Vr2Io said: 5) Do you understand /mnt/user or /mnt/diskX their different, it not just use either one, don't mix use. Yes, i believe i do understand that they are different as one being user and the other a disk. But maybe I don't as I type cannot see a reason to mnt/user. I am also unsure why you ask, is it because i was referring to the unassigned drive being a device id versus the disk array having a disk #? ----------------------------------------------------------------------- The more time i sink into this, the more i realize unraid may be over my head. But i am fairly stubborn and would love to learn it. So I think for the sake of my data security I will just continue with trying to get this one disk copied on to unraid and leave the others and even this disk alone afterward and put in back in the old system. This until i feel more secure with unraid. But i really need to complete this transfer at this point as i cannot for the life on me give up on something that i thought would be of little difficulty going into this. I thought getting the dockers, vm, etc.. all setup and talking to my devices would be the challenge, at this point can only imagine that rabbit hole. Do all new users with unraid just understand how to easily copy their data over? Do all users have strong command background and come from server backgrounds, nas, qnap or synology type systems? Nobody just starts with unraid for their first system and need existing data transferred? Based on documentation i have NOT found it seems this may be the case. I would have thought there would have been some guide, tutorial or youtube video out there with some break down of a beginner setup coming from a place with prior data disk trying to migrate to unraid. Not just fresh setups and now start new data collection. Quote Link to comment
Vr2Io Posted November 2, 2020 Share Posted November 2, 2020 8 hours ago, Getting Goin said: Ok, so maybe I am doing this wrong. But I have put the disk physically in the Unraid machine, then mounted it via Unassigned Devices (UD). With your post its making me believe i should have connected to it differently. But unsure as in the actual machine is still local I believe. Which way do you recommend for me to connect the drive the best? Physically in machine, Ethernet or USB? USB would seem less ideal to me than physically in machine, but maybe Unraid reacts differently to external drives, unsure? For best operation, you are correct, all source disk and destination disk under Unraid ( in local ). But as mention, Linux won't well handle NTFS or Windows won't well handle XFS/BTRFS, so keep NTFS in Windows and XFS/BTRFS in Unraid then share it by SMB thr ethernet will be best. 8 hours ago, Getting Goin said: I was planning on copying them exactly like they were 1 disk for 1 disk sort of speak. 8 hours ago, Getting Goin said: Been focusing on how to transfer. 8 hours ago, Getting Goin said: So I think for the sake of my data security I will just continue with trying to get this one disk copied on to unraid Yes, use your family way to perform the job. 8 hours ago, Getting Goin said: Do all users have strong command background and come from server backgrounds, nas, qnap or synology type systems? Start in basic, I use NAS, QNAP, Synology for 8yrs, Unraid 4yrs, still in learning. The more you know then it usually help reach purpose quickly and effectively. 8 hours ago, Getting Goin said: Nobody just starts with unraid for their first system and need existing data transferred? In simple say, beginner will copy from source machine to Unraid machine. No complicate. Quote Link to comment
JonathanM Posted November 3, 2020 Share Posted November 3, 2020 On 11/1/2020 at 12:52 PM, Getting Goin said: I would have thought there would have been some guide, tutorial or youtube video Have you browsed https://www.youtube.com/channel/UCZDfnUn74N0WeAPvMqTOrtA Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.