I have been overwhelming myself the last few days with info on this forum.  As a Newbie, I am sure that is all too easy to do.  I'm not really sure if this is the proper place to post this or not. If not then where?

 

I am wanting to start moving my data over to my NEW UnRaid.  I have a 1) 2tb parity and 3) 1tb data drives, for the moment.  I have precleared all drives and have mounted them and started my array.  I have started parity, but I have not done the parity check, plan to start this tonight when I get home.  As soon as I have done this, I will want to start loading some data on to the system and begin testing with my media streamers.

 

I have my data storage setup as follows:

 

 1 I have most of my data in use on my Qnap 509 NAS( 5) 2tb hard drives) that I am going to be migrating from, almost 6tb of my movies.

 2 I have another 1.5tb of movies on 2) WD MyBook usb 2tb external drives.

 

 3 All of 1 and 2 is backed up onto 6) 1tb hard drives, and 1) WD MyBook usb 2tb external drive.

 

---------------------------------------------------------------

 

Option 1

I have already installed my 6) 1tb hard drives (that I use as my backups) inside of my case of my UnRaid.  I was thinking that I would be able to copy the files off of these drives, and put them on the UnRaid drives.  And after I copied the first three, I could then preclear those and add them to my array.  Then I could just repeat for the last three.

 

But this brings me to my lastest confusion.  Can I plug those 6 drives in, not mount them to the array, and be able to copy then (seems the fast speed since no network is involved)? If so, please tell me how.  I thought I read that I could, but now I am not sure.

 

option 2

Or do I need to remove them from my case, and use my Esata dock to copy and paste them through my Vista computer.  

 

If so, then maybe I should just copy and paste them directly from my Nas, but this would still go through my Vista?

 

option 3

Is there another option?

 

----------------------------------------------------------------

 

Also, I have read that to make things faster, you should wait to hookup you parity drive, did I make a mistake their?  Or can I take it out of the array, and then add it back when I have downloaded my data?

 

I would appreciate any help / point in the right direction.

Option 1 is very easy to do if you are just going disk to disk copy (and I am assuming that the existing data is on NTFS drives and all are SATA drives - do not know of 1TB IDE drive)

You also must install Unmenu to facilitate the process too - it is more newbie friendly.

 

Under the disk management you will have options to mount the unassigned drives and also to share them on the network if you want to do so.

The tricky part is since you have already 11 drives in the system you will have to know their Linux order (they will be assigned the device numbers sda to sdk - I will make a blank assumption here but the actual order may be very different on your system);

-flash is sda

-parity is sdb

-data1 to data3 disks assigned in Unraid are sdc to sde.

-the six external disks are sdf to sdk

 

I will assume you want to do a straight copy from the first disk (sdf) to the first Unraid data disk and you are on your console with another computer nearby - go to the third tab on your "unmenu" from the other computer - it is called if I am not mistaken "Disk management" - you will see there you have six disks external to Unraid. Every one of them has a few available buttons to click on - go to the first one - it should be sdf - and click on the button named "Mount". Please excuse me if there are slight discrepancies as I do this from memory.

Your disk is now mounted and you can have access from the console (the keyboard for Unraid)

 

Now go to the console and type:

 

cp -r -v /mnt/disk/sdf1/* /mnt/disk1

 

There is a space between the * and the /

 

You will see the action on your Unraid monitor (if you have one).

 

A copy of 1TB should take at least good few hours (double or triple that if you have parity enabled) and I hope someone else from the Linux gurus here will correct me if I am wrong.

 

Once the copy is done you should go back to the other computer - unmenu - Disk management - and click on the "Unmount" button.

 

At this point you have your data from the first disk transferred - you should check if it is correct and if you kept the "parity" drive assigned in theory it is protected now.

 

Press "F2" on the console to open another session, login again as root and you can start the preclear procedure for this disk - sdf.

 

Press "F1" on the console to return back to the first session and you can start copy the next external disk to the Unraid disk2

 

Rinse and repeat but check also these:

http://lime-technology.com/forum/index.php?topic=3182.msg26747#msg26747

 

http://lime-technology.com/wiki/index.php?title=Transferring_Files_from_a_Network_Share_to_unRAID

 

Please keep in mind this (as you do not have location in your profile)

Caution: If you have directory or file names with non-English characters, commonly found in titles and names for media files from European and South American countries, I would recommend to use Windows Explorer. The Linux commands either skip the whole file or directory, or replaces the "strange" letter with an underscore.

 

Press "F2" on the console to open another session, login again as root and you can start the preclear procedure for this disk - sdf.

 

Press "F1" on the console to return back to the first session and you can start copy the next external disk to the Unraid disk2

It is Alt-F1 and Alt-F2 to switch between the virtual consoles, not just F1 and F2.

 

Joe L.

Thanks for the reply.

 

As I am getting ready to leave work and go home, I have briefly read through.

My drives are as you described:

flash  -sda

parity -sdb

Data 1-sdc

Data 2-sdd

Data 3-sde

 

The rest do follow the same.  I thought that maybe the simple and safest way, would be to only plug in the drive that I was going to copy from, that way I would have no confusion.

 

I have read a little on Unmenu.  I looked at downloading it, but a little confused and have a question.  I appears that you have to be connected to the internet with the UnRaid to be able to do the download.  At the moment I am not, I normally don't have any of my storage on a network that is directly connected to the internet, but I can do it if need.  But, do you have to be connected to the internet to run the program also, or is it just for the download process?

 

As for non-English characters, I am in USA, so I don't have any files that would cause any issue.

I have read a little on Unmenu.  I looked at downloading it, but a little confused and have a question.  I appears that you have to be connected to the internet with the UnRaid to be able to do the download.  

You will need to be connected to be able to have it install itself.  other than that, you'll need a connection if you attempt to download any of the packages in the package manager.  (The links are all there, so if you click on them on a different PC you can transfer the downloaded files to /boot/packages on the flash drive.) You can then proceed, since if it sees the files aready exist, it will permit you to go forward with the actual install.

 

At the moment I am not, I normally don't have any of my storage on a network that is directly connected to the internet, but I can do it if need.  But, do you have to be connected to the internet to run the program also, or is it just for the download process?

Not needed to run unMENU, just for download.

As for non-English characters, I am in USA, so I don't have any files that would cause any issue.

No problem.

My drives are as you described:

flash  -sda

parity -sdb

Data 1-sdc

Data 2-sdd

Data 3-sde

 

The rest do follow the same.  I thought that maybe the simple and safest way, would be to only plug in the drive that I was going to copy from, that way I would have no confusion.

 

It is up to you especially if you have a nice case with hot swappable bays but the advantage to leave them as is now is that you know that the assignment order won't change.

 

The purely assumption that I made above never works for me - usually the first ones are the drives on the Supermicro controller, then the first four on-board SATA ports, then the flash and then the last two on-board SATA ports or in other words every time I add an extra drive the on-board ones will change.... so you can make mistake this way too.

 

And this is how Unmenu looks:

 

 

However many of the drives are the older IDE type and they are assigned as hda, hdb,...etc.

The specific devices assigned by Linux is not fixed.  It depends on which hardware is scanned first on the bus and that can change as you add or delete controllers/disks and which initializes first.  It can change when you reboot.  unRAID does not use the three letter "device" when assigning disks, but uses the PCI hardware address. (that does not change unless you change the hardware)

 

When clearing disks you must know the correct three letter device name.  You can learn it by typing

ls -l /dev/disk/by-id

The "device" is the three letter name at the end of the line (sdX or hdX, where X is a letter a-z).  The device names with a trailing "1" should be ignored. Those are the names of the first partition on the disk.

 

 

 

Can parity be disabled just this once for the initial copy?

 

If you can't disable the parity, Would it not be better if he has not copied anything to the server yet to rebuild the server, not add the parity, copy the data, and don't delete the data from the NAS just yet, then once the copy has completed enable parity and start a parity check?  Would that not be faster?

 

Or could he enable one of his drives as a cache drive and copy to the cache drive then allow unRAID to copy data off the cache drive to the protected drives?

 

Sideband Samurai

Can parity be disabled just this once for the initial copy?

It could, but only if you trust the disks copied to, and do not re-use the original disks until AFTER parity is configured and initially calculated.

If you can't disable the parity, Would it not be better if he has not copied anything to the server yet to rebuild the server, not add the parity, copy the data, and don't delete the data from the NAS just yet, then once the copy has completed enable parity and start a parity check?  Would that not be faster?

Exactly as I said.  Yes, it would be faster... not necessarily smarter, but faster.  I personally would never disable parity as I value my data more than the time it might save.  I am not in that much of a rush.

Or could he enable one of his drives as a cache drive and copy to the cache drive then allow unRAID to copy data off the cache drive to the protected drives?

The cache drive is almost useless in this manner.  Highly likely it is far too small the hold the total amount of data to be copied, and the transfer to the protected array will still occur just in the middle of the night, and be no faster than if copied directly.

I ran my "check" parity last night.  It took about 7.5 hours to run.  Which if I calculate correctly using 2tb as drive size of my parity, then it ran at 2tb/7.5hours=74mb/s. Or is it, 3) 1tb for my data disks, then it actually ran at 3tb/7.5hours=111mb/s?  When I added my parity drive, it took about 12 hours to process.

 

I have come up with this as my options, if I can choose to have parity "online" of "offline" then I think it would be:

 

A) No parity (faster data copy)+ add parity process and then parity check = data copy time(?) + 12 hours parity process + 7.5hours parity check = (ready to go data protected)

 

B) With Parity (slower data copy) + no parity process and no parity check = data copy time(?) = (ready to go data protected)

 

C) With parity (slower data copy) + no parity process but YES parity check = data copy time(?) + 7.5hours parity check = (ready to go data protected)

 

It seems to me, that unless the copy time is extremely slower, that option B and C might be the faster.

 

And if I can wait (don't have to run parity "check") until I have precleared the first 3) drives (that I copied from) and added them to the array and then run the parity "check" here, then I would be able to copy the next 3 drives, and that might make option B the faster way to go.

 

I would suspect that the parity "check" will take long as the number of data drives would increase.  And my A,B,C options might be wrong, but that is why I am here posting to all of you who I am sure better know this stuff then I.  So is my thinking correct, and which way would you think I should proceed?

 

 

I ran my "check" parity last night.  It took about 7.5 hours to run.  Which if I calculate correctly using 2tb as drive size of my parity, then it ran at 2tb/7.5hours=74mb/s. Or is it, 3) 1tb for my data disks, then it actually ran at 3tb/7.5hours=111mb/s?  When I added my parity drive, it took about 12 hours to process.

I would suspect that the parity "check" will take long as the number of data drives would increase.  And my A,B,C options might be wrong, but that is why I am here posting to all of you who I am sure better know this stuff then I.  So is my thinking correct, and which way would you think I should proceed?

Unfortunately the time to initially calculate parity IS NOT an indication of how long it will take to transfer your data.   The initial calculation is reading every block of data sequentially from all the data disks and writing sequentially all the blocks on the parity disk.  It is limited primarily by the sustained read/write speed of the disks involved and the bandwidth of the data buses involved.   Typically it is somewhere between 50 and 100 MB/s.

 

Unless you are restricted by bus bandwidth (all drives on a PCI bus, for example), or controller bandwidth (disk controller only handling one request at a time rather than in parallel ... older IDE controllers fall into this category), adding more drives to the array does not slow down parity calculations very much.

 

The time to write to the array once parity protected is mostly limited by the rotational speed of the slowest disk involved being written to.   To copy data from your NTFS disk the array must read the NTFS disk, read the corresponding blocks on both the parity disk and the data disk being written, then allow the parity and data disks to rotate fully to allow it write to both the parity disk and the data disk.  There are three "read" I/O's involved, and two "write" I/O's for each block copied.

 

Typical "write" speeds with modern hardware and disks (and parity enabled) is somewhere between 30 and 40 MB/s.  

 

It seems there is roughly 7.5TB of data to be migrated.   If we assume a write speed of 30 Mb/s it will take 33.3 seconds to transfer 1000MB (1 GB )

Since there are 7500 GB to be transferred, that will be roughly 33.3 * 7500 = 250,000 seconds, or roughly 70 hours.  If you can write at faster than 30 MB/s, the time will be shorter. (40 MB/s = 52 hours)

 

Joe L.

You precleared your four disks (parity and 3 data), you build the parity and you checked it.

At this point one is to assume his system works as it should.

Any data that you copy from now on will be protected.

 

Now you can start transferring your data. If one is to assume writing speed to a protected disk to be 30-40 MB/s a conservative estimate for direct copy of 1TB data is 9-10 hours.

 

So you can proceed in this way:

 

After coming from work today you can start the copy from the first external data disk. By the next morning it should have been completed and you can start the second copy. Returning back home from work it should have completed too and you can start the third copy. By the next morning it should have completed and you can initialize a parity check. Your parity check will take you probably the same 7.5 hours.

It is up for you to decide if you should wait for the parity check to finish first, then check your data on the three Unraid disks compared to the data on the first three "external" ones and then start the preclearing of the three 'external" 1TB data drives (you should count around 18 hours for that).

Then you can add the three freshly precleared disks to the Unraid (will take you a few minutes) and you can start the secound round of data transfer.

 

Good luck

Unfortunately the time to initially calculate parity IS NOT an indication of how long it will take to transfer your data.  The initial calculation is reading every block of data sequentially from all the data disks and writing sequentially all the blocks on the parity disk.  It is limited primarily by the sustained read/write speed of the disks involved and the bandwidth of the data buses involved.  Typically it is somewhere between 50 and 100 MB/s.

 

Unless you are restricted by bus bandwidth (all drives on a PCI bus, for example), or controller bandwidth (disk controller only handling one request at a time rather than in parallel ... older IDE controllers fall into this category), adding more drives to the array does not slow down parity calculations very much.

 

My build is the US budget build as list, with only difference being the PS being different.  I have 6 Sata outputs on the motherboard, (first 4 are being used by the Unraid array, next 2 will be data drives to copy from) and then the add-on card has 4 more drives connected to it.  It will be interesting to see if there is a speed difference between the connections on the motherboard and the add-on card.

 

 

 

You precleared your four disks (parity and 3 data), you build the parity and you checked it.

At this point one is to assume his system works as it should.

Any data that you copy from now on will be protected.

 

I do like the idea that it is protected.

 

Now you can start transferring your data. If one is to assume writing speed to a protected disk to be 30-40 MB/s a conservative estimate for direct copy of 1TB data is 9-10 hours.

 

So you can proceed in this way:

 

After coming from work today you can start the copy from the first external data disk. By the next morning it should have been completed and you can start the second copy. Returning back home from work it should have completed too and you can start the third copy. By the next morning it should have completed and you can initialize a parity check. Your parity check will take you probably the same 7.5 hours.

It is up for you to decide if you should wait for the parity check to finish first, then check your data on the three Unraid disks compared to the data on the first three "external" ones and then start the preclearing of the three 'external" 1TB data drives (you should count around 18 hours for that).

Then you can add the three freshly precleared disks to the Unraid (will take you a few minutes) and you can start the secound round of data transfer.

 

Good luck

 

This sounds like a good plan.  I am not really pressed for time, just excited to get things going, so I think that I will take the time to do the parity check prior to the preclearing of the 3 drives.  Rather keep my backup drives intact until I am sure that everything is alright.

 

Tonight I will see about getting my internet connected to my UnRaid, so that I can do the install of the UnMenu.  If that goes well, I might be able to start the first copying tonight. 

Didn't get to start copying files last night.  Took me too long to get my Internet network to work on my UnRaid, media network is static ip and internet network is DHCP.  Probably could have been easier, but not by me.  But all said, UnMenu is now up and running, so I am ready to begin!

 

I have a question before I do begin tonight.

 

All of my movies are in the own [directory] like this:

 

  | --[Movie Title 1] (Directory)

  |    |

  |    |  Play the movie (folder)

  |    |            VIDEO_TS (folder)

  |    |                      vob (files)

  |    |  TviX.jpg

  |    |  folder.jpg           

  |    |  castmember (folder)

  |                  TviX.jpg

  |

  |

  | --[Movie Title 2] (Directory)

  |    |

  |    |  Play the movie (folder)

  |    |            VIDEO_TS (folder)

  |    |                      vob (files)

  |    |  TviX.jpg

  |    |  folder.jpg           

  |    |  castmember (folder)

  |                  TviX.jpg

 

       

 

I will want to have them in a main directory that I will share, that I will  call VIDEO.  Should I make the directory now before I copy my movies to the separate drives?  And if I do this, after making my directory on say disk1, would the copy command then be:  cp -r -v /mnt/disk/sdf1/* /mnt/disk1/VIDEO or would this be wrong?  Also, is upper / lower case an issue?

 

Or should I just dump the data on each drive, and then setup the structure after all is done?

 

Yes, create the VIDEO directory on the disks first.

 

mkdir /mnt/disk1/VIDEO

mkdir /mnt/disk2/VIDEO

 

Yes, upper/lower case is significant.  Make all the VIDEO directories the same..  They can be capitalized as you like, but don't make one VIDEO and another Video and a third video.  They'll not be treated as one directory in the user-shares.

 

Once you create the top level directory you can issue the "cp" command as you described.  You'll need to stop and re-start the array for it to export the folder you create using "mkdir" on the LAN.  (Just press "Stop" and then "Start")

 

You'll not be able to stop the array is a copy is in progress as the disk will be "busy" so create the top level VIDEO folders first, then Stop the array re-Start it, then perform the "cp" command.  You'll be able to follow along on the LAN as things are migrated.

I am having problems.  I am attaching my syslog.  It is done after I have attached the wires to only one of my hard drives (should be sdf).  It takes about 15 or so minutes when I push the power button until the console comes up to login.  I am not able to do anything from here, or I am not going to until I see what you all think.  

 

I have had all of my drive up and running, before I set up the array.  When all were up, I had made sure that all appeared in the drop down for being able to assign to mount.

 

Let me know what I can do or info I can give to help you help me.

 

Note: I powerdown with the push and hold power button and then remove that hard drive, and everything comes backup as normal.

syslog-2010-09-16.txt

You have problems with cable or power connector to this drive.

 

Sep 16 07:27:56 Tower kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

Sep 16 07:27:56 Tower kernel: ata5.00: link online but device misclassifed

Sep 16 07:27:56 Tower kernel: ata5: link online but 1 devices misclassified, retrying

Sep 16 07:27:56 Tower kernel: ata5: reset failed (errno=-11), retrying in 5 secs

Sep 16 07:27:56 Tower kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

Sep 16 07:27:56 Tower kernel: ata5.00: link online but device misclassifed

Sep 16 07:27:56 Tower kernel: ata5: link online but 1 devices misclassified, retrying

Sep 16 07:27:56 Tower kernel: ata5: reset failed (errno=-11), retrying in 5 secs

Sep 16 07:27:56 Tower kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

Sep 16 07:27:56 Tower kernel: ata5.00: link online but device misclassifed

Sep 16 07:27:56 Tower kernel: ata5: link online but 1 devices misclassified, retrying

Sep 16 07:27:56 Tower kernel: ata5: reset failed (errno=-11), retrying in 30 secs

 

Power down, check the connections if they are secure. If you are not using locking connectors this can happen easily when you connect and disconnect multiple cables inside - this is why I suggested  to do not touch anything as you boot order is not going to change.

 

Use the stop and the powerdown from the Unmenu instead.

I will check my cables and power.  I am using the lock on type cables.  I would have used the powerdown, but was unable.  I also tried to use the powerdown command in the console but it did nothing.

 

While I was waiting, and was shut down, I removed the cable for that drive and plugged in the next drive.  It came up just fine.  I will investigate more.

 

Thank you for responding so fast!

Well, I have the problem in hand.  I lost my first drive before I even got to use it in the UnRaid. 

I just removed it from my case and placed it in my estat dock.  It does just about nothing.  So its dead.

 

It was my 1st disk in my backup.  If there is a good part to it, last week, I made a copy of it onto one of my Mybook usb external drives.  Guess I will be delayed a little bit on getting my data on. But atleast it did it before it became a part of my array.

 

Thanks for the help, 

The system ran all night using the copy & paste with Teracopy, to get my data off of my Usb drive.  All went well and it finished this morning just before I left for work.

 

As for the drive that failed.  The only thing that I have used this drive for is backing up of data.  One write and verify check, and two reads, one when I changed all drives in my Nas and one when I copied it to the Usb drive last week.  I am wondering if maybe the Esata dock and oh course me, might be the cause of the failure.  Since the dock doesn't have and cooling on it, the hard drive does get pretty warm.  I think I might have baked it doing the copy last week.  It took about 11 hours to copy over from Esata dock to Usb drive.

 

The good news, none of the array has ever been in the Esata dock.  The bad news, all of the other drives that have my backup data is on have been.  I guess I will find out how it goes after I copy the next drive and then run the preclear on it.