Is there a way to import large amounts of data when first setting things up? The cache disk doesn't really help since I want to import 3TB and my cache disk is only 1TB.

 

I'm thinking things like maybe like disabling the parity drive while copying files over, and then calculating parity over night? Or possibly attaching an external disk (NTFS) to the machine and having it copied to the array over firewire/usb2 ?

I'm thinking things like maybe like disabling the parity drive while copying files over, and then calculating parity over night?

 

That's the typical way it's done.

Is there a way to import large amounts of data when first setting things up? The cache disk doesn't really help since I want to import 3TB and my cache disk is only 1TB.

 

I'm thinking things like maybe like disabling the parity drive while copying files over, and then calculating parity over night? Or possibly attaching an external disk (NTFS) to the machine and having it copied to the array over firewire/usb2 ?

If you do not assign the parity drive, it will not be used while initially populating the array.

 

Yes, you can connect a USB drive directly and copy from it.  It will speed things up a bit. 

http://lime-technology.com/wiki/index.php?title=Copy_files_from_a_NTFS_drive

 

Joe L.

Set the array up and install the all the disks. Leave the disk you want to use as parity unassigned. Personally, I'd ensure it's there on the drop down but then put the drop down back to nothing. Set-up the user shares as you want them to work. Be careful of the include/exclude disks and the split level so your data gets copied the way you want - ask about your specific data situation if you're not sure. Copy all the data to the server.

 

Then, assign the parity drive and allow the parity built to complete. VERY IMPORTANT - do another parity check to ensure there are no errors because errors could indicate a system issue that must be addressed.

 

Finally, you can delete the data from the origional location.

 

Peter.

Yes, you can connect a USB drive directly and copy from it.  It will speed things up a bit. 

http://lime-technology.com/wiki/index.php?title=Copy_files_from_a_NTFS_drive

 

Joe L.

 

I have a few questions in regards to the info in that Wiki page.

• In what aspect of this does the well documented data loss danger exist?  I'm guessing 1) If you assing the drive to unRAID and 2) if you attempt to write to the drive? 
  
• I understand the part above "Contents" and I understand the part below "Contents" but something is missing between there.  Where am I going to run these commands described in the bottom part?
  
• At the top we assumed the drive is /dev/hdj but in the commands you are referring to /dev/hdj1.  Was this your intent or is it a typo?
  

Yes, you can connect a USB drive directly and copy from it.  It will speed things up a bit. 

http://lime-technology.com/wiki/index.php?title=Copy_files_from_a_NTFS_drive

 

Joe L.

 

I have a few questions in regards to the info in that Wiki page.

• In what aspect of this does the well documented data loss danger exist?  I'm guessing 1) If you adding the drive to unRAID

Yes, you are physically adding the drive to the unRAID server, but you are not assigning it to the array on the "devices" management page in the user interface.
 
There are risks.
You could drop the drive when physically moving it from one server to the other.
You could physically damage either the source PC or the unRAID server when moving the drive.
You could snap the USB flash drive on the unRAID server when you move the case to install the disk you are copying from.
You could accidentally cause electrostatic-discharge-damage (ESD damage) to either PC or to the disk drive if you do not wear a anti-static wrist strap.
You could electrocute yourself if you forget to unplug the PC or the server before transferring the drive and put your fingers in the wrong place in the PC or the unRAID server.
You could bend the pins on the disk-drive if you do not plug the cable in properly damaging it.
 
Some of these risks are small, others have some real potential (no pun intended) for damage.

and 2) if you attempt to write to the drive? 

You will not be able to write to the drive you are copying from...  The "-r" option to the mount command forces the disk to be mounted as read-only.  It will be impossible to perform any writes to the drive if you type "mount -r ...... ".

• I understand the part above "Contents" and I understand the part below "Contents" but something is missing between there.  Where am I going to run these commands described in the bottom part?

 

once you "mount" the drive on a mount-point (a mount-point is an empty directory) you can change directory to it and use one of the commands to copy the data from it.  You would log in via telnet or on the system console and then type one of the "cp" (copy) commands.

 

The wiki was written before SATA drives were common, so it is possible the drive you are copying from is a /dev/sdX  instead of a /devhdX.  Don't let that confuse you.

• At the top we assumed the drive is /dev/hdj but in the commands you are referring to /dev/hdj1.  Was this your intent or is it a typo?
  

It is NOT a typo.  /dev/hdj1 is the first partition on /dev/hdj.  In the same way, if a disk had two partitions, /dev/hdj2 would be the second partition, /dev/hdj3 the third, etc...  You mount a partition on a mount-point.  The entire disk is known by the base name without any numeric partition suffix numbers, but you seldom refer to the entire drive that way unless you are partitioning it, or running a SMART report on it, or using the "hdparm" command on it.

 

Do not leave the "-r" option off of the mount command.  The "ntfs" driver included with unRAID is not capable of writing to the ntfs formatted drive. It is a read-only driver.

I was confused where i would run these commands.  It would appear that this could be cone on the unRAID console after logging in, correct?

I was confused where i would run these commands.  It would appear that this could be cone on the unRAID console after logging in, correct?

Yes, you would type them either on the system console after logging in, or in a "telnet" session after logging in.

So I followed the info on this link http://lime-technology.com/wiki/index.php?title=Copy_files_from_a_NTFS_drive mentioned at the top of this thread.  I'm at work so I'm not sure of all my specs and cant post any logs or anything but Ill try and list as much info as i can to get this started.

 

Im running unraid 4.5.3 on a BIOSTAR TA790GXE MB with CPU AMD ATH II X2 250 AM3 3.0G,  2 GB of RAM  Im attempting to transfer off about 225 gigs of data(DVD rips) off a 250g WD IDE HD (NTFS) to a 1.5 TB SATA drive connected directly to the MB.  Currently the unraid setup is 2 - 1.5 TB SATA drives and 1 400g SATA drive.  Parity is turned off while I move the data.

 

I started off trying to hang the drive on my XP SP3 box and do a file copy but it would crash explorer so I tried to use robocopy and it would fail as well.

 

Now I have the drive hanging off the unraid box and mounted it and started a sweet cp command to copy the data over to disk1.  The new unmenu has a page that shows the speed that the drives are writing at which is 200kbs (I can download files from the internet faster than that).  I let it run over night last night and in about 9 hours it had copied about 16GB of data.  This seems way way way too slow and at this rate it will take several days to get the data moved over.  Does this seem right to anyone?  It was showing this speed for Disk 1 and my parity disk as well which is kinda strange since I have parity turned off.

 

Has anyone else used this method to move data?  How was performance? am I missing a setting somewhere?

Right now the common denominator is the old 250g HD.  Maybe its on its last leg?  Ive got one other 250g drive with mostly pictures and MP3's that i need to move over as well so I would like to resolve this issue.

 

Thanks in advance.

Chris

 

 

So I followed the info on this link http://lime-technology.com/wiki/index.php?title=Copy_files_from_a_NTFS_drive mentioned at the top of this thread.  I'm at work so I'm not sure of all my specs and cant post any logs or anything but Ill try and list as much info as i can to get this started.

 

Im running unraid 4.5.3 on a BIOSTAR TA790GXE MB with CPU AMD ATH II X2 250 AM3 3.0G,  2 GB of RAM  Im attempting to transfer off about 225 gigs of data(DVD rips) off a 250g WD IDE HD (NTFS) to a 1.5 TB SATA drive connected directly to the MB.  Currently the unraid setup is 2 - 1.5 TB SATA drives and 1 400g SATA drive.  Parity is turned off while I move the data.

 

I started off trying to hang the drive on my XP SP3 box and do a file copy but it would crash explorer so I tried to use robocopy and it would fail as well.

 

Now I have the drive hanging off the unraid box and mounted it and started a sweet cp command to copy the data over to disk1.  The new unmenu has a page that shows the speed that the drives are writing at which is 200kbs (I can download files from the internet faster than that).  I let it run over night last night and in about 9 hours it had copied about 16GB of data.  This seems way way way too slow and at this rate it will take several days to get the data moved over.  Does this seem right to anyone?  It was showing this speed for Disk 1 and my parity disk as well which is kinda strange since I have parity turned off.

 

Has anyone else used this method to move data?  How was performance? am I missing a setting somewhere?

Right now the common denominator is the old 250g HD.  Maybe its on its last leg?  Ive got one other 250g drive with mostly pictures and MP3's that i need to move over as well so I would like to resolve this issue.

 

Thanks in advance.

Chris

 

 

You un-assigned the parity drive but did not set a new configuration without it. Basically you are simulating a failed parity drive, not a server with none installed.  To set a new disk configuration you would have needed to press the button labeled as "restore" which has nothing to do with data, but in setting a new disk configuration based on the currently assigned and working disks. Pressing it immediately invalidates parity (which in this case, you don't care).

 

as far as 200k/s yes... way slow.

 

Have you considered posting a syslog for analysis?  Who knows, one of your disks might be running in PIO mode...

I copied about 8TB of data from a file server to my unraid.  I just started the copy on a Friday afternoon, and it was done the following Tuesday when i got back to the office (holiday weekend).  If you can go without, i'd let it run overnight and into the next day.

You un-assigned the parity drive but did not set a new configuration without it. Basically you are simulating a failed parity drive, not a server with none installed.  To set a new disk configuration you would have needed to press the button labeled as "restore" which has nothing to do with data, but in setting a new disk configuration based on the currently assigned and working disks. Pressing it immediately invalidates parity (which in this case, you don't care).

 

as far as 200k/s yes... way slow.

 

Have you considered posting a syslog for analysis?  Who knows, one of your disks might be running in PIO mode...

 

Interesting  I just clicked cancel as it was building the parity and assumed that it would not be writing anything to that drive.  Im sure the copy will still be running when I get home this evening so I will attempt to "invalidate" the parity.  If that does not work Ill post up my syslog.

 

I copied about 8TB of data from a file server to my unraid.  I just started the copy on a Friday afternoon, and it was done the following Tuesday when i got back to the office (holiday weekend).  If you can go without, i'd let it run overnight and into the next day.

At this rate it will be more than a long weekend.  and 250gb of data vs 8TB of data is a pretty big difference.  If I had 8TB of data it might take a month or two. 

 

 

Thanks for the replys

If I had 8TB of data it might take a month or two. 

 

8 TB = 8192 GB = 8,388,608 MB = 8,589,934,592 KB

 

at 200 KB/S = 42,949,672.96 seconds ~= 715,827.9 minutes ~= 11,930.5 hours ~= 497 days

 

So I removed my parity disk from the array and restarted my file transfers.  Now Im getting upwards of 25,000-30,000 KBs transfer rates.  Much better.  Once I get this data moved and the parity built I can then focus on my crashplan client http://www9.crashplan.com/consumer/index.html up and running.

HAPPY CAMPER 

 

Thanks so much Joe!!!!

 

8 TB = 8192 GB = 8,388,608 MB = 8,589,934,592 KB

 

at 200 KB/S = 42,949,672.96 seconds ~= 715,827.9 minutes ~= 11,930.5 hours ~= 497 days

 

 

Thats like 2 months  lol

So I removed my parity disk from the array and restarted my file transfers.  Now Im getting upwards of 25,000-30,000 KBs transfer rates.  Much better.  Once I get this data moved and the parity built I can then focus on my crashplan client http://www9.crashplan.com/consumer/index.html up and running.

HAPPY CAMPER 

 

Thanks so much Joe!!!!

 

 

There is a wiki entry on how to get crashplan up and running on your unRAID server.  It takes a little bit of work but is not to bad.

Ive been reading it.  So I think Im ready.  I follow crashplan on twitter and I think the tweet when there is an update so Ill use that as a reminder to re-tar everything up. 

This will work with an internal HDD, too. Right ?

//Edit: Ok my fault, I forgot after the hdj the 1.

 

So I assume it's working now, because the writes on the disk are increasing. And the folders are on the Server.

But I don not see a progress Bar or anything else in the putty window

 

So when will I see when it's done ?