Moving Data off old drives...


Recommended Posts

It sounds like you have the right order....build your new unRAID server using three NEW drives (one parity, two data), copy the data from your XP system over from your existing two drives, then pull the existing drives out of your PC and install in your unRAID system.

 

You can't do it the other way (i.e., just add a parity drive and install all three in your UNRAID system), since unRAID uses the ReiserFS file system and will need to reformat all the data drives.

Link to comment

Depending on how much data you need to move the simplest way would be to just copy the data over the network.  If you had a usb enclosure you might be able to connect it directly to the server to copy the files

hmm - oh well - thought it would be this way.  It's 1TB worth of movies :).  Guess it will take a while
Link to comment
  • 2 weeks later...

Depending on how much data you need to move the simplest way would be to just copy the data over the network.  If you had a usb enclosure you might be able to connect it directly to the server to copy the files

hmm - oh well - thought it would be this way.  It's 1TB worth of movies :).  Guess it will take a while

You might be able to install the drives in the unRaid server but DO NOT ASSIGN THEM in the disks page on the web-management utility.  Instead, write down their disk assignment slots.

 

In other words, they will be at /dev/hda, /dev/hdb, /dev/hdc, etc... but the actual hd will be listed along side of the drive's serial number in the drop-down-field on the disk assignment page.  Again, do not assign the disks, but write down their position in the array. (you must temporarally stop the array to see the physical device names of the disks in the drop-down-lists, but leave them unassigned to unRaid slots)

 

As an example... let's assume your old data disk was put in the physical device /dev/hdj ... type the following...

 

mkdir /mnt/old_data

 

mount -r -type ntfs /dev/hdj /mnt/old_data

 

or... if a FAT filesystem

 

mount -r -type vfat /dev/hdj /mnt/old_data

 

then, you can use the unix cp command to copy your files to an existing data drive you previously added to your unRaid array.

 

To copy all your files to disk1 type this:

 

cd /mnt/old_data

cp -r * /mnt/disk1

 

Lastly, to unmount the old disk after copying all the files from it type:

 

umount /dev/hdj

 

(Yes, it is umount, not unmount)

 

Once the data is off of the old NTFS/FAT drive and copied to an unraid data drive, and the old drive is then unmounted, you can then use the devices web-management page to asssign the old drive to a slot in the unraid array. (you must stop the array to assign it)  It will then be cleared and formatted and made available as part of the unRaid array.

 

Now, just to let you know... I've never done this on my array, but I see no reason why it should not work.  use of the -r option to the mount command will mount the drive as read-only so you should be safe... only way to mess up would be to assign the old drive to unRaid and clear/reformat it before you got your data copied from it.

 

Let us know how it works for you...

 

Joe L.

 

 

 

 

 

Link to comment
  • 4 weeks later...

How were you guys mounting the NTFS file systems? When I tried to mount, I was told that ntfs was an unknown file system type. Were you using fuse (I think that's it) from SourceForge?

 

Cheers,

 

 

I was pretty sure that NTFS used to be in there as a "read-only" filesystem driver, but I just looked at my current 4.0 final server.  It's not in there... ???

 

Tom, is my memory failing me?

 

Joe L.

[pre]

root@Tower:/proc# cat filesystems

nodev  sysfs

nodev  rootfs

nodev  bdev

nodev  proc

nodev  debugfs

nodev  sockfs

nodev  usbfs

nodev  pipefs

nodev  futexfs

nodev  tmpfs

nodev  inotifyfs

nodev  eventpollfs

nodev  devpts

        reiserfs

        ext2

nodev  ramfs

        msdos

        vfat

        iso9660

nodev  smbfs

nodev  mqueue

[/pre]

 

Link to comment
  • 4 weeks later...

How were you guys mounting the NTFS file systems? When I tried to mount, I was told that ntfs was an unknown file system type. Were you using fuse (I think that's it) from SourceForge?

 

Cheers,

 

 

I was pretty sure that NTFS used to be in there as a "read-only" filesystem driver, but I just looked at my current 4.0 final server.  It's not in there... ???

 

Tom, is my memory failing me?

 

Joe L.

 

 

Joe,

 

Is there an earlier version that you know has the ntfs file system.  I wanted to use your proposed method for moving data to an unraid drive.  I have an extra tower where I can use my additional unraid os to move the files and then move the file between two unraid systems (one unraid would have the current build, and the other would have the build you suggest that has the ntfs drive support)

 

Thanks

Unraid

Link to comment

You could also use a live CD to do that. Choose one that have ntfs support and you're done. (or a flash based one if you don't have a cd drive handy)

 

Orb,

 

This looks like a great option for moving files from NTFS to an UnRaid system.  I downloaded "Live CD" and then booted using the iso.  I had an NTFS drive and an unraid drive hooked up.  I was able to mount both, but I could not copy data from the NTFS drive to the Unraid drive.  I made sure to turn on read/write permissions, but that still did not allow for files to be created, or copied to the unraid drive.  I am sure it is a setting in Live CD that I am missing.

Are there any tips that you can give?

 

Albin

Link to comment

I guess it depends on which LiveCD you used, Ubuntu ? Slax ? Knoppix ? DSL ?

 

I would use Slax because it is Slackware based as is unRAID, but I will check that reiserFS is compiled with write support in the LiveCD you currently have, and also look into access permission on hte drive you want to copy to (to makes things easy you could put it to 777 if it's not already, and change it back after the copy to whatever it was)

Link to comment

I guess it depends on which LiveCD you used, Ubuntu ? Slax ? Knoppix ? DSL ?

 

I would use Slax because it is Slackware based as is unRAID, but I will check that reiserFS is compiled with write support in the LiveCD you currently have, and also look into access permission on hte drive you want to copy to (to makes things easy you could put it to 777 if it's not already, and change it back after the copy to whatever it was)

 

Orb,

 

I downloaded Slax and burned the iso to a cd.  I then tried to boot to the iso but got the "blank Screen" issue that seems to be related to my on-board video card.  Instead of waisting to much time on that I managed to try Slax in a different tower (different MB and different video card) and it booted fine (I booted using the StartX command).  After booting I mountedthe unraid formated drive and a ntfs drive and then copied a few files from the ntfs to the unraid drive.  (if anyone needs pointers on mounting drives in the slax gui, just let me know.  I am sure I am not the anyone struggling as a newbie with this kind of stuff) I then powered down and placed the unraid drive back in the unraid tower and powered up unraid.  I then browsed to the mapped network unraid drive and the files that I copied over from the ntfs drive to the unraid drive (with the use of slax and www.slax.org ) were present.  (I downloaded SLAX Standard Edition v 5.1.8.1)  So I had success with your proposed method.  Thanks so much for the tip.  I also noticed that when playing around with Slax I will be able to learn many of the copy, move, and other commands that will most likely be helpful with UnRaid.

 

Again thanks for the tip, and now I can move (consolidate) all my files off of several ntfs drives and on to my UnRaid drives at incredible speeds, and without the use the network or an xp machine.

 

By the way, All I did with the UnRaid machine, was to power down, pull the hard drive (not the parity drive) that I was going mount in Slax, then when I was done with the moving/copying of files from the ntfs drive to the unraid drive and had powered down slax, I then pulled the UnRaid drive from the machine that I used for booting to Slax and placed it back in the unraid machine and booted up Unraid and the files showed up.  I guess I could perform a parity check, but unraid says parity is valid (I am still learning about when to check parity, so an comments on this would be great, and if anyone see issues with what I did above please comment).

 

One other note.  The above process maybe a lot easier for others if they can use their current unraid machine to boot to slax.  I would of done it this way, but I would be sure to disconnect all drives except for the drive you want to move data to, and connect the ntfs drive.  If you want to have parity when moving files, then any of the above methods mentioned will not be helpful.

 

I know this is a long post, but I figured it may help others.

 

Albin

Link to comment

I guess I could perform a parity check, but unraid says parity is valid (I am still learning about when to check parity, so an comments on this would be great, and if anyone see issues with what I did above please comment).

Albin

Albin,

 

Good to know your file copy worked. Hopefully, Tom will add  NTFS filesystem support back in soon.  However...

Your  parity is now out  of date.  (almost worse than not having it at all)

 

You must now re-calculate party on the unRaid server.   You did your file copy when it was not looking (booted up on the other CD), so the current Parity contents  do not know of any of your new files. 

 

Please, force  a parity check now... sooner even,  as your existing, out of date parity data would clobber/manngle the contents of any  drive you attempt to recover using  it. 

Expect the parity check to find many errors.  In this case they are expected, and will be because the current parity calcs were done on the original contents of your target drives. (before you copied  the ntfs data to on the unRaid disks)

 

Once the parity check has run to completion, you will be protected again.

 

Joe L.

 

Link to comment

I guess I could perform a parity check, but unraid says parity is valid (I am still learning about when to check parity, so an comments on this would be great, and if anyone see issues with what I did above please comment).

Albin

Albin,

 

Good to know your file copy worked. Hopefully, Tom will add  NTFS filesystem support back in soon.  However...

Your  parity is now out  of date.  (almost worse than not having it at all)

 

You must now re-calculate party on the unRaid server.   You did your file copy when it was not looking (booted up on the other CD), so the current Parity contents  do not know of any of your new files. 

 

Please, force  a parity check now... sooner even,  as your existing, out of date parity data would clobber/manngle the contents of any  drive you attempt to recover using  it. 

Expect the parity check to find many errors.  In this case they are expected, and will be because the current parity calcs were done on the original contents of your target drives. (before you copied  the ntfs data to on the unRaid disks)

 

Once the parity check has run to completion, you will be protected again.

 

Joe L.

 

 

Joe,

 

Thanks for filling me in on the need for a Parity Check.  I am going to copy over several gigs to the UnRaid drive via Slax and then I will be sure to force a Parity Check.

 

Albin

Link to comment

Hi,

 

as a happy new unRAID user, I would like to share my experience of moving some gigs of music and movies the past couple of days.

A lot of the files have non-English characters, like ñ in Spanish or ø in Norwegian.

 

As suggested in the forums and the wiki, I decided to use cp command from the console for speed reasons. After copying, I wanted to check that everything had been moved. First I did an ls command on the console, and a lot of text rushed over the screen, looked promising. Then I used my alternative browser, FreeCommander, which also contains a synchronization tool. I was very surprised when all the files turned up as missing in target. Looking in the ordinary explorer part of FreeCommander, I could see them. Very strange. So, I did a compare using SyncBack and got a totally different result. SyncBack reported that most of the files were in the target, but I discovered quite a few files missing - and then whole directories missing.

 

A closer inspection showed that all files and directories containing Norwegian characters æ, ø  and å had not been copied. Files with Spanish, Portuguese, French and German special letters were copied, but the letters had been replaced with an underscore.

 

My next step was to go to Windows Explorer to copy the missing files. Bug surprise again: No files in the target directories! After some investigation, where FreeCommander gave me the clue as it shows the Attributes, it turns out that the cp command turns on a the Hidden flag and the System flag on each and every file, resulting in all files being invisible in Windows Explorer.

 

I managed quite easily to change the attributes with the chmod command, but when I realized how long it would take me to correct all the underscores and manually copy the missing files, I decided the far easiest solution was to delete everything I had moved, and use Windows Explorer on the source PC to copy files to the unRAID server. I didn't have any means to take the time on any of my transfers, but my gut feeling is that the difference is not big, and the result turned out perfect with Windows Explorer. You can even start several copy sessions in parallel before going to bed, as Windows does a great job in this respect.

 

Well this turned into a long post, but my conclusion is very simple: avoid the Linux commands for file operations, as they don't give you the needed language and Windows support. I have been working professionally in the IT industry since 1983, and I have seen the lack of language support from US developers being repeated again and again. Now, as far as I know, Linus Torvald is Finnish, so he should know better...

 

Hakon

(my name is actually Håkon, but many mail systems still have problems with the å...)

 

ps. I have updated the wiki description of transfer from network shares to unRAID.

 

 

Link to comment

Hi,

 

as a happy new unRAID user, I would like to share my experience of moving some gigs of music and movies the past couple of days.

A lot of the files have non-English characters, like ñ in Spanish or ø in Norwegian.

 

As suggested in the forums and the wiki, I decided to use cp command from the console for speed reasons.

 

...

 

 

Use of the 'cp' command is documented in the wiki, but we never intended this to be the primary means of moving files.  This is mainly because one of our design goals is to hide the underlying linux OS as much as possible.  Also, as you have observed, there is not much of a speed improvement over moving files with Explorer anyway.  Nevertheless, as unRAID OS evolves, we plan to make it easier to cusstomize the platform and fix these annoying problems, such as...

 

My next step was to go to Windows Explorer to copy the missing files. Bug surprise again: No files in the target directories! After some investigation, where FreeCommander gave me the clue as it shows the Attributes, it turns out that the cp command turns on a the Hidden flag and the System flag on each and every file, resulting in all files being invisible in Windows Explorer.

 

This is just an artifact of the default file create permissions for 'root'.  What's happening is that Samba (which implements Windows Networking protocol) is "overloading" some of the unix file permission bits to implement the DOS 'Hidden' and 'System' attributes (which are meaningless in the unix world).  This is a known "issue" for which you can search to find more info on.  In a nutshell, since unRAID is meant to be used in a Windows Networking environment, we enable the Samba feature that implements 'Hidden' and 'System'.  We plan to make this configurable in an upcoming release.

 

...

Well this turned into a long post, but my conclusion is very simple: avoid the Linux commands for file operations, as they don't give you the needed language and Windows support. I have been working professionally in the IT industry since 1983, and I have seen the lack of language support from US developers being repeated again and again. Now, as far as I know, Linus Torvald is Finnish, so he should know better...

 

Don't blame Linus!!  We take full responsibility for not including all the necessary code pages in the unRAID build.  This was done to conserve space, but again, we plan to correct this in a future release.

 

Hakon

(my name is actually Håkon, but many mail systems still have problems with the å...)

I guess those would be the mail systems created by US developers  ;D

 

ps. I have updated the wiki description of transfer from network shares to unRAID.

Thank you!

Link to comment

Thanks Tom,

 

I have no problems at all that these features are not being "built into" the underlying OS, I think you have done the right decision to prioritize a small footprint. I just want others to be aware that they can lose some files in the transfer process if they are "cheating" by using Linux commands rather than Windows OS tools. I assume what happens is that you are bypassing the layers 5-7 in the OSI model if you do this, which are taking care of formats and presentation.

 

Now I am more worried about the disk I put in yesterday, which was working fine, but now it turns up as Unformatted. It was a brand new 500G Samsung, and I have successfully moved my music and pictures there. Of course, I still have the source files, so I will push the Format button. If this fixes the problem, I actually get worried. If it is not a HW problem, why did it suddenly turn unformatted? I assume that the array should recover the lost data after formatting...  Well, I assume this belongs in another thread.

 

Hakon

Link to comment

To possibly save a  lot of time, do NOT re-format/reload. (at least not yet)

 

Instead, take the array off-line, power it down, and then power up once more.  See if the disk is detected as being formatted once more. It could be that the web-interface is confused to the disk status.

 

You might want to run a reiserfsck command on the file-system (yes,another OS command) to check the file-system for errors. (you can do a search for how to do this rather than take this thread off topic too far.

 

Joe L.

Link to comment

Joe, thank you for your advices! Unfortunately, I had already formatted the disk. I had to reformat 2-3 times, but now it has been stable for 2 days after I checked the connectors and changed the power connector. I will let you all know (in the hardware forum if I get any more problems with the Samsung disk, as I have seen some more posts reporting problems with it.

 

Håkon

Link to comment
  • 1 month later...

How were you guys mounting the NTFS file systems? When I tried to mount, I was told that ntfs was an unknown file system type. Were you using fuse (I think that's it) from SourceForge?

 

Cheers,

 

 

I was pretty sure that NTFS used to be in there as a "read-only" filesystem driver, but I just looked at my current 4.0 final server.  It's not in there... ???

 

Tom, is my memory failing me?

 

Joe L.

[pre]

root@Tower:/proc# cat filesystems

nodev  sysfs

nodev  rootfs

nodev  bdev

nodev  proc

nodev  debugfs

nodev  sockfs

nodev  usbfs

nodev  pipefs

nodev  futexfs

nodev  tmpfs

nodev  inotifyfs

nodev  eventpollfs

nodev  devpts

        reiserfs

        ext2

nodev  ramfs

        msdos

        vfat

        iso9660

nodev  smbfs

nodev  mqueue

[/pre]

 

 

I ran into the same issue on unraid 4.1 however I discovered that the kernel module for ntfs support IS included.  It is just not loaded by default.

 

I loaded it temporarily using the following command:

 

modprobe ntfs

 

and the response to cat /proc/filesystems:

 

root@tower:~# cat /proc/filesystems
nodev   sysfs
nodev   rootfs
nodev   bdev
nodev   proc
nodev   debugfs
nodev   sockfs
nodev   usbfs
nodev   pipefs
nodev   anon_inodefs
nodev   futexfs
nodev   tmpfs
nodev   inotifyfs
nodev   devpts
        reiserfs
        ext2
nodev   ramfs
        msdos
        vfat
        iso9660
nodev   smbfs
nodev   mqueue
        ntfs

 

I hope this helps someone else trying to copy data from an ntfs disk.

 

-kenshin

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.