XFS conversion advice\remedial education


TSM

Recommended Posts

Hi everyone,

 

I was hoping for some advice and remedial instruction.

 

I want to convert all my drives to XFS.  I've got 14 data drives of varied size and space used.  I've also got files from 5 user shares strewn across pretty much each of the drives.  All drives are included in each share.  I started the process already by simply copying the contents of some of my smaller drives (1TB) onto my newest largest data drive(6TB), utilizing Windows explorer on my Windows 7 workstation over the network.  Copying 1TB of data from one drive to another on the unraid via my Windows 7 machine, over my home network took a very long time.  I'm not sure how long exactly, because each time there was an error or 2 that would pop up that I'd have to address, but I might not have known about it for a few hours.  I'd say both of my 1TB drives, and a 1.5TB drive, each individually took less than a day, but more than 12 hours to clean off in this way.  So using this methodology my larger drives would take several days.  Plus I will soon get to the point where I don't have enough space on a single drive to move the contents of some of my larger drives to.    My idea there for example, would be to exclude the drive from each of the user shares, and then copy it's contents onto each user share individually.  I think there is enough space on the rest of the drives to accommodate the files I have on the 6TB drive, but they would all have to take some files.  Once I did that it would be easy to move the files from the next largest drive onto the 6TB drive, and keep going like this until all drives are converted.  But this method will be very time consuming, and seems a bit silly when logically I know there has to be a better way.

 

I'd like to be able to do this all from the unraid box itself.  Leaving the network and my Windows 7 machine out of it.  Assuming I'm a complete newb, and know absolutely nothing, how would I go about this process?  I'm already familiar with telneting to the sever, which I've done on occasion or 2 to complete specific tasks. 

 

Thanks, in advance

Link to comment

First, use screen, to install http://lime-technology.com/forum/index.php?topic=37472

 

Use MC in a telnet session (using screen so you can disconnect/shutdown/whatever) to move files, not windows explorer. Using explorer will cause you to have to do 2 moves, one to your workstation, then another back to the unraid server. MC will move disk to disk and leave the network out of it. I think a full 2 TB drive took ~12 hours, so still not fast.

 

You were on the right track, but starting with your smallest drives was where you went wrong. Should've started with the largest. Doing the smallest first means you will have to move subsequent files to multiple drives. Doing the largest first would have ensured you always have room the for the next drive.

 

I would do your next largest now, however you can, and then continue going from largest to smallest.

 

http://lime-technology.com/forum/index.php?topic=37490.0

 

 

Link to comment

First, use screen, to install http://lime-technology.com/forum/index.php?topic=37472

 

Use MC in a telnet session (using screen so you can disconnect/shutdown/whatever) to move files, not windows explorer. Using explorer will cause you to have to do 2 moves, one to your workstation, then another back to the unraid server. MC will move disk to disk and leave the network out of it. I think a full 2 TB drive took ~12 hours, so still not fast.

 

You were on the right track, but starting with your smallest drives was where you went wrong. Should've started with the largest. Doing the smallest first means you will have to move subsequent files to multiple drives. Doing the largest first would have ensured you always have room the for the next drive.

 

I would do your next largest now, however you can, and then continue going from largest to smallest.

 

http://lime-technology.com/forum/index.php?topic=37490.0

Thanks for the response.  I've tried to use Midnight Commander before from a telnet window, but the screen just looked like a garbled mess.  I'm at work right now, but when I get home I'll post a screen shot of it.  Never tried it from the console, I guess I could attach a monitor to the box and do it from the console.  Can I move files from a disk to a user share in Midnight commander?  Does it intelligently handle folder merges and things like that? 

Link to comment

Thanks for the response.  I've tried to use Midnight Commander before from a telnet window, but the screen just looked like a garbled mess.  I'm at work right now, but when I get home I'll post a screen shot of it.

You have to set the right character set to get it to display sensibly in a telnet window.

Never tried it from the console, I guess I could attach a monitor to the box and do it from the console.

That should work fine and I do not think it suffers from the character set issue.

  Can I move files from a disk to a user share in Midnight commander? 
Midnight Command is not 'unRAID aware' so the same caveats apply as when using a PC/Mac to move files.  In other words you should avoid mixing disk and user shares in a copy or you may suffer data loss.  Another thing to note is that unless you take special action you will be running mc as the root user which messes up file permissions by changing the owner to root during copies so that you need to run the 'newperms' script afterwards to get them right again.

Does it intelligently handle folder merges and things like that?

Not quite sure what you mean by this.  Think of it as a gui enabled version of the Linux cp/mv command.

 

Link to comment

...  Can I move files from a disk to a user share in Midnight commander?  Does it intelligently handle folder merges and things like that?

Seems like you should move from disk to disk to better control where things end up since your objective is to make sure some disks get emptied while not making other disks too full.

 

In any case, there is a known bug that you should be aware of. See here. Probably safer to just do disk to disk moves.

 

Disk to disk moves using mc in screen is exactly what I did when I converted everything to XFS.

Link to comment

...  Can I move files from a disk to a user share in Midnight commander?  Does it intelligently handle folder merges and things like that?

Seems like you should move from disk to disk to better control where things end up since your objective is to make sure some disks get emptied while not making other disks too full.

 

In any case, there is a known bug that you should be aware of. See here. Probably safer to just do disk to disk moves.

 

Disk to disk moves using mc in screen is exactly what I did when I converted everything to XFS.

Holy Crap!, thank you for the link to that bug post.  I could have done something really bad.  I'm surprised LimeTech hasn't made that a top priority to fix.  Granted it will only cause a problem with specific situations for some users, but for those users in that situation it's a pretty big problem.  (Like Me!)

 

If I exclude the disk from the user share, and then create a new folder at the root of the disk.  Move all of the top level "Share Name" folders into that folder.  And then move them from within the new folder, that should work without a problem I think. 

 

Is there a way to make Midnight Commander aware of the user share?  Like could you mount it somehow from the Linux command line before opening MC? 

 

I agree disk to disk would be better.  But I'm trying to do this the easy way, and there might not be an easy way.  As long as I have space in the entire user share to accomodate the contents of the drive, why would a single drive getting too full be a problem?  Especially considering that once I get this step down, the drive in question will probably eventually get reformatted anyway. 

Link to comment
... and then create a new folder at the root of the disk...
Any folder at the root of cache or an array disk is automatically a user share, so don't be surprised when this new folder you create is listed among your user shares.
Is there a way to make Midnight Commander aware of the user share?...
The disks are at /mnt/diskn, cache is at /mnt/cache, and the user shares are in /mnt/user, so mc is already aware of the user shares.

 

There is also a /mnt/user0. This is the user shares excluding any files still on cache.

 

Once you figure out mc and see how things are laid out you may find that moving from disk to disk is not as complicated as you might think. Each disk will have a subfolder for each of the user shares that include that disk. For example, if you have a TV share and it has some of its files on disk1, then those will be at /mnt/disk1/TV. If disk1 also has files for a file called Movies, then there will also be a /mnt/disk1/Movies. If TV also includes disk2 then those are at /mnt/disk2/TV and so on.

 

mc by default is arranged with a left pane and a right pane, and some function buttons across the bottom for file operations. You can sort of think of it as similar to two Windows explorers windows, with one folder in the left pane, and another folder in the right pane. Your "selector" can be moved up and down with the cursor keys, and can be moved between the left and right panes with tab. There are plenty of guides on the internet, but it's pretty easy to figure out if you just play with it a little.

Link to comment

... and then create a new folder at the root of the disk...
Any folder at the root of cache or an array disk is automatically a user share, so don't be surprised when this new folder you create is listed among your user shares.
Is there a way to make Midnight Commander aware of the user share?...
The disks are at /mnt/diskn, cache is at /mnt/cache, and the user shares are in /mnt/user, so mc is already aware of the user shares.

 

There is also a /mnt/user0. This is the user shares excluding any files still on cache.

 

Once you figure out mc and see how things are laid out you may find that moving from disk to disk is not as complicated as you might think. Each disk will have a subfolder for each of the user shares that include that disk. For example, if you have a TV share and it has some of its files on disk1, then those will be at /mnt/disk1/TV. If disk1 also has files for a file called Movies, then there will also be a /mnt/disk1/Movies. If TV also includes disk2 then those are at /mnt/disk2/TV and so on.

 

mc by default is arranged with a left pane and a right pane, and some function buttons across the bottom for file operations. You can sort of think of it as similar to two Windows explorers windows, with one folder in the left pane, and another folder in the right pane. Your "selector" can be moved up and down with the cursor keys, and can be moved between the left and right panes with tab. There are plenty of guides on the internet, but it's pretty easy to figure out if you just play with it a little.

Thanks for all your help trurl.  I was able to get into Midnight Commander from the console, and I might muck around with something more powerful than telnet.exe, like putty with screen, in the future,  if I want to run it sitting at my Windows 7 machine.  But the 2 computers are sitting 2 feet from eachother, so as long as I have a monitor and keyboard connected to the unraid box, it's not that big a deal to use the console. 

 

With my original post, and the bug posting you linked to in mind, I guess the upshot is that no matter what tools I use to move the files off of the 6TB drive, it will be a manually intensive process, since I don't have a single disk with enough space on it to move all the files to.  I'll probably need to manually move individual folders across 7 to 10 disks, to make this work.  Even if I hadn't have migrated the smaller drives first, I'd have still had to spread it across maybe 4 to 7 disks.    Thus why the idea of moving the files from disk to user share was appealing to me.  Oh Well!  Maybe I'll wait a few months to do this and ask my wife to get me another 6TB drive for my birthday.  Or hey, maybe by then a higher performance 8TB drive will be out. 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.