Need Serious Migration Assistance


Recommended Posts

I have been a VERY long time user of UNRAID, I migrated from a very early version to my current system about 6 years ago and it's served me very well.  Unfortunately this system was built when hard drives were smaller than they are now so I have a system with way too many drives in it.  I'm starting to have drive issues and I would like to come up with a plan to upgrade from my current 4.5.6 version of UNRAID and get away from all these smaller disks and get to an array that has only maybe 5 total disks but as large as I can afford.

 

Where I'm struggling is how to get the upgrade and the data migration done, smoothly and without loss  I would be willing to purchase new computer hardware and start with a totally new system but I'm not sure i need to do that as the current SuperMicro system was top of the line many years ago and the hardware is running fine.

 

Can I do an in-place upgrade to 6.X on the existing box, put in a larger parity drive and start moving files around...?  I'm thinking this may be risky, what would be the steps.  I have 16 total drives today and not a single spare driver bay so that needs to be considered as well.  I would like to go with at least 4TB drives, and I have about 20TB of data to move onto the new drives I buy so that I can remove the old ones.  That too is a challenge as removing a drive from the array is never simple.

 

Any thoughts on the best way to accomplish this large task...?  And what the steps would be to this OS and data migration...?

Link to comment

First step would be to check that your current hardware is suitable to run V6 on.  Because if it isn't then plans are going to be different than if you try and use your current hardware.

 

Would be a good idea to post your machine specs so we can advise on suitability.  

Edited by CHBMB
Link to comment

Do you have backups? Make sure you have any important and irreplaceable files copied to another system before making any changes.

 

Have you read the Upgrading to UnRAID v6 wiki? It has a bit on how to make sure you can run 64bit V6.

 

After upgrading, you should be able to access all your disks just as before. Then replace parity with a larger disk and let it rebuild. Then replace one of your data disks with a larger disk and let it rebuild.  At that point you should be able to copy files from one (or more) of your older disks to that new larger disk so you can remove it (them). Then you might consider adding a 2nd parity disk.

 

After you have enough free bays, I would start with adding a new larger disk as XFS, then you can start converting all your disks to XFS as explained in the File System Conversion wiki.

 

Read the wikis mentioned and then come back with questions, and we can take this one step at a time until you get the hang of it.

Link to comment

Hey folks, thanks for the guidance...!  It seemed to me that there was little risk just creating a new flash drive and giving it a go.  That way if all did not go well I could still put in the old version 4 flash and boot from it.  Well it looked like it was going to actually work OK.  Everything came up, I was able to re-assign all the drives and then started up a new parity check. I looked last night before bed and I had 6 hours left to the check.  This morning the array is alive but the parity drive was marked with an X and I could see that drive 15 had a shit load of errors.  Down in the control area it said parity was stopped by user, I did not stop it.  Drive 15 was one that I was having issues with running V4 so I decided to remove it from the array but once I did that my parity drive is now gone as well as drive 15.  And I can't re-assign the parity drive as it's no longer even in the drop down.

 

HELP....!

Link to comment

UPDATE - A cold re-start of the box and now the original parity drive is shown again so I re-assigned it.  Started a new parity check and it's running BUT crazy slow this time.  I normally get an average of 60 MB/Sec but it's now running at just above 4 MB/Sec and says it will take 5 days at this speed.

 

2nd UPDATE - I let the parity check run for a while and it has picked up some speed, not normal speed yet but it's close.  I'm going to let it run till finished and see what I have then.

Edited by rharvey
Link to comment

I am just seeing this thread. 

 

When dealing with an older, "fragile" array, I would recommend leaving it alone and building a new modern server. You didn't mention the total data capacity of your existing server, or the actual disk sizes involved, but with 8T drives you'd be able to drop the drive count significantly. Once the new server is ready, you could copy the data over the lan without any changes to the old server, which could live on as a backup server.

 

But given where you are, this is what I would do. I would buy some 8T drives, preclear them, and create a new array. Maybe 3 drives to start? You probably have to physically remove some of the existing disks (or remove then all, see below) to make room. I would not assign a parity drive to start, feeling that speed is going to be valuable to get data moved over quickly from the fragile older drives. But having parity installed has advantages to ensure the new array is working and reliable. There is also a write mode called turbo write you could use. It would be up to you. You can install the unassigned devices plugin to mount your existing data disks a few at a time to copy data over to the new disks. You'd be rebooting and physically swapping out the ones you copied from, and swapping in the drives you removed. Once all the data is copied over, you'd build parity if you didn't already have.

 

You might consider removing all the existing disks from the server, with plan to install them one (or a few) at a time to migrate their data. That wound give time to preclear the new disks and test out the new array without the older disks going along for the ride. 

 

I'd be worried about the cabling. It is so delicate, and even a small nudge can cause drives to go offline and errors to occur. Problem is exponentially worse with lots of drives. I strongly recommend locking cables and drive cages. If you don't have these, be especially diligent that the corrections on both sides are firm and secure. If you have cables that slide off and on with near zero effort, replace those with different cables that provide better friction. Another reason to remove all the old drives is to simplify your cabling. If you have 3 8T and two of the older drives, that's only 5 disks. Not sure your current count, but imagining 12+.

 

Newer unRaid versions have some issues with some controllers, notably the SuperMicro SASLP and SASLP2. There are settings you can change to likely be able to made these work, but initially I'd focus on using motherboard ports. You may not need addon controllers with the bigger disks.

 

Post back with questions.

 

Good luck!

  • Upvote 1
Link to comment
On 3/4/2017 at 6:18 AM, rharvey said:

UPDATE - A cold re-start of the box and now the original parity drive is shown again so I re-assigned it.  Started a new parity check and it's running BUT crazy slow this time.  I normally get an average of 60 MB/Sec but it's now running at just above 4 MB/Sec and says it will take 5 days at this speed..

 

It's been about 5 days... It should almost be done by now :) On my old server, this would always happen to me after poking around inside the case.  Re-seating or replacing the SATA cables would correct it.  

Link to comment
On 3/8/2017 at 7:25 PM, dboonthego said:

 

It's been about 5 days... It should almost be done by now :) On my old server, this would always happen to me after poking around inside the case.  Re-seating or replacing the SATA cables would correct it.  

Sorry for the delayed response, yes that parity check did finally complete.  From that point I started moving files off drives that I was not sure about and had issues with previously.  Got the array down from 16 data drives to 12 without issue, just took some time but coping right on the server is way faster.  So now I'm in the process of replacing old 1TB and 1.5TB drives with new Seagate IronWolf NAS 4TB drives.  I'm on data drive 3 of 5 right now, as expected this is taking time to complete but it's working so far.  One very odd thing is that when the data re-builds are done the GUI becomes un-reachable, this has taken place on both of the two first drives I swapped out. The array is alive, the shares are reachable, even PLEX Server is working (a nice new benefit of version 6) but the GUI is not.  A re-boot brings everything including the GUI back, I just don't like powering the box down without looking into it's real state at the time.  This is my first upgrade in MANY years and my expectations were very high for a far more capable and stable UNRAID.  What I think I have found with just a few days experience is a far more capable product (like docker) but not one that is more stable, it still seems very finicky.  Once I have ALL of the old disks out and ONLY new NAS quality drives in it maybe it will be more stable.  So once I have all the 6 new 4TB (1 parity, 5 data) drives swapped in I will move the rest of the files on the old smaller drives over to the new ones and then lastly pull all the old drives out.  At that point I will have an array about the same size I had but with far fewer drives and higher quality ones at that.  Hopefully things will be more stable by then and I can enjoy a trouble free UNRAID for years to come.  One thing I have not been doing is pre-clear, I know it would have been smart to do but without a spare box to pre-clear with it would have added several days to the entire process.

Edited by rharvey
Link to comment
52 minutes ago, rharvey said:

One thing I have not been doing is pre-clear, I know it would have been smart to do but without a spare box to pre-clear with it would have added several days to the entire process

Rebuilds followed by a parity check is a test to some extent. Be sure you setup Notifications so unRAID can notifiy if any of your disks begin to look suspect.

 

If you want any help diagnosing any issues you will have to post a diagnostic.

 

Link to comment

Hey guys - The migration continues but I hit a roadblock....!  Have all the 4TB drives installed and they are all working fine.  I'm now starting to MOVE data from the smaller (old) drives over to these new 4TB drives and I'm having an issue with it.  I used the "New Permissions" script to allow me to make quick and simple work to copy about 10GB and thousands of files nested in folders.  That script does not seem to work on all files and my "mv" commands are failing.  I'm not good with command line stuff so I'm a little lost as to why the script does not always work.

 

Any thoughts...?

Link to comment
3 minutes ago, rharvey said:

Hey guys - The migration continues but I hit a roadblock....!  Have all the 4TB drives installed and they are all working fine.  I'm now starting to MOVE data from the smaller (old) drives over to these new 4TB drives and I'm having an issue with it.  I used the "New Permissions" script to allow me to make quick and simple work to copy about 10GB and thousands of files nested in folders.  That script does not seem to work on all files and my "mv" commands are failing.  I'm not good with command line stuff so I'm a little lost as to why the script does not always work.

 

Any thoughts...?

We would need to see exactly what errors you are getting with exactly which files.

Link to comment

I think I just figured out my issue, the script in the Lime GUI only set "rw" on files and does not set the "x" which would allow me to delete the files.  The error I'm getting is that the files can't be deleted.  DUH - Is there any way to adjust the script to include "rwx" on files...?

Link to comment
1 hour ago, rharvey said:

I think I just figured out my issue, the script in the Lime GUI only set "rw" on files and does not set the "x" which would allow me to delete the files.  The error I'm getting is that the files can't be deleted.  DUH - Is there any way to adjust the script to include "rwx" on files...?

You only need x on the directories, and newperms does this for you.

Link to comment
I think I just figured out my issue, the script in the Lime GUI only set "rw" on files and does not set the "x" which would allow me to delete the files.  The error I'm getting is that the files can't be deleted.  DUH - Is there any way to adjust the script to include "rwx" on files...?

The X stands for eXecutable has has no bearing on being able to delete or not

Sent from my SM-T560NU using Tapatalk

Link to comment

Unix filesystem ACL in a nutshell:

there are 3 permission groups (owner, group, other)

owner - if you own the file, these permissions apply

group - if you are a member of the group, these permissions apply

other - if you are not the owner or member of the group, these permissions apply

there is no deny permission, so if one group grants it, it is granted.

 

To read a file: read permission is needed. r--

To write a file: write permission is needed. -w-

To execute a file (as a script, or binary): execute is needed. --x

 

To list a directory: read and execute is needed. r-x Weird things happen otherwise

To create/delete files in a directory: write is needed on both the file and the directory. -w-

 

so for a file /mnt/user/share/a/b

drwxrwxr-x 1 nobody users 2 Mar 15 11:57 a/
-rw-rw-rw- 1 nobody users 2 Mar 15 11:57 a/b

Other than root, nobody or members of users. the file b would be impossible to delete, since the write permission to the directory is missing.

The file however, can be overwritten by anybody.

  • Upvote 2
Link to comment

Correct me if I am wrong, but can you not just open a terminal (i.e putty)  login with root and pw

and run midnight commander?   type mc and enter.  Then browse to /mnt/disk(whatever the old disks number is)  and then on the right side, /mnt/disk(new disk number) and then F6 to move over the files.

 

You switch sides with the TAB key (You can also use the mouse). To select several folders hit the insert key on each folder so they turn yellow.

 

I am currently at work so I can't show a proper screenshot of it but this is how the program looks like.

maheshadragonbsd-mc.jpg

Link to comment
8 hours ago, ken-ji said:

Unix filesystem ACL in a nutshell:

there are 3 permission groups (owner, group, other)

owner - if you own the file, these permissions apply

group - if you are a member of the group, these permissions apply

other - if you are not the owner or member of the group, these permissions apply

there is no deny permission, so if one group grants it, it is granted.

 

To read a file: read permission is needed. r--

To write a file: write permission is needed. -w-

To execute a file (as a script, or binary): execute is needed. --x

 

To list a directory: read and execute is needed. r-x Weird things happen otherwise

To create/delete files in a directory: write is needed on both the file and the directory. -w-

 

so for a file /mnt/user/share/a/b


drwxrwxr-x 1 nobody users 2 Mar 15 11:57 a/
-rw-rw-rw- 1 nobody users 2 Mar 15 11:57 a/b

Other than root, nobody or members of users. the file b would be impossible to delete, since the write permission to the directory is missing.

The file however, can be overwritten by anybody.

 

I was tempted to literally move your post straight to the FAQ!  And add some FAQ-ness to it, preceding questions like "Please explain Linux file and folder permissions!", and "What's it take to delete a file?", and probably others that users ask.  Would you be interested in starting a FAQ entry based on the above?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.