Re: Format XFS on replacement drive / Convert from RFS to XFS (discussion only)


Recommended Posts

37 minutes ago, johngalt said:

I read this thread too late and think I messed up while trying to convert from Reiserfs to xfs. I was trying to replace a 6TB drive with an 18TB drive, selected xfs as the file system, formatted the drive, and started the rebuild. Now it looks like the new drive is empty while the rebuild is completing (first post in this thread supports that). What's the best method here to try to recover the data? Just put the old 6TB drive in another computer and transfer over?

When I moved my array from RFS to XFS (12disk) my process was:
 

  • Install a new disk to array (giving ME a total 13 disk in this example)
  • Format to XFS
  • SSH into the array
  • start a 'screen' session
  • Create a folder on my BLANK XFS disk of the last 3 digits of the serial number of the RFS disk with the data on it (this was only to keep track of which disk I was moving data from and not necessary) 
    • mkdir /mnt/disk2/812
    • run copy command
    • cp -rpv /mnt/disk8/* /mnt/disk2/812/
  • this example: This command will start copying data from disk 8 (RFS) to disk 2 (xfs)
  • once the above has completed and data is confirmed, I would select disk 8 (RFS) in the unraid GUI, and format it to XFS
  • from there I would rinse and repeat this process to each disk

 

The process you did is incorrect afaik. This is because you pulled a 6TB (RFS) and installed a 16TB. Your process to do this correctly, and remember this is ONLY my opinion:

  • Make sure the data on your 6TB contains the data your 'missing' DO NOT INSTALL THIS DISK INTO THE ARRAY AGAIN BEFORE CONFIRMING DATA
    • you can access your 6TB disk on a Ubuntu system (workstation or laptop) boot ubuntu from USB and use a HDD dock to mount the disk on this workstation system and confirm data is there
    • If the dats is still on the 6TB disk you pulled out of the array ...be thankful.
    • as the 16TB is a blank XFS disk IN your array as you stated above, you can take your 6TB disk and HDD dock and plug it into your array via USB
    • From there you will need a plugin/app called unassigned devices
    • Mount the 6TB disk (I do believe it can mount RFS) I may be wrong and it may only see FAT/NTFS disk
      • If I am correct, and UD detects the RFS disk you can copy all data from the 6TB to your 16TB
      • If I am wrong, you will be required to:
        • Copy to 6TB data to a spare NTFS disk and then mount that NTFS disk in UD -or-
        • Copy the data from the 6TB disk over the network to your array 16TB disk

 

You disk is blank because when you installed your 16TB the array detected the new disk and rebuilt the data to this disk, which is correct. It also saw the disk FS was RFS, which is correct.

However, when installing the new disk, as you did, you do not want to rebuild > format as XFS >  that will format the disk and all data will be gone.

Perhaps my method is overly complex, but it worked really well for me at the time of this conversion.

 

To sum it up, when moving from RFS to XFS in an existing RFS array, my opinion is you need to install an additional disk to the array, copy data from RFS disk to XFS disk

Then

Format the RFS disk to XFS, and carry on the process.

Wow long winded :-) I was attempting to be as detailed as possible, and I hope NOT too confusing. The community is great here and may have a simpler method. I would like to restate this is my method.

 

Keep us posted how you make out.

Cheers

Edited by bombz
  • Like 1
Link to comment
32 minutes ago, johngalt said:

I was trying to replace a 6TB drive with an 18TB drive, selected xfs as the file system, formatted the drive,

You cannot change the file system type during a rebuild process.  This would format the emulated drive to an empty XFS files system and updated parity to reflect this.  The rebuild then makes the physical disk match the emulate one so you end up with the empty XFS file system.

 

36 minutes ago, johngalt said:

What's the best method here to try to recover the data? Just put the old 6TB drive in another computer and transfer over

This.  You could also mount it under Unassigned Devices on the Unraid server.

 

You could also probably recover most of the data by running reiserfsck against the rebuilt drive but this does not  seem necessary in this case.

 

Link to comment

Yeah, I now know where I went wrong (and also shouldn't start a process I don't fully understand at 5am). It's just a matter of determining the best path forward. I figured at this point I should just let the drive continue the rebuild, but didn't know if stopping that and putting the old drive back in might have a recovery solution.

 

Looks like I'll just let the new disk continue the data rebuild process, then move the files from the old one to the new disk, then start moving the data from the old rfs disks to the new xfs and so on until all the disks are xfs.

Link to comment
1 hour ago, johngalt said:

Looks like I'll just let the new disk continue the data rebuild process, then move the files from the old one to the new disk, then start moving the data from the old rfs disks to the new xfs and so on until all the disks are xfs.

That would be what I would recommend as the easiest way to proceed.

 

You can check that the old disk mounts OK and you can see all its contents ready to be copied.   Only if that is not possible would I consider doing anything else.

 

Link to comment
  • 4 weeks later...

It's been a bumpy road but I now have three new 18TB drives in my server and am ready to start the conversion from ReiserFS to XFS. I currently have (7) 6TB, (5) 10TB, and (2) 18TB drives, plus an 18TB parity drive. The two 18TB drives are already XFS and I have 27TB of free space for moving files around. I've read through this thread and the wiki for the process. I was planning on using unBalance to move the files from disc to disc.

 

I just had a question about the following: "small shares may be scattered across the drives; some data may be moved multiple times". Is this anything I need to worry about? If something gets moves multiple times, what do I look for?

 

I was also thinking about just mapping each drive in windows and using File Explorer to move the files from one drive to another and was wondering what the downside and risk would be with this method?

Link to comment
2 hours ago, johngalt said:

"small shares may be scattered across the drives; some data may be moved multiple times".

Do you have an actual quote from somewhere for context?

 

2 hours ago, johngalt said:

I was also thinking about just mapping each drive in windows and using File Explorer to move the files from one drive to another and was wondering what the downside and risk would be with this method?

I usually discourage sharing disks since you can have unintended consequences including data loss if you try to work with user shares and disks at the same time. And getting another computer in the middle of this over the network has got to be slower than moving things on the server. unBALANCE seems like a good approach.

 

When I did this many years ago I used Midnight Commander (mc from the command line), which is built-in to Unraid. It is a text-GUI similar to some of the file managers from the DOS days. Google it to see if you might like it.

Link to comment
10 minutes ago, trurl said:

Do you have an actual quote from somewhere for context?

 

 

It's in the wiki for the file system conversion:

 

"Share based, no inclusions, preserving parity

Note: only for systems with no share inclusions and exclusions, that is, all shares are allowed to use all data drives

Use rsync or unBALANCE or MC to save all of the data on a ReiserFS drive to the other data drives; then format it with XFS, and repeat on another data drive, continuing until all drives are converted

Advantages: simpler to understand, no drive swapping

Disadvantages: some planning and space management needed; small shares may be scattered across the drives; some data may be moved multiple times, slowing down the overall speed; normal system operation is possible"

Link to comment

Seems confusing to me. I probably read the original post it was based on long ago.

 

Since you will be working with disks and not user shares I don't see what shares has to do with it.

 

And since you are going to be reformatting each disk as you empty it and then move files to the reformatted disk I wouldn't worry too much about moving things multiple times. You could probably minimize that to some extent by converting the largest drives first.

Link to comment

I think that method could result in a lot of data being moved more than once.  I used this method but it does require adding another data disk to the array.  (Or , at least, figuring out how to completely empty one data disk as a preliminary step...)   (BTW, with moving the data only once takes about about 2-to-3 hours per TB as I recall...)

 

https://wiki.unraid.net/index.php/File_System_Conversion#Mirroring_procedure_to_convert_drives

 

However, it is very confusing to read and I made a table of the disk numbers involved for every step of the procedure.  I checked off each step as I did it.   (Maybe not necessary but it is just too easy to become confused and a misstep could result in data loss.) 

Link to comment

First disk completed successfully using unBalance and onto disk 2 now. Will probably take a break after that to ensure that everything is up and running so I can broadcast the Super Bowl throughout the house on Sunday.

 

This might be a stupid question, but as long as there are no files left on the disk, can I assume the files were transferred successfully? No risk of an error happening during the transfer and the file got moved, but corrupted during the process?

 

I also noticed that my appsdata folder was spread across all 14 of my disks. My plan for this is to move everything from each disk's appsdata folder to a single disk and then add each disk to the exclude list for appsdata as I go. Theoretically it would then be consolidated to a single disk, and every other disk is on the exclude list. I'll add a SSD for a cache drive after everything is xfs and then move the folder over to that. Does this sound like a good way to go about that?

 

From someone who is a casual "set it and forget it" user, this forum has been a great help. As you can tell, I hardly do any maintenance on my server and am getting caught up on a lot right now. I'm really appreciative of all the help you guys provide.

Edited by johngalt
Link to comment
  • 5 months later...

Finally getting around to formatting my Reiser FS formatted drives to XFS.  I'm sure there were more efficient ways to do this, but here's where I'm at now:  I've used unBALANCE to move just about everything off one of my RFS drives to my new big XFS drive.  But there's still some stuff that it couldn't move.  A couple of temp files, and maybe some directories? 

 

Here's a screenshot from within the unBALANCE GUI:

tLDNrws.png

 

Is it likely safe to reformat that (nearly) empty drive?

If so, what are my next steps?  I'm a "set it and forget it" type of guy, so I have to learn how to mess with these disks every time I need to change something.  After I stop the array, is it a simple option in the GUI to just reformat the empty disk? 
 

  • Like 1
Link to comment
  • 1 month later...

Preparing to do Reiserfs to XFS conversion on some drives. Running 6.9.2 PRO version.

4TB parity, all assigned data drives 2TB – 5 reiserfs, 3 xfs (2 of these are empty), 2 – 3TB drives precleared/unassigned.

 

Question using the 'Mirroring procedure' https://wiki.unraid.net/index.php/File_System_Conversion ,

If using an empty 2TB disk as the swap disc, which is already assigned and formatted xfs, can/should I still do step 4 settings/global shares settings/excluded disks" with the array stopped?

 

An additional confusion point is the "NOTE" above the procedure which says "Note: the swap drive should not be formatted with XFS (the resultant file system); if it is, then you will need to clear it or format it to something else first"

Any clarification will be appreciated.

Link to comment
  • 3 weeks later...

I followed the procedure in the linked wiki, and after step 16, the data in shares on the formatted drive do not appear in those shares. 

I copied disk22(swap) from disk21(old ReiserFS) and reassigned their drive numbers so now disk21(XFS former swap) and disk22(old ReiserFS)

now in /mnt/user/ShareName the data which should reside on disk21 does not appear but there is a /mnt/user/disk21 directory no other disk appears under /mnt/user

Edited by oryhara
incorrectly stated disk21 was not with other /mnt/disk## but it is i just didn't see it edit again clarified summary of state
Link to comment
33 minutes ago, oryhara said:

I followed the procedure in the linked wiki, and after step 16, the data in shares on the formatted drive do not appear in those shares. 

I copied disk22(swap) to disk21(old ReiserFS)

now in /mnt/user/ShareName the data which should reside on disk21 does not appear but there is a /mnt/user/disk21 directory no other disk appears under /mnt/user

 

I seem to recall that this can happen if you miss putting the trailing slash on the command line.   I have circled the one that I am talking about.

image.thumb.png.eb81092514b06bf67269dc8be8170d43.png

 

The trailing slash is a real hassle to most of us who are not Linux/Unix command line Gurus.  Most of the time, it does not make any difference.  But when it does weird things happen....    (&*$#**  and many other four letter words at this point.)

 

As a point of information, once you get a set of the command lines used for the conversion of the first disk, use the BASH shell editor to recall each line and then edit it to change the parameters that need to be modified.  It can save making these types of mistakes.  The up, down, right, left arrow keys work as does the <delete> key.   Try it out by working with the following command.  (You can't do any damage with it!)

 

ls -al /mnt/disk1/

 

Edited by Frank1940
Link to comment
  • 3 weeks later...

Just a couple of suggestions from the penny section on the “File System Conversion” doc. (Which is great, thank you!) Because of what I put in the red rectangle makes what’s in the green rectangle a little confusing since the file system is not shown, it’s now set to auto…. 
 

Also, when you talk about the checkbox for parity and formatting, you may want to mention it’s a bit obscured at the bottom of the page..

4A05D25D-0420-48E4-851E-6212CAE26210.jpeg

Link to comment
  • 1 month later...
1 hour ago, dtyree said:

it's mentioning v6.2. 

 

I thought the latest version was 6.11 

I assume that you are talking about things like this:

 

image.thumb.png.4e76caf9db45f0acf6203f974b80631e.png

 

Please note that it says v6.2 or later.   This is the procedure that you want to use for 6.11.x (or even 6.10.x).   If you are still running a version before 6.2, then there is a separate thread to be used for those folks.      Version 6.2 was released in March of 2016... 

 

 

 

Link to comment
  • 1 month later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.