Re: Format XFS on replacement drive / Convert from RFS to XFS (discussion only)


Recommended Posts

I believe when I did that it did not prompt me to format the drive, it just said "unmountable"

Did you look at the whole screen? I believe there is a checkbox near the start button that enables formatting of all unmountable disks.

 

There sure was... thanks!

 

This isn't the first time unRaid has made me feel particularly illiterate

Link to comment

I believe when I did that it did not prompt me to format the drive, it just said "unmountable"

Did you look at the whole screen? I believe there is a checkbox near the start button that enables formatting of all unmountable disks.

 

There sure was... thanks!

 

This isn't the first time unRaid has made me feel particularly illiterate

It's not entirely your fault. The main GUI layout has never to my knowledge been redone from the ground up. It's had bits and pieces added, moved around, improved, modified, etc, but never been totally reworked from scratch. There are many elements there that are recognizable from the first iterations that I used, almost 9 years ago.

 

Arguably, it works, so why mess with it, but then you get head scratchers like this for people who haven't been around it as long. If it really bothers you, it probably bothers others. If you have a constructive way to make it work better, PLEASE submit a feature request. Little details make a good product really great.

Link to comment

Is there a definitive RFS to XFS guide? Is it safe to follow http://lime-technology.com/forum/index.php?topic=37490.msg346739#msg346739 ?

 

That guide does still work.  However, I've made some improvements, and documented a complete step by step procedure here.  I think once you've read and understood the details of how both procedures work, you can invent your own customized procedure.

 

Please correct me if I'm wrong.

 

As far as I can understand, the main difference between the first guide (from bjp999) and the last one (from RobJ) is this:

 

- The first guide uses the cp command to make the copy and THEN the rsync -nrcv command to verify the integrity of the data.

- The last guide uses only the rsync -avPX command (rsync -rcvPX is optional)

I believe that the guide in the wiki (http://lime-technology.com/wiki/index.php/Upgrading_to_UnRAID_v6#Converting_Drives_from_ReiserFS_to_XFS) refers to the RobJ's guide but I cannot understand this part:

When all drives are done, and the last Reiser drive re-formatted, you have a choice:

Correct all of the drive includes and excludes for all shares, plus all other drive references

Or use New Config to re-assign all drives back to their original drive number (marking parity as valid)

If someone could explain this part...

 

 

So which method should we follow because all these guides are a little confusing.

 

Thank you.

 

 

Link to comment

...

Correct all of the drive includes and excludes for all shares, plus all other drive references

Or use New Config to re-assign all drives back to their original drive number (marking parity as valid)

 

If someone could explain this part...

After moving things around to other disks, if you have any user shares that are configured to include or exclude specific disk#s, you may have to change those settings to match where things have been moved.

 

Alternatively, you could set a New Config and change the drive assignments so that the disk#s match the user share settings.

 

If you only have single parity, changing the drive assignments will not invalidate parity, so you can check the box to tell it parity is valid.

 

If you have dual parity, then parity2 will not be valid if you change the drive assignments. I don't know if it will allow you to rebuild only parity2 or if it will do both in that case.

Link to comment

...

Correct all of the drive includes and excludes for all shares, plus all other drive references

Or use New Config to re-assign all drives back to their original drive number (marking parity as valid)

 

If someone could explain this part...

After moving things around to other disks, if you have any user shares that are configured to include or exclude specific disk#s, you may have to change those settings to match where things have been moved.

 

Alternatively, you could set a New Config and change the drive assignments so that the disk#s match the user share settings.

 

If you only have single parity, changing the drive assignments will not invalidate parity, so you can check the box to tell it parity is valid.

 

If you have dual parity, then parity2 will not be valid if you change the drive assignments. I don't know if it will allow you to rebuild only parity2 or if it will do both in that case.

Thank you so much for your reply.

I have one single parity disk and all my shares have no excluded disks (Included disks: All)

 

So in my case, after converting all drives to xfs and the smallest data disk is empty, should I run New Config or should I just do nothing?

Link to comment

The verification (rsync -rcvPX) found many differences between the files of the source disk and the destination disk.

Source disk used 1.68TB but the destination disk used 1.69

Why? Is it because of the transmission (nothing was downloading, only uploading) and plex docker applications that are running?

Most of these differences were from appdata/transmission/resume folder.

Should we stop all docker apps before rsync?

 

Now I messed things up! I changed the file system of the new disk that I bought to convert to xfs, thinking that I could reformat it (delete everything) and start over.

But now it's doing a data rebuild. The new (destination) disk is now unmountable (and unformatted).

Should I let data rebuild finish? And then?

 

Please help..

Link to comment

The verification (rsync -rcvPX) found many differences between the files of the source disk and the destination disk.

Source disk used 1.68TB but the destination disk used 1.69

Why? Is it because of the transmission (nothing was downloading, only uploading) and plex docker applications that are running?

Most of these differences were from appdata/transmission/resume folder.

Should we stop all docker apps before rsync?

If you were copying appdata then yes it is likely that running apps modified that data during the copy.

 

Now I messed things up! I changed the file system of the new disk that I bought to convert to xfs, thinking that I could reformat it (delete everything) and start over.

But now it's doing a data rebuild. The new (destination) disk is now unmountable (and unformatted).

Should I let data rebuild finish? And then?

 

Please help..

If you replace an existing disk then unRAID will rebuild the old disk's data to the new disk, even if the old disk was empty or just formatted.

 

Possibly it is unmountable because it is not the expected filesystem.

 

If it is empty then you could just stop the rebuild and set a new config but then you should rebuild parity so you might as well let the rebuild continue and then we can see what you end up with.

 

Link to comment

Thanks again! I believe that instructions should point out that all docker apps must be stopped before rsync copy. Otherwise the verification (in step 9) will fail.

 

So if I understand you correctly, should I let the rebuild process finish and THEN format the new drive (that I bought to convert all array to xfs, I didn't replace any disk) to xfs and then rerun the rsync commands?

Link to comment

Thanks again! I believe that instructions should point out that all docker apps must be stopped before rsync copy. Otherwise the verification (in step 9) will fail.

 

So if I understand you correctly, should I let the rebuild process finish and THEN format the new drive (that I bought to convert all array to xfs, I didn't replace any disk) to xfs and then rerun the rsync commands?

Maybe I missed it but I don't remember new drives being part of the discussion in your previous posts. Most of this thread was geared towards letting people move things around on existing drives. Ideally you would have done the rebuild to a new disk before starting any of this.

 

If you let the rebuild finish then format the new drive, it will be still be in sync with parity after the format.

 

Many people seem unaware of what format actually means. Format means "write an empty filesystem (of some specific filesystem type) to this disk". That is what it has always meant on every computer you have ever used. unRAID handles this write just like any other, by updating parity so it stays in sync.

 

Personally, when I did this conversion many months ago I didn't bother with the verification but if you want to do it again that's fine. Appdata is probably less important than almost any other data you have though. If it's broken you can regenerate it. Not so with your important family photos and movies, for example.

Link to comment

Thanks again! I believe that instructions should point out that all docker apps must be stopped before rsync copy. Otherwise the verification (in step 9) will fail.

 

I've modified the XFS conversion post (found here), and added warnings to it, marked in blue.  I'm afraid it never occurred to me that anyone would be modifying the original drive during the conversion, but it's probably a relatively common occurrence.  I often do things differently than most people, and need feedback like yours to correct me.  Please review what I've written, and tell me if it could be improved, and any other suggestions you may have.

Link to comment

Thanks again! I believe that instructions should point out that all docker apps must be stopped before rsync copy. Otherwise the verification (in step 9) will fail.

 

I've modified the XFS conversion post (found here), and added warnings to it, marked in blue.  I'm afraid it never occurred to me that anyone would be modifying the original drive during the conversion, but it's probably a relatively common occurrence.  I often do things differently than most people, and need feedback like yours to correct me.  Please review what I've written, and tell me if it could be improved, and any other suggestions you may have.

Thank you.

I have only two Docker apps installed: Transmission and PlexMediaServer. No VMs, Plugins, or Mover...

I repeated all the steps (until 9) in your guide, but this time I stopped these apps before step 1!

The copy in step 8 completed successfully, as expected.

But the verification in step 9 took much less time to complete than step 8 (which is strange) and came up with many messages such as this:

skipping non-regular file "appdata/PlexMediaServer/Library/....."

 

However, I compared the contents of the two disks and they are identical!

To make the comparison, I used the commands dir \\tower\disk10 /s and dir \\tower\disk11 /s from my windows PC, as suggested.

Fortunately, the output of the dir commands shows the same number of files, folders and bytes for disks 10 and 11, so I guess everything is OK.

 

In unRAID's Main page I see that in XFS disk11 (4TB) uses 1.69TB (2.31TB free) and in ReiserFS disk10 (also 4TB) uses 1.67TB (2.33TB free).

That means that for the same amount of data, XFS needs slightly more space than ReiserFS. Is this normal?

 

I am now moving on to the next steps with a little variation. I see that in your guide you always keep disk11 empty, in order to receive the data from the other drives.

I find that all the assign, unassign, missing drives, unmountable, etc a little confusing for me to follow...

So what I want to do is follow bjp999's guide on the first page, but instead of creating a "t" directory on the destination disk (as he suggests) I will exclude it in Global Share Settings (as you suggest), in order to avoid duplicates in shares...

In summary, now that disk11 (XFS) has all the contents of disk10 (ReiserFS), I will exclude disk10 in Global Share Settings and then format it in XFS.

Next I will copy the contents of disk9 to disk10. Then exclude disk9, format it, copy from disk8 and so on...

I hope I understand it correctly...

Link to comment
  • 3 weeks later...

I've been following the steps outlined in RobJ's post and completed all steps 1 to 17 for my first drive (all went well).

Now I'm about to do this for my second drive but have some noobish questions..

 

Before I started my array looked like this:

Parity 8TB

Disk1 2TB (reiserfs)

Disk2 2TB (reiserfs)

Disk3 2TB (reiserfs)

Disk4 8TB (empty)

 

I started with Disk1 so I now have:

Disk1 8TB (xfs)

Disk2 2TB (reiserfs)

Disk3 2TB (reiserfs)

Disk4 2TB (xfs, empty)

 

I can fit all the data from the three 2TB drives onto the 8TB so now I want to copy Disk2 to Disk1 - do I need to change anything (excluded disks etc.) or do anything different ?

I believe I have to use rsync -avPX /mnt/disk2/ /mnt/disk1/ (is this correct) ?

 

Link to comment

Before I started my array looked like this:

Parity 8TB

Disk1 2TB (reiserfs)

Disk2 2TB (reiserfs)

Disk3 2TB (reiserfs)

Disk4 8TB (empty)

 

I started with Disk1 so I now have:

Disk1 8TB (xfs)

Disk2 2TB (reiserfs)

Disk3 2TB (reiserfs)

Disk4 2TB (xfs, empty)

 

I can fit all the data from the three 2TB drives onto the 8TB so now I want to copy Disk2 to Disk1 - do I need to change anything (excluded disks etc.) or do anything different ?

I believe I have to use rsync -avPX /mnt/disk2/ /mnt/disk1/ (is this correct) ?

I'm not absolutely positive, but it *looks* correct!  You may have to adjust your share inclusions (e.g. moving Photos from Disk 2 to Disk 1).

Link to comment

Apologies if this is a double post, but I lost two drives without parity and decided to rebuild with XFS.  When I had all my data on XFS drives I tried to start the array and it asked me to reformat my XFS drives (and lose their data)... I said NO, but now each of my drives is unmountable with no XFS or any FS. 

 

Is there a way for me to restore XFS and more importantly all of my data?

 

See attached.  All of the blank FS disks are XFS disks filled with data. 

 

LOST_FS.JPG.2052db1e7457df9fbd928f9ddb1832d3.JPG

Link to comment

Apologies if this is a double post, but I lost two drives without parity and decided to rebuild with XFS.  When I had all my data on XFS drives I tried to start the array and it asked me to reformat my XFS drives (and lose their data)... I said NO, but now each of my drives is unmountable with no XFS or any FS. 

 

Is there a way for me to restore XFS and more importantly all of my data?

 

See attached.  All of the blank FS disks are XFS disks filled with data.

 

Usually that happens when you do a New Config.  The file system type is set to Auto, which does not appear to be working correctly, not identifying the correct file system on each drive.  With the array stopped, just click on each drive and change the file system to XFS, and they should all be good again!  Make sure they are ALL correct, before restarting the array.  I don't trust the Auto setting any more.

Link to comment

Usually that happens when you do a New Config.  The file system type is set to Auto, which does not appear to be working correctly, not identifying the correct file system on each drive.  With the array stopped, just click on each drive and change the file system to XFS, and they should all be good again!  Make sure they are ALL correct, before restarting the array.  I don't trust the Auto setting any more.

 

I just tried it and I got the same result.  I set each to XFS and I checked each one.  See attached.

LOST_FS_2.JPG.60fc7626ad56c6a1288452c17f583a23.JPG

Link to comment

FWIW, I think I know what happened.

 

Because my most critical data is saved on CrashPlan, I decided to just start over.

 

All went well until I tried to add one more drive to the array.  This was an RFS drive that I copied over to Disk 1 and then formatted to XFS using UNASSIGNED DEVICES.  Unraid DID NOT accept this XFS drive, and said it needed to be reformatted. 

 

This is exactly how I lost 28TB of DATA.  I used UNASSIGNED DEVICES to format XFS, then I copied data to these drives.  UNRAID did not recognize their XFS, and required a FORMAT.  I tried overnight to recover XFS before I quit and started over.

 

I am now out of drives, but I am sure this situation is repeatable.

 

1. Format a drive XFS with UNASSIGNED DEVICES

2. Try to add it to a new array configuration

 

You will not be able to do so without reformatting the drive in the array.

 

Which is totally fine, unless you added data to the drive before trying to add it to the array like I did. 

 

I hope this helps someone avoid the loss of data that I just experienced.

 

Attached diagnostics.

grimlock-diagnostics-20160918-1519.zip

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.