Re: Format XFS on replacement drive / Convert from RFS to XFS (discussion only)


Recommended Posts

1 minute ago, tunetyme said:

Nope I got those steps right.  It was the warning that the parity drive would be erased. That is what started this discussion. See page 24 5th post down 

 

Silly me, I thought procedure was posted as a reply to my question about where you got the idea that parity was valid after completely removing a disk. The last bunch of posts about parity seem to be based on that misconception, rather than on the wording of step 16 (or the confusing web UI text)...

 

So where did you get the idea that parity was still valid if a drive was removed?

 

Well, I guess it doesn't really matter as long as it wasn't from a resource relating to this thread. Happy conversion and may your files live long and prosper! :)

Link to comment

If you follow through the process, after you swap the new drive with all the files copied on it then you format the old RFS drive as XFS (formatting would destroy parity) or you could put another drive in that is already formated XFS (Like a larger drive) and repeat the process. Read the last several steps.  

Edited by tunetyme
clarify
Link to comment
31 minutes ago, tunetyme said:

If you follow through the process, after you swap the new drive with all the files copied on it then you format the old RFS drive as XFS (formatting would destroy parity) or you could put another drive in that is already formated XFS (Like a larger drive) and repeat the process. Read the last several steps.  

No, you read the last several steps! :D

 

Formatting does not destroy parity as it is done while the disk is assigned to the array and the array is started. Therefore the parity is updated to reflect the formatting. After all, formatting a disk means writing an empty filesystem to it. It is just as any other set of writes, parity does not care. It only cares about the raw bits.

 

And no, you absolutely cannot put another drive in instead here. You have to use exactly the same set of drives to maintain parity.

  • Upvote 1
Link to comment
38 minutes ago, tunetyme said:

If you follow through the process, after you swap the new drive with all the files copied on it then you format the old RFS drive as XFS (formatting would destroy parity) or you could put another drive in that is already formated XFS (Like a larger drive) and repeat the process. Read the last several steps.  

The "new drive" is already part of the array, so anything written to it from that point forward updates parity, and as already mentioned, everything is a write at the bit level, whether it is creating a new file, deleting a file, or formatting the whole disk.

 

Where exactly does it say you can put in a different disk?

Link to comment
7 minutes ago, trurl said:

The "new drive" is already part of the array, so anything written to it from that point forward updates parity, and as already mentioned, everything is a write at the bit level, whether it is creating a new file, deleting a file, or formatting the whole disk.

 

Where exactly does it say you can put in a different disk?

 

I think it is implied when you format the disk. In effect it is all zeros and a new previous prepared drive is also all zeros. ???? And the difference is?

 

Back to one of my original questions is you "can you upgrade drives (plural) and convert to xfs."  I have said many times that, if I can't maintain parity, it is no big deal. I copy an old drive to a new drive. Repeat until the new drives are install and each one has the correct files on each disk and the are in their designated slot. Format the old drives to xfs and repeat the process until all the rfs drives have been replaced . Build parity. Done. Copying should take a lot less time since I am not updating parity.  As I have asked before is maintaining parity "mission critical" for all users.  If you don't have a backup, maybe. Storage is cheap and Icy Dock 4 in 3 are inexpensive and makes life real easy. What else are you going to do with the drives after you do your upgrades that still have life in them? Buy a few more on eBay.

 

Now perhaps you understand why I thought the process was convoluted.  What you are saying is you can only introduce one new drive. Copy format repeat reusing the drive rfs drive and going through your whole array.

 

My method would be preclear all the new drives and format them xfs. Let's use 2 new 4TB drives. Unassign parity drive. Copy one of the old 2TB drives to a new 4TB drive, repeat for second drive. Swap places for the drives and format or preclear and format the drives if you want make sure they are still reliable and repeat the process of copying until you have worked through your whole array.  Build parity. Done.

 

I guess that's about 4 cents worth.

Link to comment
17 minutes ago, tunetyme said:

I think it is implied when you format the disk. In effect it is all zeros and a new previous prepared drive is also all zeros. ???? And the difference is?

If the formatted disk was precleared just before formatting, then most of it would be zeros, but some of it would be the bits that represent the empty filesystem, and those would not be zero.

 

If the formatted disk was not precleared, then some of it represents the empty filesystem, but the rest of the disk is unchanged from what it was before the format. You can assume much of it is not zeros.

 

A formatted disk is not a clear disk. Format literally means "write an empty filesystem". And all filesystems are not the same, so how an empty filesystem is actually represented in the bits is different for different filesystems. And none of the filesystems represent an empty filesystem as all zeros.

 

Formatting a disk when it is already part of the array maintains parity. The contents of the disk were already part of parity before the format, and formatting updates parity, because formatting is simply writing a few bits that represent an empty filesystem.

 

Adding a clear disk to a new slot in the array maintains parity, because a clear disk is all zeros and has no effect on parity.

 

Replacing a formatted disk with any other disk, whether clear or not, requires a rebuild of the replacement, or a rebuild of parity. Because the bits on the original and replacement disks are not the same.

 

As far as parity is concerned, there are no files, and there are no filesystems. It is all just bits. A disk with an empty filesystem is a bunch of bits, and at least some of them are not zero, or there wouldn't be a filesystem there.

Link to comment

If you are not concerned about maintaining parity, then of course you don't have to.

 

The only thing you have to do to convert a disk from RFS to XFS is reformat it to XFS. The whole point of moving things around is so you don't lose any files when you reformat.

 

There is absolutely no requirement to follow any instructions from anybody else. unRAID isn't going to care, since at any time a New Config will just tell it you are starting over with whatever is already on the disks that you assign at that time.

 

I certainly didn't follow the wiki for this (that wiki didn't exist at the time). I just did it based on my understanding and my specific situation. I didn't need to add any drives since I already had enough empty space on my drives to allow me to move everything off them one at a time so I could reformat. And I maintained parity throughout.

 

Other methods are certainly possible. For example, if you have good backups of everything then just start with a bunch of freshly formatted XFS disks (don't even need to be clear), copy everything to them, and build parity.

 

But the backups for the really important irreplaceable files must be really trustworthy. I have multiple backups of those.

Link to comment

I did preclear the disk because I could not get the format button to appear with the rfs drive after it was copied.  As I have said I don't mind preclearing 6 year old drives to make sure they are healthy and still have reliable life in them.

 

I know that formating does not write zeros to the drive but perhaps it should.

Link to comment
40 minutes ago, tunetyme said:

I know that formating does not write zeros to the drive but perhaps it should.

That would be different than format normally works for all other operating systems, and for all other filesystems. And it would take a lot of needless time to write all those zeros when all formatting is trying to accomplish is a new filesytem. We already have another, separate process (actually more than one) for writing zeros to an entire disk.

Link to comment

I have a lot to learn.  It is apparent that I need to learn all the different ways to skin this cat.

 

RE: disk3 share

 

I opened mc and there it is under User 0 is disk3 share.  To delete, it I will need to delete the entire disk and restore from a backup.

 

Is there and easier way?

Link to comment
23 minutes ago, tunetyme said:

Is there and easier way?

Open the GUI and click on the shares tab. In the user share section there should be a disk3 share. Click on the corresponding "compute" link on the size column. It will take a few seconds, and should return a list of physical disks that the share resides on. Once you get that information, open mc and navigate the left panel to the /mnt/diskX that corresponds to the physical disk shown by the compute command so that the contents of the disk3 folder shows, and navigate the right panel to the same spot, but only to where only the disk3 folder itself is shown. Using the tab key, the arrow keys and the insert key, highlight the contents of the disk3 folder in the left panel, and press F6 to move the highlighted folders to /mnt/diskX in the right. After this is complete, you should be able to go back to the web GUI, click on the disk3 share in the user share list, and delete it on the share properties page.

  • Upvote 1
Link to comment
1 hour ago, tunetyme said:

I opened mc and there it is under User 0 is disk3 share.

jonathanm already gave good advice on how to proceed. Just wanted to comment on this bit.

 

/mnt/user is the user shares. /mnt/user0 is the user shares excluding any files that are still on cache. If you see it in /mnt/user0, then it is also in /mnt/user. /mnt/user0 is really more of a "system" folder. The mover script works by moving files between /mnt/cache and /mnt/user0.

 

But you don't want to work with the user shares for this, you want to work with the disks as jonathanm instructed.

 

11 hours ago, trurl said:

Please be careful with upper/lower case and spaces when discussing this since they are critical.

 

If you are not precise with this, it can cause confusion for yourself and those trying to help, and lead to instructions that might not work exactly right, or at all. As you know, computers take things very literally, and spaces and upper/lower case are significant in folder/filenames in linux.

Link to comment
8 hours ago, jonathanm said:

navigate the left panel to the /mnt/diskX that corresponds to the physical disk shown by the compute command so that the contents of the disk3 folder shows, and navigate the right panel to the same spot, but only to where only the disk3 folder itself is shown. Using the tab key, the arrow keys and the insert key, highlight the contents of the disk3 folder in the left panel, and press F6 to move the highlighted folders to /mnt/diskX in the right.

 

I want to be sure that I get this one right the first time.

 

S is my video parent directory on all video disks

 

In MC

Left Panel  (physical disk)

/mnt/disk3/disk3

 

highlight S

 

Right Panel

/mnt/user0/disk3/S

 

Move from the rigth panel to the left panel??

 

I need to clarify this, as I read your directions it appears that I go to the physical disk on both left and right panels and move it to itself /mnt/disk3/disk3/S to /mnt/disk3/disk3.

I would have thought that I would move files from /mnt/user0/disk3/S to /mnt/disk3/disk3 which I would need to swap panels to move file from left to right.  

Link to comment

That does not look right!   I am reasonably sure it will lead to data loss.  You do not want to mix disk drives and user shares in a copy/move as a rule as there is too much chance of an error.

 

i would think that you want something like:

left:   /mnt/disk3/disk3/S

right: /mnt/disk3/S

and move all the contents of the disk3/S folder on the left to the top level S folder on the same drive.   In fact you could leave off the 'S' part on both sides I would think and get the same results.

Link to comment
3 hours ago, tunetyme said:

 

Error

 

can't do a /mnt/disk3/disk3

only able to do /mnt/disk3 on both sides

If that is the case then the spurious disk3 folder that is showing up as a User Share is not actually on the physical disk3.

 

have you tried what I suggested earlier of:

- go to the Shares tab in the unRAID GUI

- find the disk3 share and click on the folder icon at the right

normally that takes you to a page that shows you the disks where the files/folders for that share are actually located. 

Link to comment

Regrettably, I rebooted the server.

tree /mnt -L 2

This is the recommendation from "Fix Common Problems"

While there *may be* (doubtful) valid use cases for having a share named identically to a disk share, this is only going to cause some confusion as if disk shares are enabled, then you will have duplicated share names, and possibly if disk shares are not enabled, then you might not be able to gain access to the share. This is usually caused by moving the contents of one disk to another (XFS Conversion?) and an improperly placed slash. The solution is to move the contents of the user share named disk3 to be placed within a validly named share. Ask for assistance on the forums for guidance on doing this.

 

Under user0

/..

/s

/p

/disk3 

 

No files or directories under disk3

 

/mnt/disk3

/s

/disk3

 

No files or directories under disk3

Link to comment
23 minutes ago, tunetyme said:

disk3 shows up under

 

disk3 -S -disk3

user0 -disk3

user -disk3

 

the "-" is to signify a subdirectory

So is this the output of the tree command asked for?

 

It would be much better to see the complete output as a screenshot exactly as it appears rather than some sort of transcription.

 

How are you getting to the command line? Keyboard/monitor or telnet/ssh? Should be easy to get a screenshot from telnet/ssh client.

 

For example, this is what that command gives me:

tree.thumb.png.e71723d46110356463bdcc16f2369a8e.png

Link to comment
38 minutes ago, tunetyme said:

That is the display I saw but not clear how to copy and paste to the forum

 

5 hours ago, trurl said:

How are you getting to the command line? Keyboard/monitor or telnet/ssh?

Answer this and if telnet/ssh, what OS are you working from? (Windows, iOS, etc.)

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.