So I have a basic question - is it possible to format a disk with unRaid's version of RFS (3.6?) such that data sectors begin at sector 64 vs sector 63?

So I have a basic question - is it possible to format a disk with unRaid's version of RFS (3.6?) such that data sectors begin at sector 64 vs sector 63?

 

Yes, you can, using fdisk, but if you did it, unRAID would see it as an invalid disk, and reformat it.

Yes, you can, using fdisk, but if you did it, unRAID would see it as an invalid disk, and reformat it.

 

Interesting ...

 

I am assuming that you are saying that unRAID would see the disk as unformatted and enable to format button.  It wouldn't actually format the disk unless you pressed the format button.  (I'm pretty sure of this as unRAID never automatically formats a disk).

 

Ok, so let's say I precleared a disk, added it to my array, fdisked the disk (to start data sectors at 64 instead of 63), and then formatted the disk myself.  Could I then mount the disk myself, set up a custom samba share, and copy data to the disk?  Would unRAID maintain parity protection even though it is totally ignorant of the contents?

 

(I'm not recommending anyone do this - just trying to understand the inner workings better).

Yes, you can, using fdisk, but if you did it, unRAID would see it as an invalid disk, and reformat it.

 

Interesting ...

 

I am assuming that you are saying that unRAID would see the disk as unformatted and enable to format button.  It wouldn't actually format the disk unless you pressed the format button.  (I'm pretty sure of this as unRAID never automatically formats a disk).

 

Ok, so let's say I precleared a disk, added it to my array, fdisked the disk (to start data sectors at 64 instead of 63), and then formatted the disk myself.  Could I then mount the disk myself, set up a custom samba share, and copy data to the disk?  Would unRAID maintain parity protection even though it is totally ignorant of the contents?

 

(I'm not recommending anyone do this - just trying to understand the inner workings better).

No, because if the partition does not start on sector 63 and extend the full size of the disk and be able to be mounted as a reiserfs it will be considered as a foreign disk and unRAID will clear it upon starting, then partition it as I described and give you the option of formatting it.

No, because if the partition does not start on sector 63 and extend the full size of the disk and be able to be mounted as a reiserfs it will be considered as a foreign disk and unRAID will clear it upon starting, then partition it as I described and give you the option of formatting it.

 

Thanks Joe.  This is new info for me.  Hope I can ask a couple follow up questions so I fully understand.

 

In my hypothetical ...

 

1 - The disk was precleared

2 - The disk was added to the array

3 - The disk was re-fdisked (to change the partition)

4 - The disk was formatted with RFS

5 - The disk was manually mounted

 

At what point in the process would unRAID "clear it upon starting", or would unRAID allow this to happen but then the next time the array is started, "clear it".

 

Also can you explain what you mean by "clearing it" in this context.  Do you mean repartition the disk?  or zero the disk?  I am surprised it would take either a measure on a disk that is already in the array.

 

Thanks!

or would unRAID allow this to happen but then the next time the array is started, "clear it".

The next time you start the array unRAId would think of it as not part of the array, as it would not match what it expects.

Also can you explain what you mean by "clearing it" in this context.

Completely write zeros to the entire drive.

  Do you mean repartition the disk?  or zero the disk?  I am surprised it would take either a measure on a disk that is already in the array.

It would not start the array, it would wait for you to confirm you really wanted to start the array.  When you press "Start" it will write zeros to the entire drive.  Only after that is complete will it bring the array online.  At that point in time the new drive will show as un-formatted.  (I think it is already partitioned at that point, but it might wait till you press "Format" to do that step.)

 

You can experiment if you wish... but I know the code in emhttp looking for valid unRAID disks explicitly looks for the partition to start on the second cylinder leaving the first unused.

Humor me with a couple more questions along this line and then I will move on ...

 

What if you never restarted the array.  You just added the precleared disk, re-fdisked it, formatted it, and mounted it. 

 

First, would it let you?  Second, any reason that unRAID would not maintain parity on the disk?

Humor me with a couple more questions along this line and then I will move on ...

 

What if you never restarted the array.  You just added the precleared disk, re-fdisked it, formatted it, and mounted it. 

 

First, would it let you?  Second, any reason that unRAID would not maintain parity on the disk?

You might get away with it, but you'd not be able to deal with a disk failure... For that you need to stop the array.

 

Joe L.

So the point being - unRAID could maintain parity on basically any disk, because the md driver is operating at a low level, and not dependent on the format of the disk.  The trick is getting past some edits that are in place to ensure that drives are following well-defined partitioning rules.

 

Ok ... moving forward ...

 

What if Tom made a change to unRAID and looked for a different signature - a partition starting at sector 64 extending to the full size of the disk.  And altered the format option to fdisk and format the disk to produce that result.

 

Now those of us with existing arrays would be screwed, but would this change allow someone creating a new array to do so with either old (512 byte) sector disks or new (4k) sector disks without alignment issues (assuming all WD 7-8 jumpers are removed)?

I think part of that path was discussed... many options and complex situations need to be accounted for. IIRC, the gist was unRAID needs to allow for sector 63 and sector 64 partition tables in the short-term. In the long term, perhaps GPT and any filesystem is the way to go.

 

One of the more complex situation was when you had a sector 63 partition drive fail and is replaced with a drive that prefers sector 64 partitions. What should the recovery process do? It's further complicated because there doesn't seem to be any means of detecting the WD Advanced Format drives have the jumper in place or not.

 

 

I think part of that path was discussed... many options and complex situations need to be accounted for. IIRC, the gist was unRAID needs to allow for sector 63 and sector 64 partition tables in the short-term. In the long term, perhaps GPT and any filesystem is the way to go.

 

One of the more complex situation was when you had a sector 63 partition drive fail and is replaced with a drive that prefers sector 64 partitions. What should the recovery process do? It's further complicated because there doesn't seem to be any means of detecting the WD Advanced Format drives have the jumper in place or not.

 

You're jumping ahead ... but that's ok.  I do NOT believe that we can allow a mixed sector 63 and 64 alignment array.  It has to be either one or the other.  Otherwise rebuilding a disk gets too complicated.  You wind up having to update parity in order to do it.  And then if you need to restart you're SOL.  No, either we stick with 63 or move to 64 - not both.  (I could see Tom allowing a hybrid model to allow for a transition from one to the other, but I could also see some complicated support issues arrising and he might decide it is better to just make a clean break.)

 

The issue of whether the jumper is installed or not is not really a problem.  There is a sure fired way to know.  Ask the human .  Seriously, that is no different than now.  How many people have not installed the jumper when they should have.  If Tom moves to 64 sector alignment, some will install the jumper when they should not.  In the end, they'll just have to get it right or live with poor performance.  I believe that fewer people will put the jumper on in error than leave the jumper off in error.

 

So if 64 sector alignment is better than a sector 63 sector alignment, the next question is ...

 

How would the average joe user get their array migrated from the 63 alignment to the 64 alignment?

 

Several brute force methods quickly comes to mind - and we can discuss them.  These would be time consuming and require a new disk or two to jumpstart the process.

 

But is there a more elegant approach that would twiddle a few sectors on the disk and magically accomplish the transformation?

 

(You also mention the GPT replacement of the MBR.  I think we should not mix that discussion with this one yet.  Let's get to the logical conclusion here and then discuss GPT in detail).

It's been a long time since I had to work with low level partition tables, partitions, and filesystems so here's what's probably a rally silly question ... Is there no slack in there to allow for it to start at sector 64 without messing up the reiser filesystem?

So IF Tom made this switch to 64 sector offset instead of 63, and we had to migrate our arrays, a few approaches come to mind:

 

1 - Magic bullet.  Some magical program that manipulates the partition and magically reorganizes it in a short period of time.  I am not sure how technically feasible this is, but I would say "unlikely" that this will be an option.  Not sure I would want to be the guineey pig!

 

2 - Brute force - one server.  Run full parity check on array.  Carefully check all disks for signs of problems before proceeding, as you'll be unprotected for the next series of actions.  Load "new version" of unRAID that supports 64 sector offset, using old parity and at least 1 new / empty disk (which you've precleared using a new version of the preclear program).  One by one, manually mount the old array disks (unmenu will do this) and copy (preferrably with crc checking) their contents to the newly created array.  Once the new array is full, you should run a parity check on the new array to ensure that it is functioning.  (I woud also do some spot checking).  Once confident that the data is copied over, you will need to shut down, remove jumpers (if any) from the emptied old array disks, run preclear (new version) on them (skipping the pre and post reads, since the disks have already been proven), and add them to the new array.  Repeat mounting, copying, verifying, preclearing, and adding until all disks are migrated.

 

3 - Dual servers.  The process is similar to #2 above, but it allows you to maintain parity protection on the old array a bit longer.  If your new array has enough capacity to store all of your data, it would be the safest approach, but this is unlikely for larger arrays.

 

So options 2 and 3 would take a while ...

 

1 - Preclearing new disks - 1 day

2 - Parity checking old array - in parallel with 1

3 - Copying data (estimate 1.5 days for 2T data - with crc checking)

4 - Parity check (new array) - 1/2 day

5 - Preclearing old disk (skipping pre/post reads) - 1/2 day (can do multiple in parallel - overlapping with parity check adds risk)

6 - Adding to new array - quick

 

So how long in total would this take on a full 20TB array (assuming you have one new 2T drive to work with)

 

If my math is right, it is 26 days!

 

With 2 2T drives to work with .... 21 days.

 

Worth it?  What do you think?

Here's an idea....

 

Let's separate the "md" device issues from the emhttp (and array management ones)

 

Currently unRAID uses the first partition of a given drive when it defines the "md" devices.   This means the first 63 sectors are not parity protected at all.  (and there is no need to, since the first sector, the MBR is specifically defined, and the others from sector 1 through 62 are all zeros.)

 

If instead unRAID were to use the entire disk instead of the first partition it would result in it not caring where the specific partition (or partitions) exist, or start, or end.

 

That is not a huge change and can probably be implemented at any time as a first step in the conversion process.  Once that is done, the issue of re-building a disk from one starting on sector 63 to one starting on sector 64 will just occur.  There might be some way to subsequently re-locate the resulting file-system, but that is a different issue than re-constructing an image of your data.

 

There are side benefits to this... and complications, especially when re-constructing onto a larger disk since the disk geometry is stored in the MBR.  Some disk utilities might get confused... or a larger disk might all of a sudden look like the smaller one according to its MBR.  I'm sure this can be worked around, but it is a complication when you no longer can use any of the information in the MBR to determine geometry.

 

On a non 4k-sector disk re-construction onto a 4k aligned replacement, I'd be happy with all of my movies, even if random read access is not optimal on a reconstructed disk.

 

After the RAID is on the entire disk we can deal much more easily with alternate file systems, alternate partitioning types, and migration schemes.  Those all involve emhttp.

 

Edit: According to this page: http://manpages.ubuntu.com/manpages/lucid/man8/gdisk.8.html

gdisk can convert an MBR to a GUID partition record.  The conversion is probably not impossible.

 

Joe L.

Here's an idea....

 

Let's separate the "md" device issues from the emhttp (and array management ones)

 

Currently unRAID uses the first partition of a given drive when it defines the "md" devices.   This means the first 63 sectors are not parity protected at all.  (and there is no need to, since the first sector, the MBR is specifically defined, and the others from sector 1 through 62 are all zeros.)

 

If instead unRAID were to use the entire disk instead of the first partition it would result in it not caring where the specific partition (or partitions) exist, or start, or end.

 

I am pretty ignorant of recent developments with MBR-less drives, GPT.  And I was never an expert in disk structure anyways.  But am trying to follow your suggestion but struggling ...

 

Are you suggesting no MBR and starting the RFS partition at sector 0?  Or are you suggesting that unRAID parity protect the entire disk starting at sector 0?  I'm assuming the former interpretation, sorry if this was not what you were suggestiong.

 

First impression is that I'm scared that eliminating the MBR could have consequences - some of which we can predict and some we may not be able to.   Could we still load an unRAID disk into our Windows box (or other Linux boxes) and read the data?  Would we need BOIS updates making old motherboars obsolete for unRAID?

 

That is not a huge change and can probably be implemented at any time as a first step in the conversion process.  Once that is done, the issue of re-building a disk from one starting on sector 63 to one starting on sector 64 will just occur.  There might be some way to subsequently re-locate the resulting file-system, but that is a different issue than re-constructing an image of your data.

 

There are side benefits to this... and complications, especially when re-constructing onto a larger disk since the disk geometry is stored in the MBR.  Some disk utilities might get confused... or a larger disk might all of a sudden look like the smaller one according to its MBR.  I'm sure this can be worked around, but it is a complication when you no longer can use any of the information in the MBR to determine geometry.

 

I am not understanding the benefits.  I see the complications and risks.  LOE effort aside, I think I just need some help understanding why this is a good thing and reassurance that we're not eliminating a lot of current unRAID benefits.  

 

On a non 4k-sector disk re-construction onto a 4k aligned replacement, I'd be happy with all of my movies, even if random read access is not optimal on a reconstructed disk.

 

If the alternatives are lose your data or restore it (even if performance accessing the restored data is poor), I'm all for restoring it!  But is that the design point we're looking for?  Wouldn't it be better to be able to upsize or replace a disk and not be left with a mess on your hands to get it into normal performance structure?  I'd rather take my lumps all at once than leave myself in this situation.

 

After the RAID is on the entire disk we can deal much more easily with alternate file systems, alternate partitioning types, and migration schemes.  Those all involve emhttp.

 

emhttp or not - they are changes that Tom would have to make.  And I am still struggling with what specific benefit this change provides.

 

Edit: According to this page: http://manpages.ubuntu.com/manpages/lucid/man8/gdisk.8.html

gdisk can convert an MBR to a GUID partition record.  The conversion is probably not impossible.

 

Joe L.

 

Now a GPT does not eliminate a MBR - which is what you were suggesting above (forgive me, I'm trying to put the pieces together).  There are lots of discussions about using GPT with a "protective MBR".  I don't understand all of it but seems that GPT is the future with a lot of thought of backward compatibility.  If we could move from MBR to GPT (with appropriate "protective MBR" in place to provide compatibilty), and get to a 4 byte alignment (be it 64, 60, 56, ...16, 8, 4 - doesn't matter), that would seem to be a good thing both architecturally and practically.  But this seems different from discussion above.

 

With the gdisk ability to convert from MBR to GPT, if Tom implemented the sector alignment with MBR intact, would the move to GPT be as simple as running gdisk?

I am not suggesting the elimination of the MBR.  I am suggesting that instead of only parity protecting the first (and only) partition on the disk that parity protect every sector on the disk.

 

That way, as far as parity and all its routines care, the contents and MBR /GUID structures do not matter.  That change could be made today and nobody would need to change any partitioning.  (It may require some changes to emhttp, as it force writes the MBR currently along with the MBR style partition table.  That can stay, for now... but when it changes it (emhttp) will be able to start the partition anywhere. (theoretically) using any partitioning scheme.

 

If a "standard" partitioning scheme is used, a window's PC, or at least a newer windows PC should be able to deal with the partition.

Linux has never needed a partition... You can "dd" a full partition holding a file-system and put it in a file as an example.  I do that frequently when backing up a window's PC when no other backup tools are available.

 

So... by using a GUID we can still get to the data by mounting an individual disk on either windows or Lniux.

 

As far as migrating the file-system...  Tricky to do in place.  possible, but tricky.  It involves moving the entire partition upward.  That is complicated since the partition currently uses all the space, so therefore, you must first shrink the existing file-system by one sector (not too hard as long as it is not full) and then copy every sector up one.  (copy the second to last sector to the last sector, then the third from last tot he second to last, etc...) It will be a VERY LONG process.  Probably longer than re-constructing a disk.

 

When you finish moving the data that represents a partition, the partition table itself must point to the new start.  That is probably easy compared to the rest.

 

With the gdisk ability to convert from MBR to GPT, if Tom implemented the sector alignment with MBR intact, would the move to GPT be as simple as running gdisk?

No,  all of the routines in emhttp dealing with upgrading disks, determining native unRAID disks vs. pre-cleared disks vs. foreign disks need to be updated.  It is a HUGE job.  Everything is hard coded to expect an MBR, including the preclear_disk.sh utility I wrote. It too would need the corresponding changes to deal with a GUID partition table.

Would a side effect of this also be the ability to not be forced to use reiser allowing more portable file systems like ext3/4 to be used?

Would a side effect of this also be the ability to not be forced to use reiser allowing more portable file systems like ext3/4 to be used?

That is a matter of several items, none of which have anything to to with the partitioning scheme or where it lives.

 

To allow a different file system type, and continue to allow a smaller disk to be replaced with a larger one you need a file-system type that can be re-sized (made bigger) to fit the larger file-system on the replacement disk.  That would eliminate some file system types.

 

You would need a way to identify and properly mount an unRAID disk while recognizing foreign disks.  This is much more difficult than just dealing with reiserfs file systems.

 

Neither of those depend on the MBR/GUID scheme, or where the partition starts, or even if it is a single partition on a disk.  Those just all complicate getting the partitions mounted.

so no then

Most of this is way over my head, though Joe L's suggestion of protecting every sector on the disk conceptually makes sense to me.

 

Anyway, I think I understand why we can't (or at least shouldn't) have a mixed 63 and 64 solution.  So it seems to me that the simplest solution (not necessarily the crowd-pleaser, but the simplest) would be to have two versions of unRAID:

 

unRAID63 that forces you to use non-advanced format disks or advanced format disks in compatibility mode (like the WD EARS w/ a jumper)

 

unRAID64 that forces you to use advanced format disks.  This means that the majority of our current disks will not work with this version.  However, it would be very useful for someone building a new server after January 2011 when basically all available hard drives will be advanced format.

 

Now this solution has the potential to screw those of us who currently have a lot of non-advanced format disks.  Basically, we will be able to keep using our servers with unRAID63, but wouldn't be able to add new disks (unless we searched specifically for old non-advanced format disks).  At that point I would probably try to build a new unRAID64 server and sell off all my old disks.  Or maybe keep the unRAID63 server as a backup to the unRAID64 server.  Still, I have the luxury (in both time and money) to do this.  Many people don't.  So I'm a bit worried as to what those people will do.

 

Perhaps unRAID64 would also be able to support 3TB drives, whereas unRAID63 couldn't.  That is a separate issue, of course.

From the standpoint of a software developer, I can tell you that is a disaster waiting to happen.

 

First rule of development is that end users are brain-dead morons.  Any choice left to a user to decide intelligently is doomed to failure.  Let them have all the choices they want with respect to bell and whistles, but when it comes to a choice where if you chose wrong you lose data or you get crappy performance, giving that choice to users is like giving a loaded gun to gorilla.

 

First, I believe it would not be hard to allow both 63 and 64 partitions in the same system... unRAID can check and handle both.  But as has been pointed out, you have a disaster waiting when you have to rebuild a failed 63 drive with a drive that should be a 64 drive and vice versa (replacing a 64 drive with an EARS with the jumper).  The second issue is which to use (63 or 64) for new drives added to the system.  64 can be used for EVERY new drive... EXCEPT a WDEARS with the jumper (WARNING:  This is based on the current crop of drives... other manufacturers may also have issues with drives that are not yet on the market... hence why I say wait for them to shake out).  The only option is to give the user a choice.  If you have a choice, you need a way to fix the choice being chosen wrong.

 

What I would do is add a "copy disk" option.  Given a precleared disk of the appropriate size, it could be added to the array, with a different starting sector for the partition than the failed disk, and drive contents copied on a file-by-file basis, while maintaining parity.  Most importantly, this could be done with a failed disk, to facilitate replacing a 63 with a 64.

 

Then you are stuck with good parity, a clean new disk that has the same contents as the failed disk.  To maintain parity, you need to be able to zero out the failed disk and remove it from the array.  (A feature that is in and of itself quite useful).

 

The idea of resizing a 63 to a 64 is also possible.  For example, ParitionMagic will resize and relocate parititions.  You don't have to move every sector though, you just have to have enough free space to move the files AS FILES that are in the way, flush the data, and then move/update the directories, inodes, etc.

You also don't have to start at 64.  You can use 58, 32, etc., which makes the partition larger, not smaller.  Or even go partitionless.  Relocate the FS root, move and update the directories, inodes, metadata, etc.

From the standpoint of a software developer, I can tell you that is a disaster waiting to happen.

Boy, do I ever agree...

First rule of development is that end users are brain-dead morons.

There is some percentage that are not brain-dead... and another percentage that are not morons... but enough of both to have a substantial overlap that meet both criteria.

 Any choice left to a user to decide intelligently is doomed to failure.

Again only a percentage of the users will be doomed to failure

 Let them have all the choices they want with respect to bell and whistles, but when it comes to a choice where if you chose wrong you lose data or you get crappy performance, giving that choice to users is like giving a loaded gun to gorilla.

I usually think of it being equivalent of giving matches and dynamite to a two year old.  A disastrous result is almost guaranteed.

 

First, I believe it would not be hard to allow both 63 and 64 partitions in the same system... unRAID can check and handle both.  But as has been pointed out, you have a disaster waiting when you have to rebuild a failed 63 drive with a drive that should be a 64 drive and vice versa (replacing a 64 drive with an EARS with the jumper).  The second issue is which to use (63 or 64) for new drives added to the system.  64 can be used for EVERY new drive...

There need be NO exceptions.   unRAID works in stripes, not at the individual block level.  An occasional extra rotation of the disk will not hurt you to where you notice performance change.   Look at how many users of disks say "I've been using my disk without jumpers for months... and they work fine" (until they add the jumpers)

What I would do is add a "copy disk" option.  Given a precleared disk of the appropriate size, it could be added to the array, with a different starting sector for the partition than the failed disk, and drive contents copied on a file-by-file basis, while maintaining parity.  Most importantly, this could be done with a failed disk, to facilitate replacing a 63 with a 64.

That might be a good way to handle it.  I'm sure there are others that do not require an additional disk slot though.

The idea of resizing a 63 to a 64 is also possible.  For example, ParitionMagic will resize and relocate partitions.  You don't have to move every sector though, you just have to have enough free space to move the files AS FILES that are in the way, flush the data, and then move/update the directories, inodes, etc.

That might work for some file-systems... not sure about reiserfs, never investigated.  Again... complications exist when the disk is full.. where do you move the last sector on the disk to if you are trying to move everything upward in place...  there is no "next-sector-up" from there.

You also don't have to start at 64.  You can use 58, 32, etc., which makes the partition larger, not smaller.  Or even go partitionless.  Relocate the FS root, move and update the directories, inodes, metadata, etc.

Duh... I should have thought of that... Sometimes it is so obvious... after somebody else says it.

That might be a good way to handle it.  I'm sure there are others that do not require an additional disk slot though.

 

Only another logical slot, not a physical slot.  You could create a new type of logical slot, like the cache drive, for "folding in" a new drive.  If you are folding in a failed drive, just pull the failed drive out to free the physical slot.  And the neat part -- you can do the same to a non-failed drive if you don't have another physical slot.  Power down, remove it so unRAID sees it as missing, and then proceed.