Clearing Confusion on AF Drives


Recommended Posts

Here are the instructions for dealing with Advanced Format (AF) drives in upcoming unRAID Server versions 4.6.1 and 5.0-beta3.

 

Bottom line:

  • A hard drive already formatted by unRAID version 4.6/5.0-beta2 or below, will be recognized and left as-is in version 4.6.1/5.0-beta3 (regardless of being AF or not).
  • If you format a hard drive using version 4.6.1/5.0-beta3 it will create partition 1 starting in sector 64, instead of sector 63, regardless of being AF or not. The reported usable size of the hard drive will not change.  NOTE: If later you need to revert to unRAID version 4.6/5.0-beta2 or earlier, your drive will appear to be unformatted.  So once you upgrade to 4.6.1/5.0-beta3 any drives you subsequently format will not be backward compatible with earlier unRAID versions.

 

Action items for the WD20EARS drive:

  • If you are installing a new WD20EARS drive, and you are running version 4.6.1/5.0-beta3, do not install the jumper.
  • If you already have the jumper installed, and the drive has already been formatted by unRAID version 4.6/5.0-beta2 or earlier, you do not have to do anything.  Your drive is already operating at full speed, and I recommend you just leave it this way.  If however, later you re-format the drive in version 4.6.1/5.0-beta3, you should first remove the jumper (see next point).
  • If you have O.C.D. and must remove that jumper from an already-formatted hard drive, you must first backup any data you want to keep.  Then shutdown your server, remove the jumper from the drive, and power up your server.  You then may Format the drive using version 5.0-beta3 by clicking the Format button in the webGui page for the disk.  This will position the start of partition 1 on sector 64.  You MUST then run a parity-sync operation because parity will be silently invalid.  So, if you already have that jumper installed, and the drive has already been formatted, just leave sleeping dogs alone if you possibly can!

 

Action items for other AF drives:

  • To get maximum performance out of these drives (that do not have the WD style jumper), you should first backup any data you want to keep.  Then reformat the drive using version 5.0-beta3 by clicking the Format button in the webGui page for the disk.  This will position the start of partition 1 on sector 64.

 

Questions:

If I remove the jumper from a formatted WD20EARS or other AF drive, how can I force version 4.6.1 to reformat the drive?  I don't recommend you do this because it involves executing shell commands; and if you type the wrong thing you could destroy data.

 

If I use a WD20EARS for parity, should the jumper be on or off?  Same situation as a data drive.  The parity drive is also "formatted" but only to the extent that an MBR with partition 1 defined is created (no file system is created obviously).  So if you're upgrading an existing server with valid parity to 4.6.1/5.0-beta3, leave the jumper in place.  To go jumperless it's a bit easier.  Power down server, remove Parity, power up server and Start.  This will "disable" parity.  Now power down again, install jumperless Parity, power up and start Parity sync.

 

If a drive fails or is being upsized, can you simply pull the disk, install an unjumpered AF drive, and let it rebuild?  Will it automatically do the partition change?  Yes.

 


 

Ok, in the interest of lessening the confusion (hopefully), let me explain how the code works and that might shed some light.

 

- Currently, every disk in the array, whether a data disk or the parity disk, has a "partition 1" that starts in sector 63 and extends to the end of the disk.

 

- Parity is calculated only across the data of partition 1 of the data drives, and written to partition 1 of the parity drive.

 

- When you click Start, for any drive marked as "new", the code will check the drive's MBR, and if it's not described as above, then it re-writes it so that it is described as above.

 

The change in the software involves two things:

 

1. Any time the code writes the MBR it will now define partition 1 as starting in sector 64 instead of sector 63, and extending to the end of the disk.

 

2. When the code checks the MBR, it will accept it if partition 1 starts either in sector 63 or sector 64.  If it starts anywhere else, or if it's length does not comprise the whole disk, then it re-writes it so that partition 1 starts in sector 64 and extends to the end of the disk.

 

Now you might be wondering: if you move the start of partition 1 up by one sector, won't partition 1 end up being 1 sector smaller?

 

No because when I said "extending to the end of the disk", that's not exactly correct.

 

Partition 1 is ultimately going to be written with a ReiserFS file system, and this file system (like most file systems), has a 4K unit of allocation.  So when the code calculates the size of partition 1 it ensures that it's a multiple of 4K (i.e., 8 sectors).

 

Now since in current code partition 1 starts in sector 63, and 63 is an odd number, and the number of sectors on a disk drive is always an even number, then there will always be at least 1 unused sector at the end of every disk.

 

Hence, if we start partition 1 in sector 64, we are guaranteed that it's size, in terms of 4K blocks, must be exactly the same as it would be if it started in sector 63.

 


 

Where did this sector 63 nonsense start anyway?  Well you can research it, start by googling Master Boot Record.  Basically it all started when some idiot hardware engineer decided to start sector numbering at 1 instead of at 0.  This meant that a 6-bit field could only hold 63 numbers.  So in a cylinder/head/sector addressing scheme, a "track" can hold at most 63 sectors.  And what the guys who created the MBR scheme did was decide to put MBR into sector 0 (that's ok), and reserve the remainder of the track (which can be at most 63 sectors because sector numbering started at 0 instead of 1).

 

Anyway, when I first started tinkering with what would become unRaid I guess around 6 years ago, there were still plenty of drives around that used the old cylinder/head/sector scheme for identifying sectors (instead of Logical Block Address used exlusively nowadays).  I specifically remember thinking, "this is pretty stupid to start a partition on an odd-sector boundary", and I originally had my partitions starting in sector 32.  But... turns out that some boot utilities (like Grub), use the "unused" sectors between the MBR (sector 0) and the start of partition 1 to hold "secondary boot code".  These utilities tended to "rely" on there being 62 unused sectors in there and I didn't want to break the use of these utilities, so I went ahead, followed the pack, and started partition 1 on sector 63 like those idiots at M$.  ::)

Link to comment
  • Replies 95
  • Created
  • Last Reply

Top Posters In This Topic

I wondered what you would do if you had an AF drives as parity. Would you want to have the jumper in place or removed?

 

Also - do these rules apply only to AF drives, or will any drive installed under 4.6.1/5.0b3 be partitioned at sector 64.

 

If a drive fails or is being upsized, can you simply pull the disk, install an unjumpered AF drive, and let it rebuild?  Will it automatically do the partition change?

Link to comment

I wondered what you would do if you had an AF drives as parity. Would you want to have the jumper in place or removed?

Added that question to the post.

 

Also - do these rules apply only to AF drives, or will any drive installed under 4.6.1/5.0b3 be partitioned at sector 64.

Clarified in the post: yes rules apply to all drives 2TB and less in size.  High capacity drives are another kettle of fish.

 

If a drive fails or is being upsized, can you simply pull the disk, install an unjumpered AF drive, and let it rebuild?  Will it automatically do the partition change?

Added that question to the post, answer: yes.

Link to comment

Followup question:

 

Say someone is currently running 4.6 or older and using unjumpered WD EARS drives (against the better judgment of these forums).  Say this person then upgrades to 4.6.1 or newer.  Any potential issues?

 

It will run at less than max efficiency like it would under those conditions in 4.6.  You could reformat however just by clicking a button.

Link to comment

This is great, timing is perfect for me!  :)

 

Just one additional scenario though - if you are migrating a drive from an existing filesystem, do you just treat it as if it was a new drive in the new versions of unRAID - I.e. Add it to the system unjumpered and the select format in the GUI, or does something else have to be done in addition to get it ready for unRAID?

Link to comment

This is great, timing is perfect for me!  :)

 

Just one additional scenario though - if you are migrating a drive from an existing filesystem, do you just treat it as if it was a new drive in the new versions of unRAID - I.e. Add it to the system unjumpered and the select format in the GUI, or does something else have to be done in addition to get it ready for unRAID?

 

If you mean with some other file system type, such as NTFS?  Yes in this case unRaid 4.6.1/5.0-beta3 will treat as an unformatted drive and create partition 1 starting in sector 64 and then make a ReiserFS file system in partition 1.

Link to comment

Perhaps it is asking the obvious...

 

When running with the new version and new drives are being added, is it recommended that all drives are formatted the same or can old 'jumpered' drives starting on sector 63 live happily next to new 'unjumpered' drives starting at sector 64?

 

In other words future expansion does not require a rebuild of anything existing?

 

Link to comment

what about people who used the alignment tool ?

I have not yet started to fill my 2g drives.. so I will definitely reformat them

but I believe I read somewhere (hopefully not in my dreams) that 'aligned' disk would mean trouble....

 

R

 

What alignment tool?   ???

 

lol.sorry I have to learn to be more specific.

 

I am talking about the WDalign (windows) tool.

when using the drive on windows WITHOUT the jumper this tool can be run to align the partition

 

there is also a tool from paragon (paragon alignment tool) which is supposed to do some handling for all type of disks.

it was free (got a copy) but now seems to be 30$

 

if my memory is not too messy I had the impression that a drive that have been aligned (at least with wdalign) can be a mess later.

but i can't find this information again, so I wonder if I did not make it up.... lol

 

R

Link to comment

Perhaps it is asking the obvious...

 

When running with the new version and new drives are being added, is it recommended that all drives are formatted the same or can old 'jumpered' drives starting on sector 63 live happily next to new 'unjumpered' drives starting at sector 64?

 

In other words future expansion does not require a rebuild of anything existing?

 

 

If you are installing a "new" drive, such that unRaid will want to format it, you should have jumper removed.

 

The guy at WD who thought this jumper was a good idea should have his head removed  >:(

Link to comment

what about people who used the alignment tool ?

I have not yet started to fill my 2g drives.. so I will definitely reformat them

but I believe I read somewhere (hopefully not in my dreams) that 'aligned' disk would mean trouble....

 

R

 

What alignment tool?   ???

 

I am talking about the WDalign (windows) tool.

when using the drive on windows WITHOUT the jumper this tool can be run to align the partition

 

there is also a tool from paragon (paragon alignment tool) which is supposed to do some handling for all type of disks.

it was free (got a copy) but now seems to be 30$

 

if my memory is not too messy I had the impression that a drive that have been aligned (at least with wdalign) can be a mess later.

but i can't find this information again, so I wonder if I did not make it up.... lol

 

R

 

 

Ok, right. All that tool does is move the partitions around so they are aligned on 4K boundaries.  Once you plug into your server, unRaid will want to format it; in this case make sure jumper is removed (for version 4.6.1/5.0-beta3).

Link to comment

what about people who used the alignment tool ?

I have not yet started to fill my 2g drives.. so I will definitely reformat them

but I believe I read somewhere (hopefully not in my dreams) that 'aligned' disk would mean trouble....

 

R

 

What alignment tool?   ???

 

I am talking about the WDalign (windows) tool.

when using the drive on windows WITHOUT the jumper this tool can be run to align the partition

 

there is also a tool from paragon (paragon alignment tool) which is supposed to do some handling for all type of disks.

it was free (got a copy) but now seems to be 30$

 

if my memory is not too messy I had the impression that a drive that have been aligned (at least with wdalign) can be a mess later.

but i can't find this information again, so I wonder if I did not make it up.... lol

 

R

 

 

Ok, right. All that tool does is move the partitions around so they are aligned on 4K boundaries.  Once you plug into your server, unRaid will want to format it; in this case make sure jumper is removed (for version 4.6.1/5.0-beta3).

 

thank you boss... :D

man I feel like an ignorant.

 

 

Link to comment

Good work Tom. Is your head spinning yet trying to track all the issues these drives are causing?

 

 

what about people who used the alignment tool ?

I have not yet started to fill my 2g drives.. so I will definitely reformat them

but I believe I read somewhere (hopefully not in my dreams) that 'aligned' disk would mean trouble....

 

R

 

There was a post about a drive having a bunch of issues when someone has use WDalign in Windows and then tried to use it in unRAID but he had added the jumper as well. Here's the post;

 

http://lime-technology.com/forum/index.php?topic=8333.0

 

I believe you will be fine since you will not be adding the jumper. It seems some WD AF drives throw fits when you add the jumper after it was formatted. Typically, it seems that writing 00's to the first part of the drive (I think the first 64 sectors) will make it behave again but not always.

 

Of course, I also wonder if some people are jumping the wrong pins and that messes the drive up....

 

Peter

 

Link to comment

Good work Tom. Is your head spinning yet trying to track all the issues these drives are causing?

 

 

what about people who used the alignment tool ?

I have not yet started to fill my 2g drives.. so I will definitely reformat them

but I believe I read somewhere (hopefully not in my dreams) that 'aligned' disk would mean trouble....

 

R

 

There was a post about a drive having a bunch of issues when someone has use WDalign in Windows and then tried to use it in unRAID but he had added the jumper as well. Here's the post;

 

http://lime-technology.com/forum/index.php?topic=8333.0

 

I believe you will be fine since you will not be adding the jumper. It seems some WD AF drives throw fits when you add the jumper after it was formatted. Typically, it seems that writing 00's to the first part of the drive (I think the first 64 sectors) will make it behave again but not always.

 

Of course, I also wonder if some people are jumping the wrong pins and that messes the drive up....

 

Peter

 

 

Well that is an interesting thread.  It appears WD's tool might set a "secret bit" somewhere via ATA command that flags the firmware somehow, and that interacts with the jumper in some fashion.  I guess this is plausible but it's hard to believe even WD would be that stupid.  Still, I can see some manager yelling at his firmware team: "Just MAKE IT WORK with Windows, I don't care how you have to do it!"  So far I've resisted getting one of these boat anchors, but looks like we'll have to buy a couple  >:(

Link to comment
Well that is an interesting thread.  It appears WD's tool might set a "secret bit" somewhere via ATA command that flags the firmware somehow, and that interacts with the jumper in some fashion.  I guess this is plausible but it's hard to believe even WD would be that stupid.  Still, I can see some manager yelling at his firmware team: "Just MAKE IT WORK with Windows, I don't care how you have to do it!"  So far I've resisted getting one of these boat anchors, but looks like we'll have to buy a couple  >:(

 

There have been many threads started because of issues getting an already formatted and jumperless WD AF drive to accept the jumper and this occurs regardless of using the WDalign tool or not. The drive acting up or not also seems quite random. So, I'm not sure the jumper and WDalign were acting together. I suspect the issue all started simply due to adding the jumper to an already formatted drive and the use of WDalign was just a red herring. Starting with these new releases, an existing formatted drive won't need the jumper installed so there should be no issues. I guess we'll know once some people try it.

 

Peter

 

Link to comment
Once you plug into your server, unRaid will want to format it; in this case make sure jumper is removed (for version 4.6.1/5.0-beta3).

 

I thought that, at one stage, there was a suggestion (I don't recall whether it came from you) that the final 4.x would recognise a sector 64 start (allowing reversion from 5.x to a 4.final system), but would always format to a sector 63 start (allowing reversion from 4.final to an earlier 4.x system).  Obviously, there would be pros and cons for this approach.  I guess that you've concluded that this would not be a good idea.

Link to comment

Once you plug into your server, unRaid will want to format it; in this case make sure jumper is removed (for version 4.6.1/5.0-beta3).

 

I thought that, at one stage, there was a suggestion (I don't recall whether it came from you) that the final 4.x would recognise a sector 64 start (allowing reversion from 5.x to a 4.final system), but would always format to a sector 63 start (allowing reversion from 4.final to an earlier 4.x system).  Obviously, there would be pros and cons for this approach.  I guess that you've concluded that this would not be a good idea.

Based on other posts from Tom, I think he is still working towards exactly that goal.

I think 4.6.1 will have the ability to deal with partitions at sector 63 or 64.) We'll know for sure when it is released.

 

Joe L.

Link to comment

Once you plug into your server, unRaid will want to format it; in this case make sure jumper is removed (for version 4.6.1/5.0-beta3).

 

I thought that, at one stage, there was a suggestion (I don't recall whether it came from you) that the final 4.x would recognise a sector 64 start (allowing reversion from 5.x to a 4.final system), but would always format to a sector 63 start (allowing reversion from 4.final to an earlier 4.x system).  Obviously, there would be pros and cons for this approach.  I guess that you've concluded that this would not be a good idea.

Based on other posts from Tom, I think he is still working towards exactly that goal.

I think 4.6.1 will have the ability to deal with partitions at sector 63 or 64.) We'll know for sure when it is released.

 

But the (implied) question was whether 4.6.1 would always format to start at 64.  Tom's post above (and the description in the Roadmap) suggests that it will.

Link to comment

5 is still in beta and users badly want AF support, even though it's not the show-stopping issue the internet FUD would make you believe, so the fix which makes the most sense is to just start formatting every drive so the partition starts at sector 64.

 

Just keep running 4.6 if you want to retain backwards compatibility to earlier versions. I expect the only change in 4.6.1 will be the disk formatting.

 

Peter

 

Link to comment

Once you plug into your server, unRaid will want to format it; in this case make sure jumper is removed (for version 4.6.1/5.0-beta3).

 

I thought that, at one stage, there was a suggestion (I don't recall whether it came from you) that the final 4.x would recognise a sector 64 start (allowing reversion from 5.x to a 4.final system), but would always format to a sector 63 start (allowing reversion from 4.final to an earlier 4.x system).  Obviously, there would be pros and cons for this approach.  I guess that you've concluded that this would not be a good idea.

Based on other posts from Tom, I think he is still working towards exactly that goal.

I think 4.6.1 will have the ability to deal with partitions at sector 63 or 64.) We'll know for sure when it is released.

 

But the (implied) question was whether 4.6.1 would always format to start at 64.  Tom's post above (and the description in the Roadmap) suggests that it will.

That is my understanding also.  But to be more accurate, it will create a VALID MBR with the first partition starting at sector 64, but only if the disk does not already have an MBR that is valid.  It will not re-partition an existing disk with an existing file-system on partition 1. 

 

Also, from a more recent note from Tom, his new scheme will accept pre-cleared disks as it always has, but write a partition table to start the 1st partition at sector 64.  (good news for unRAID users, and me, since the existing pre-clear script I wrote would need no changes to specifically deal with sector 64)

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.