Need Community input on 4K-sector drives

[Joe L.]

The result was a large number of hardware IO errors. Something in unRAID (or unMenu) was trying to read the very last sector of the drive which for some reason was unreadable.

The pre-read and post-read both work on the "reported geometry" and constantly access the first and last sector on the disk in addition to random sectors, and also linearly all the sectors in between.   It sounds as if the geometry as reported after adding the jumper is confusing the pre-clear script, and the drive is having a tough time getting to the final sector. (which might be one too high for all I know)

 

In any case, writing zeros to the first few blocks, including the MBR, will cause it to report a more sane geometry, and then all is back as expected.

dd if=/dev/zero of=/dev/sdX count=4

should do it if you add the jumper and then want to perform a pre-clear again.

 

Joe L.

[dgaschk]

 

In any case, writing zeros to the first few blocks, including the MBR, will cause it to report a more sane geometry, and then all is back as expected.

dd if=/dev/zero of=/dev/sdX count=4

should do it if you add the jumper and then want to perform a pre-clear again.

 

Joe L.

 

Joe, Can you add the dd command to the preclear script?

[Joe L.]

 

In any case, writing zeros to the first few blocks, including the MBR, will cause it to report a more sane geometry, and then all is back as expected.

dd if=/dev/zero of=/dev/sdX count=4

should do it if you add the jumper and then want to perform a pre-clear again.

 

Joe L.

 

Joe, Can you add the dd command to the preclear script?

I could... but not sure if it is a universal solution.  I don't want to blindly do it in every drive.

[BRiT]

For those who do not read the other topics but have the Samsung drives, here is vitally important information for you.

 

WARNING: There is a really scary issue with all the Samsung F4 drives, as reported in this thread: http://lime-technology.com/forum/index.php?topic=9339.msg89029

 

The bug report indicates Samsung F4's can generate Bad Blocks when hdparm or smartmontools or any other disk tool is used on them. They will silently corrupt your data!

 

Stay away, far away from these drives if you value your data!

[PeterB]

All I/O in the unRaid driver is executed relative to the start of each disk's partition 1.

 

That being the case, then my suggestion could be used, even now, for any 'Advanced Format' drive.

 

What happens if you let unRAID format the drive, then use parted to move the partition onto a 4kB boundary?

[limetech]

All I/O in the unRaid driver is executed relative to the start of each disk's partition 1.

 

That being the case, then my suggestion could be used, even now, for any 'Advanced Format' drive.

 

What happens if you let unRAID format the drive, then use parted to move the partition onto a 4kB boundary?

 

Unfortunately for current unRaid versions, unRaid expects the MBR to be a precise format, with start of partition 1 in sector 63.  So if you use some utility to move the partition up one sector, the drive will appear "unformatted" to unRaid.

[PeterB]

Unfortunately for current unRaid versions, unRaid expects the MBR to be a precise format, with start of partition 1 in sector 63.  So if you use some utility to move the partition up one sector, the drive will appear "unformatted" to unRaid.

 

Ah, okay.

[mbryanr]

Samsung HD204UI

Samsung link

Seagate ST2000DL03 (incorporates SmartAllign)

 

Seagate also has 1.5TB and 1TB versions with Advanced Format.

 

It is unclear whether Hitachi new 7K3000 drives are Advanced Format.  I have sent an email to Hitachi support to clarify.

Hitachi states that the interface is 512B...but that may just be the logical interface.

 

 

Response from HitachiGST:

Advance format drive meaning as 4k sector, yes this drive is.

[pras1011]

For those who do not read the other topics but have the Samsung drives, here is vitally important information for you.

 

WARNING: There is a really scary issue with all the Samsung F4 drives, as reported in this thread: http://lime-technology.com/forum/index.php?topic=9339.msg89029

 

The bug report indicates Samsung F4's can generate Bad Blocks when hdparm or smartmontools or any other disk tool is used on them. They will silently corrupt your data!

 

Stay away, far away from these drives if you value your data!

 

I have a reply from Samsung!!!!!

 

Our HQ engineering are aware of this problem and a solution will be ready very soon. I believe we will have a new firmware

 

and will let you when it will be available for download from our website.

 

Regards,

Chas.

 

QA manager - SSEL

[cwr]

is it possible to get some idea of how these advanced format drives will work in unraid 5.x for the people already using them? basically what i think people need to know is;

 

- if we are using the drives with the jumper, is this ok with unRAID 5.x?

- if we are using the drives without the jumper, will this work ok in unRAID 5.x?

- again for drives without the jumper, can they be reformatted to take advantage of the new format without a risk to the data

 

atleast where i live it's getting almost impossible to find non-advanced format drives, so i kind of need to start planning for this now even if unraid is X months away from a 5.x release supporting these drives

[lionelhutz]

I think the below answers this.

 

So it will not be difficult to add code (in version 5.x) to put the partition 1 offset at 64 for new drives installed in the server.

 

Basically, I'm reading that Tom will begin to format newly installed drives with the partition starting at sector 64 instead of 63. So, looking at your points;

 

- EARS with jumper would transition over with no changes and remain aligned.

 

- EARS without jumper would transition over with no changes but remain unaligned.

 

- EARS without jumper could be rebuilt with a new partition that would start at sector 64. I would expect you just do a stop/unassign/start/stop/assign/start and then the drive would be rebuild with this a new starting sector. Basically, you make the array thing the disk is lost and then it will treat that disk as a replacement. However, it might also require a further clearing step to erase old partition data. This process puts the array into a "degraded" state while it rebuilds which puts data at risk. However, it would be no more at risk than during any other drive failure.

 

Peter

 

 

[greybeard]

I would expect you just do a stop/unassign/start/stop/assign/start and then the drive would be rebuild with this a new starting sector.

 

My preference would be to use a spare drive swap approach. Something like this.

1. Shutdown server.

2. Replace drive to convert with with a fully tested spare.

3. Startup and do rebuild.

4. Run CRC checks and a verify.

5. Assuming all is good, remove the jumper from the replaced drive.

6. Install in a test system and test it.

7. If necessary use any of the document processes to fix problems.

8. Now you have another tested spare to use for the next drive to be migrated.

 

I am sure there other variations on the process I outlined that would work just fine too. Obviously you would need to start with the largest drive and work back to the smallest. This process preserves your original files until you are sure they are properly rebuilt on the spare drive.

[lionelhutz]

That could work too and it could allow close to a full recovery if another failure occured during the rebuild steps.

 

I'm not sure why you're worried about removing the jumper off a WD drive though. There is absolutely no benefit in getting rid of the jumper unless your compulsive nature just can't allow you to sleep at night until you do so. The WD drives with a jumper will continue to perform just as well as an unjumpered WD drive that is using the proposed new formatting. The rebuilding could benefit the Seagate, Samsung and Hitachi drives as well as any WD drives that were installed in the array without the jumper.

 

Peter

 

[limetech]

I think the below answers this.

 

So it will not be difficult to add code (in version 5.x) to put the partition 1 offset at 64 for new drives installed in the server.

 

Basically, I'm reading that Tom will begin to format newly installed drives with the partition starting at sector 64 instead of 63. So, looking at your points;

 

- EARS with jumper would transition over with no changes and remain aligned.

 

- EARS without jumper would transition over with no changes but remain unaligned.

 

- EARS without jumper could be rebuilt with a new partition that would start at sector 64. I would expect you just do a stop/unassign/start/stop/assign/start and then the drive would be rebuild with this a new starting sector. Basically, you make the array thing the disk is lost and then it will treat that disk as a replacement. However, it might also require a further clearing step to erase old partition data. This process puts the array into a "degraded" state while it rebuilds which puts data at risk. However, it would be no more at risk than during any other drive failure.

 

Peter

 

This is correct.  The software change would involve:

 

1a. The unRaid 'Format' function would have a checkbox that says something like "Advanced Format Drives", which, if checked, would cause partition 1 to start in sector 64, if not checked, start in sector 63 as it does now (and the default would be "not checked").

1b. Alternately, the checkbox would be on the individual 'Disk' pages & maybe we can generate code that automatically determines if this is an Advanced Format drive [anyone know a way to do this?] in order to set the default checked-ness of the box.

2. Other code would be changed to recognize either partition format as valid: partition 1 starting in sector 63, or partition 1 starting in sector 64.

 

That's pretty much it:

- Existing formatted jumpered WD drives you should just leave jumper on.

- Existing formatted non-advanced format drives, just leave them alone.

- Installing new drives in an existing array: well you have a choice.  If the drives are Advanced Format WD's I'd probably be inclined to install the jumper and use the old-style partition scheme (because of issue below).  If non-WD Advanced Format, well go ahead and use new partition format.

 

Here's the main issue with this as far as I'm concerned: Once a drive gets formatted with partition 1 starting in sector 64, then all previous versions of unRAID OS will think this is an "unformatted" drive.  So whatever release this change is put into, better be a pretty solid release

 

<thinking-out-loud>

This presents an implementation issue for me.

 

If I put this change in version 5.0 then, since 5.0 is still beta, there will be a lot of pressure to release a 5.0 "final" in order to use this feature in "production" servers.  But 5.0 includes a lot of changes, and will require a one-time execution of a utility to change all the file permissions on the array.

 

If I put this change in the 4.x series (e.g., maybe 4.6.1), then this delays 5.0 release, and there are other reasons why I want to move to 5.0 - in particular there are fixes in the newest linux kernels for the mvsas driver, and fixes in newest Samba for Win7 issues... I really don't want to upgrade these components and make, e.g., a 4.7 release.

</thinking-out-loud>

 

So the issue for me is not the fix itself, but how to release the fix.

[madpoet]

Being selfish I'd like a 4.6.1   Even if this was the only fix to it.  But I understand if you want a stable 5.x version for it.

[lionelhutz]

1a and 1b - why have the option? Could you just not begin to use sector 64 for every drive in the future? All drives will be advanced format drives very soon so there won't be any benefits to allowing the sector 63 partition in the future.

 

I hear you about the release schedule. That's a tough one to decide. Maybe try for an initial 5.0 release that does not include all the extras you wanted to add (like AFP)?

 

The big thing people need to realize is that these advanced format drives work just like any other drive. However, aligning the partition to sector 63 can mean slower reads and writes for small files (like under 5k or 10k in size), compared to the possible read/write speed for an aligned partition. However, I think it's blown out of proportion because there are few small files used these days and they read and write very quick anyways, regardless if the drive is aligned or not. Having said that though, I've been using jumpered WD drives due to my feeling that Seagate's are inferior and because I haven't find any deals on Samsung drives when I needed one.

 

Peter

 

[WeeboTech]

As much as I would like to see 4.6x being perfect, it seems inevitable that feature creep will occur and another beta cycle will be required.

 

I say get 5.0 going and slate the advanced format for 5.0.x or 5.1.

 

I'm not fond of a checkbox on the front page for the formatting of advanced format drives.

 

I would prefer an option on the individual drive page.

Frankly, There needs to be more information on the individual drive page (smart info, filesystem format type etc, etc) so a checkbox on drive partition start is not so out of line.

 

You could also have a plain text file on the boot flash that gets updated with models known to be advance format drives.

Then do the autocheck option or have a red suggestion next to the checkbox.

 

[Joe L.]

I say get 5.0 going and slate the advanced format for 5.0.x or 5.1.

I agree.  I think it is imperative to get 5.0 (non-beta) out the door before tons of added features are bundled in.  It is too easy to be caught up in a feature-creep loop.

 

Once 5.0 (non-beta) works with existing functionality you can continue with new changes to the user-interface for advanced formatting of drives. (you can use the new kernel, samba and drivers since they don't really change the user-interface)

 

Lots of threads in the 4.5 series about Win7 disconnects and file transfer problems.  The new samba will probably help.  It can be introduced in the 5.0beta series.  I personally think it would be a mistake to have both sector63 partitions and sector64 partitions possible in the first 5.0 non-beta.  No way to revert easily to the 4.x series, and you know the person least skilled in Linux will be the one who MUST use the alternate sector alignment. Too many opportunities for a data loss as the format their disk in their haste to get to their data.

 

One possible migration method might be to change emhttp to recognize a partition starting on sector 64 as valid if it can be mounted as a reiserfs.  Do not add any code to format disks that way.  No extra checkboxes or file lists etc.  The change released as 4.6.1, would just provide a way to revert to 4.6.1 if needed with a drive formatted on the 5.1 release series and have it not be considered as un-formatted.  It would not provide any way to format a drive in this fashion via the web-interface.

 

I think that AFP and NFS are some of the remaining features needed in the 5.0beta series.  What we have so far is working just fine (with the few already reported bugs)  Those are where the focus should go initially so you can get a stable release out the door.

[prostuff1]

I agree with most everything that has been said above.

 

A 4.6.1 release that allows for the mount of drives but NO formatting would probably be a good way to allow some backwards movement.  Leave 4.6 and 4.6.1 on the downloads page and explain that if you are downgrading from a 5.X version of of unRAID that you need to run 4.6.1.

 

I would like to see AFP added, and some of the power saving options put into the 5.0 release before it is finalized.  Once those bigger things are in interface tweaking and the 4K alignment issue can be put into a 5.1 release.

[JM2005]

Being selfish I'd like a 4.6.1   Even if this was the only fix to it.  But I understand if you want a stable 5.x version for it.

 

I agree as well!

[cwr]

Being selfish I'd like a 4.6.1   Even if this was the only fix to it.  But I understand if you want a stable 5.x version for it.

 

I agree as well!

[teamhood]

Lots of threads in the 4.5 series about Win7 disconnects and file transfer problems.  The new samba will probably help.  It can be introduced in the 5.0beta series.   I personally think it would be a mistake to have both sector63 partitions and sector64 partitions possible in the first 5.0 non-beta.  No way to revert easily to the 4.x series, and you know the person least skilled in Linux will be the one who MUST use the alternate sector alignment. Too many opportunities for a data loss as the format their disk in their haste to get to their data.

 

If going to 5.0 and a new kernel will help, than this is a must. If I cannot copy data to my unRAID server, then what is the point of having that beast? This is really a huge problem for me and I continue to see new threads with users having this issue.

[reggie14]

This is correct.  The software change would involve:

 

1a. The unRaid 'Format' function would have a checkbox that says something like "Advanced Format Drives", which, if checked, would cause partition 1 to start in sector 64, if not checked, start in sector 63 as it does now (and the default would be "not checked").

1b. Alternately, the checkbox would be on the individual 'Disk' pages & maybe we can generate code that automatically determines if this is an Advanced Format drive [anyone know a way to do this?] in order to set the default checked-ness of the box.

2. Other code would be changed to recognize either partition format as valid: partition 1 starting in sector 63, or partition 1 starting in sector 64.

 

That's pretty much it:

- Existing formatted jumpered WD drives you should just leave jumper on.

- Existing formatted non-advanced format drives, just leave them alone.

- Installing new drives in an existing array: well you have a choice.  If the drives are Advanced Format WD's I'd probably be inclined to install the jumper and use the old-style partition scheme (because of issue below).  If non-WD Advanced Format, well go ahead and use new partition format.

 

I would really hate to see a special option for advanced format drives, even if there's a mechanism for guessing what it should be. 

 

And I'm actually a little confused why it would help to be able to detect an advanced format drive.  I thought it was always safe to start drives at 64, except when you have a Western Digital advanced format drive with the jumper set.  Being able to detect an advanced format drive wouldn't fix that problem.

 

The other problem that I thought was possible was that there might be problems recovering a failed non-advanced format drive to an advanced format drive.  But I thought the (apparent) fact that drives have an even number of sectors saves us from that problem.

 

So, why can't you just make all newly formatted drives, including new drives that are replacing failed drives, start at 64?  If someone already has a formatted and jumpered WD drive, it would still work.  Its just that people with unformatted WD drives would have to be informed to not set the jumper.

 

If I put this change in version 5.0 then, since 5.0 is still beta, there will be a lot of pressure to release a 5.0 "final" in order to use this feature in "production" servers.  But 5.0 includes a lot of changes, and will require a one-time execution of a utility to change all the file permissions on the array.

 

If I put this change in the 4.x series (e.g., maybe 4.6.1), then this delays 5.0 release, and there are other reasons why I want to move to 5.0 - in particular there are fixes in the newest linux kernels for the mvsas driver, and fixes in newest Samba for Win7 issues... I really don't want to upgrade these components and make, e.g., a 4.7 release.

 

I really wanted to say that I'd rather see 4k support sooner than anything I'm seeing coming down the pipe in v5, but, as much as it pains me to say it, the things you mentioned sound pretty important.  The only argument I see going the other way is that we're quickly approaching a time when only 4k drives will be available, and I think the more people are using hacks like the WD jumper, the harder it will be settle in on a long term solution.

 

Though, I think Joe is probably right to suggest avoiding incorporating a big change like that in the v5 release off the bat.  So, the way to do it probably is to either put it in a 4.6.x/4.7 version, or a 5.1 version.

 

One possible migration method might be to change emhttp to recognize a partition starting on sector 64 as valid if it can be mounted as a reiserfs.  Do not add any code to format disks that way.  No extra checkboxes or file lists etc.  The change released as 4.6.1, would just provide a way to revert to 4.6.1 if needed with a drive formatted on the 5.1 release series and have it not be considered as un-formatted.  It would not provide any way to format a drive in this fashion via the web-interface.

 

A concern I would have here is that you'd be adding a feature that (essentially) isn't tested, at least, not be beta users.  Though, this sounds like a small enough change that it could be done with minimal testing.  Is that true?  Even if its not, there doesn't seem to be a better alternative.

[Ostrich]

As much as I would like to see 4.6x being perfect, it seems inevitable that feature creep will occur and another beta cycle will be required.

 

I say get 5.0 going and slate the advanced format for 5.0.x or 5.1.

 

I'm not fond of a checkbox on the front page for the formatting of advanced format drives.

 

I would prefer an option on the individual drive page.

 

Agree with weebo, lock away 4x series, get 5.0 out the door, 4K can be introduced as part of 6x (along with warnings that once you format a drive in 6x you cannot go to a previous version or else youll lose data)

 

Ost.

[gfjardim]

One thing I've not found here is compatibility with 3TB+ drives. The 4K sectors will need MD driver reformulation, but 3TB+ drives need GPT format too. Is ReiserFS compatible with GPT? Since we don't need to support GPT Boot, this task can be focused only on FS and hardware support.

 

I agree with others, version 5.0 is imperative due to migration to Slackware 13.1 base and new plugin schema, 5.0.1 should be focused on AFP, since many members here have Macs. So maybe we can postpone 4K format to 5.0.2, when 3TB drives become widely available, and controller manufactures can adapt their firmwares to better support them. It's all very immature at this point, and very risky from reliability perspective.