There is a bug in the 4.5.3 release that can make drives erroneously appear 'Unformatted'.

 

EDIT: This problem has been solved in version 4.5.4.

 

If you are using 4.5.3 or 4.5.2, please update your firmware to the latest release, 4.5.6.

 

Below is original post content.

---

Please post here only if you have encountered this bug.  I need to know the sequence of events that led to this, in addition to your hardware configuration.  If you have system log captured when drives were shown as 'unformatted', please include in your post.

 

If you can "make this happen" at will on your system, please email me: [email protected]

---

Update: I'm now able to reproduce this easily and know what the problem is... a new release, 4.5.4, will be forthcoming.  Thanks to DocBlock's post, who was able to reproduce using 4.5.3-TEST.

 

Here's what the problem was.  When the linux unraid driver is started, it eventually creates device nodes for each of the array data disks, ie, /dev/md1, /dev/md2, etc.  Actually - the driver does not create the nodes, it generates kernel events which are noticed by udev subsystem, and it's udev that creates the device nodes.

 

The problem is that there's a delay between starting the driver and the device nodes being created.  Later during unRAID server start up, we eventually try to mount the data disks like this:

 mount /dev/md1 /mnt/disk1

Trouble is that if the mount command is executed before /dev/md1 is created, the mount command will fail.  But later, /dev/md1 does exist, so the software thinks the hard disk behind /dev/md1 is unformatted.

 

Why does this fail in 4.5.3 and not previous releases?  Probably because in 4.5.2 I updated the 'udev' subsystem to the latest version, which apparently is slower under some circumstances.  Also, being an example of a true 'race' condition, whether you would see this problem depends on hardware in use.

 

Anyway, I'm working on the solution & will post it ASAP.

 

 

Tom,

 

My memory is not perfect, but didn't I report this EXACT same bug to you in 2007 in this post, and you subsequently fixed in 4.0-beta7:

http://lime-technology.com/forum/index.php?topic=578.msg3783#msg3783

 

With this image:

 

In fact, your description of the problem I identified three years ago was strikingly similar to the one you just posted in this thread. you could have cut and pasted it, it is that close:

 

Quote from Tom@lime-tech ... three years ago here

Problem was a race condition.  The unRAID driver creates a device for each disk, e.g., /dev/md1 for disk1, /dev/md2 for disk2, etc.  Actually, the driver registers the device with the kernel, which generates a hotplug event, which triggers a daemon (udevd) to actually create the device node.  Well the management utility "knows" the driver created the devices, so immediately after starting the driver, it proceeds to attempt to mount the /dev/md's.  But it can happen that the device node is not present when we try to mount, and thus mount fails and device appears "unfomatted".  [glow=yellow,2,300]Solution is to call 'udevsettle' before attempting mounts.[/glow]

 

Just figured I'd bring this old post to your attention, because either the old fix no longer works, OR you should take a different approach now as you put into place a new patch.

Joe L.

I don't see any evidence of you invoking udevsettle in the 4.5.3 syslog.  Did you remove it when you put into place the new udev system in 4.5.3?

Good News!

Yes, good memory Joe.  The 'udevsettle' command was removed from udev package several rev's back & I removed it's invokation back around 4.5-beta12 (Nov '09).  There's a new way to do this using 'udevadm settle' command.  Anyway, I hate udev 

Good news Lime Tech!

 

BTW: The current version of udev is 1.5.3 used in Slackware-13.1 Beta1 (as of May 6th). However, there does seem to be a udev 1.5.4 release. I'm not sure if there's too much of a change to also update to that, but it would be nice. Damn, keeping current is a hassle.

Good news Lime Tech!

 

BTW: The current version of udev is 1.5.3 used in Slackware-13.1 Beta1 (as of May 6th). I'm not sure if there's too much of a change to also update to that, but it would be nice.

 

Why?

I simply figured with the recent attention to this issue it might be nice to make sure the fix works on future and current versions so we can be fairly confident we won't see this same issue in the future.

I simply figured with the recent attention to this issue it might be nice to make sure the fix works on future and current versions so we can be fairly confident we won't see this same issue in the future.

 

Ordinarily I'd agree, but udev is a particularly complex subsystem & I don't want to delay this fix any more than I have to.

udev has had it's fair share of API changes, but it seems to have settled out. I wouldn't be too concerned over a slightly newer version of udev unless there were more API changes there. A quick glance doesn't look like there is.

 

And I agree with Tom, it's far better to get a patch out immediately to address the current (major) issue, than delay it while he tests a newer version of a subsystem that doesn't need an upgrade to fix the issue.

Ordinarily I'd agree, but udev is a particularly complex subsystem & I don't want to delay this fix any more than I have to.

 

I completely agree with that.

 

Sorry for the side-distraction. Perhaps for the unRAID 5 release. 

udev has had it's fair share of API changes, but it seems to have settled out. I wouldn't be too concerned over a slightly newer version of udev unless there were more API changes there. A quick glance doesn't look like there is.

 

And I agree with Tom, it's far better to get a patch out immediately to address the current (major) issue, than delay it while he tests a newer version of a subsystem that doesn't need an upgrade to fix the issue.

 

Issues with udev tend to be caused by extremely subtle race conditions - probably the worst kind of bugs.

Good stuff.

Is there an ETA on a fixed version? Need to upgrade a drive and add some drives to my array and prefer

to not take any chances with the current version.

 

Thanks

 

Mark

 

Isn't the case to mount array drives using udev rules instead? I mean, the udev captures the kernel event generated by the unRAID md driver, creates the device node and then mount it in /mnt directory, without the unRAID service assistance? Or maybe the udev rule only notify the unRAID service that the node is created, and then it could be safely mounted.

Is there an ETA on a fixed version? Need to upgrade a drive and add some drives to my array and prefer

to not take any chances with the current version.

 

Thanks

 

Mark

 

 

I'm hoping to get it posted tonight.

Isn't the case to mount array drives using udev rules instead? I mean, the udev captures the kernel event generated by the unRAID md driver, creates the device node and then mount it in /mnt directory, without the unRAID service assistance? Or maybe the udev rule only notify the unRAID service that the node is created, and then it could be safely mounted.

 

It could be done that way, but there are a few complications.  One is that the mount options change depending on whether Active Directory is enabled or not.  Another is that in some cases a 'remount' follows a 'mount' in order to increase the size of the file system (in cases where small drive is replaced by larger drive).

It could be done that way, but there are a few complications.  One is that the mount options change depending on whether Active Directory is enabled or not.  Another is that in some cases a 'remount' follows a 'mount' in order to increase the size of the file system (in cases where small drive is replaced by larger drive).

 

You're right, a monolithic service is easy to maintain and potentially less buggy. There are other ways to probe the node creation before the mount.

 

It's totally off topic, but is it difficult to add Cool'n Quiet module to the new kernel, Tom? It will be very helpful in my case, since I only have AMD platforms to run unRAID.

 

Thanks, Tom, for your patience.

add Cool'n Quiet module to the new kernel

 

How?

I'm a linux noob, but I think the "processor" and the "powernow-k8" module must be loaded in menuconfig before compiling the kernel, and the cpufrequtils service must run on startup.

 

In the Gentoo documentation I've found this:

 

Code Listing 3.1: Kernel options for Cool'n'Quiet/PowerNow/SpeedStep

 

Power management options --->

  [*] Power Management support

  [*] ACPI (Advanced Configuration and Power Interface) Support --->

    <*>  Processor

    <*>    Thermal Zone

  CPU Frequency scaling --->

  [*] CPU Frequency scaling

        Default CPUFreq governor (ondemand) --->

  -*-  'performance' governor

  <*>  'userspace' governor for userspace frequency scaling

  -*-  'ondemand' cpufreq policy governor

  <*>  ACPI Processor P-States driver

  <*>  AMD Opteron/Athlon64 PowerNow!

-*- CPU idle PM support

 

Then emerge cpufrequtils and edit /etc/conf.d/cpufrequtils with your favorite editor.

 

Code Listing 3.2: Example of /etc/conf.d/cpufrequtils

 

# Options when starting cpufreq (given to the 'cpufreq-set' program)

START_OPTS="--governor ondemand"

 

# Options when stopping cpufreq (given to the 'cpufreq-set' program)

STOP_OPTS="--governor performance"

 

Now you only have to run rc-update add cpufrequtils boot to make sure it is started every time you boot.

 

Here some documentation I've found:

 

http://www.gentoo.org/doc/en/gentoo-amd64-faq.xml#coolnquiet

http://www.ndl.kiev.ua/content/linux-kernel-undervolting-patch-amd-processors

 

I'm a linux noob, but I think the "processor" and the "powernow-k8" module must be loaded in menuconfig before compiling the kernel, and the cpufrequtils service must run on startup.

 

In the Gentoo documentation I've found this:

 

Code Listing 3.1: Kernel options for Cool'n'Quiet/PowerNow/SpeedStep

 

Power management options --->

  [*] Power Management support

  [*] ACPI (Advanced Configuration and Power Interface) Support --->

    <*>   Processor

    <*>     Thermal Zone

  CPU Frequency scaling --->

  [*] CPU Frequency scaling

         Default CPUFreq governor (ondemand) --->

  -*-   'performance' governor

  <*>   'userspace' governor for userspace frequency scaling

  -*-   'ondemand' cpufreq policy governor

  <*>   ACPI Processor P-States driver

  <*>   AMD Opteron/Athlon64 PowerNow!

-*- CPU idle PM support

 

Then emerge cpufrequtils and edit /etc/conf.d/cpufrequtils with your favorite editor.

 

Code Listing 3.2: Example of /etc/conf.d/cpufrequtils

 

# Options when starting cpufreq (given to the 'cpufreq-set' program)

START_OPTS="--governor ondemand"

 

# Options when stopping cpufreq (given to the 'cpufreq-set' program)

STOP_OPTS="--governor performance"

 

Now you only have to run rc-update add cpufrequtils boot to make sure it is started every time you boot.

 

Here some documentation I've found:

 

http://www.gentoo.org/doc/en/gentoo-amd64-faq.xml#coolnquiet

http://www.ndl.kiev.ua/content/linux-kernel-undervolting-patch-amd-processors

 

 

Ok, thanks.  This is something to do after getting this format bug fix release out 

Sorry if this is a 'duh' kind of question, but I am going to upgrade to 4.5.4 to get support for a card I just purchased.  I have been running 4.4.2 for a couple years now, and would have happily stayed on that platform, running bubbaq for the rest of the life of my hardware.

 

So, I know that the simple upgrade steps are (obviously) simple.  But, what does bubbaraid add to the mix?  Do I have to remove bubbaraid and then upgrade and add bubbaraid back?  Or do I drop 4.5.4 on top of by bubbaraid/4.4.2 system?

 

Thanks,

Chris

I haven't used it, but the sense I get is Bubbaraid is a complete replacement of unRAID's bzroot/bzimage. If that's the case, then you can not have both. If you add bubbaraid back, you're overwriting unRAID 4.5.4 and will be back down to version 4.4.2.

 

Sorry if this is a 'duh' kind of question, but I am going to upgrade to 4.5.4 to get support for a card I just purchased.  I have been running 4.4.2 for a couple years now, and would have happily stayed on that platform, running bubbaq for the rest of the life of my hardware.

 

So, I know that the simple upgrade steps are (obviously) simple.  But, what does bubbaraid add to the mix?  Do I have to remove bubbaraid and then upgrade and add bubbaraid back?  Or do I drop 4.5.4 on top of by bubbaraid/4.4.2 system?

 

Thanks,

Chris

 

Bubbaraid is compiled for specific vesrions of unraid and there is not a newer version that I'm aware of.  Therefore, you say goodbye to Bubbaraid when do you upgrade.  I was using it as well and then upgraded.  It's simple, just run the "Disable" script on your flash drive to remove/disable it, then do the upgrade by swapping the two files.  I then installed unmenu, which has all the features that Bubbaraid has and many more.  I really enjoyed bubbaraid too, but say goodbye, don't need it anymore.  Joe has done a tone with unmenu and you will really like it, and it's easy to install.

I am running 4.5.6 and am having a strange issue right now.  Last night, I could not access any of my shares, could not access the Web GUI, and could not telnet to my server.  I have a console on the box, but it was not responsive.  So...  I had to cycle the power.  On power-up the server started a parity check and before I called it a day I checked and no errors were found.

 

Now, the next day, I can access my shares but still cannot telnet or access the Web GUI or console.

 

The attached picture is what I see on my console last night and right now.  Might anyone have a clue from this pic what might be going on?  I have no access to the syslog right now.