I think that when unRAID is simulating a disk, it should be read only.  Attempts by the user to write to the disk should fail. 

 

unRAID has no official way to notify a user that they have a "red ball" disk in their array, and can therefore be under the illusion that all is well.  Allowing writes to the simulated disk continues the "all's well" illusion.  I cannot think of many situations under which a user would knowingly want to write to a simulated disk.  Instead, they should be trying to get the disk replaced / rebuilt, to restore parity protection.  By returning write errors to the user it will immediately communicate that something is wrong.  And since drives are taken out of service by write errors, it seems likely that the user was writing to the disk at the time, making this a pretty effective way of communicating the failed disk condition. 

 

By not writing to the failed disk, the "trust my parity" procedure will also work better (data written to a simulated disk will be lost if "trust my parity" is used).

I'll second that.  Even if this was only an option.  I would enable it myself.

I strongly agree.  Making it optional would be nice too, but defaulting to Fail Writes.  The only time it is valid to allow writes is if you already KNOW the drive is bad and has to be rebuilt on another, and that is almost never true.  At the very least, you want to test the drive thoroughly first before writing it off, to determine what, if anything, is wrong with it.  The chances are relatively high that it was just a cabling or other issue, unrelated to the physical drive itself.

I disagree.

 

Suppose you are logging realtime data to the disk -- you will lose data. I have all my systems remote logging to unRAID.

 

unRAID is a "fault tolerant" system which means when a fault occurs, it tolerates it and lets operations continue, perhaps at degraded performance, but not degraded features.

That is valid.  So perhaps global and per-disk options?

I'm with BubbaQ on this.

 

I would rather the time being spent using beep or an email notify within unRAID.

One the features missing from unRAID that other NAS environments have is basic email notification.

 

I'm not talking about the addon's option.

I think basic email notification should just be part of the unRAID environment itself.

 

 

I don't know if I agree... If anything, it could be configurable, but not default.  I would NOT want that behavior on my system.     I have processes running on other PCs on the LAN that use the unRAID server to log their output.  I would not want them to fail if an unRAID disk fails.  I would want them to continue, even if only writing to the "simulated" disk.

 

The reasoning behind your request of a read-only-simulated-drive is so the user of the array would notice the failure.  For many, reliance on the file-system being read-only would not bring them to realize something is wrong... in fact, many people access their array media through read-only shares.   If I relied only on my media-players, I'd never notice an that a "simulated" disk is read-only until I go to save a new file to the array.

 

Instead, I think focus should be put on making it possible to easily notice the array is in a non-standard state.

First, now that we have the ability to "beep" the speaker in the server, make it come to life in a failure situation.  Make it beep every few seconds, until silenced by a new button on the management interface.

 

Second... add the ability to send an e-mail alert.  If not that, add the ability to send udp messages on the LAN alerting us of the array status.  There are several PC clients that will work well to pop up and alert us of any status changes.  

 

Third... drastically change the appearance (color scheme and warning message) when the array status is abnormal.  A big RED banner at the top of the management interface might work.  A big message stating the abnormal status could be used for the text.  

 

Joe L.

I don't know if I agree... If anything, it could be configurable, but not default.  I would NOT want that behavior on my system.     I have processes running on other PCs on the LAN that use the unRAID server to log their output.  I would not want them to fail if an unRAID disk fails.  I would want them to continue, even if only writing to the "simulated" disk.

 

This same sentiment is echoed by several in this thread.  Basically it says that the data being written to the array is too important to fail.  I think this is a valid argument - but only for a very small percentage of users.  But for the majority of users, the data written to the array is media data.  And if the write fails, the source will be kept intact, and no data will be lost.

 

To support everyone, a configurable option would make sense if this were implemented.

 

The reasoning behind your request of a read-only-simulated-drive is so the user of the array would notice the failure.  For many, reliance on the file-system being read-only would not bring them to realize something is wrong... in fact, many people access their array media through read-only shares.   If I relied only on my media-players, I'd never notice an that a "simulated" disk is read-only until I go to save a new file to the array.

 

A simulated disk is a very fragile thing.  Say you have a 13 disk array, and one of the disks fails (turns red).  You now need 12 disks to stay functional for the 13th to be simulated.  If one of the 12 were to fail, the 13th would be lost as well.  The simulated disk is 12x as likely to fail as a normal disk.  Said another way, running the array for 1 month in that condition has as much risk of data loss on that drive as running a normal disk for 1 year.  Up that to 20 disks available now in unRAID, and you see that running in simulated mode for any period of time can be risky.

 

It is true that this would only be noticable when a write is attempted to the drive, and that may not be that common.  An alternative would be to put the entire array in read-only mode.  THAT would get some attention.  There could be a button to press on the Web GUI to put it back in read/write mode.  Although I'm not saying this is a great solution, at least it would get the user's attention - and probably pretty fast too!

 

Instead, I think focus should be put on making it possible to easily notice the array is in a non-standard state.

First, now that we have the ability to "beep" the speaker in the server, make it come to life in a failure situation.  Make it beep every few seconds, until silenced by a new button on the management interface.

 

This may be a good solution for some.  Personally, with a server in an unfinished basement, it could be days before I would notice a beeping server.  Some systems have no speaker, making them mute.  This is a good idea and a partial solution.

 

Second... add the ability to send an e-mail alert.  If not that, add the ability to send udp messages on the LAN alerting us of the array status.  There are several PC clients that will work well to pop up and alert us of any status changes.  

 

Both good ideas.  But dependent on external technology.  What if the internet is down - or if it fails in the middle of the night when no workstations are available to get the udp message.  I'd really like to see more of a product solution.  Plus, emails are somewhat easy to ignore.  unRAID needs to make the user want to FIX the problem.  Getting one email from the server may not be enough.

 

Third... drastically change the appearance (color scheme and warning message) when the array status is abnormal.  A big RED banner at the top of the management interface might work.  A big message stating the abnormal status could be used for the text.  

 

A great idea!  Still requires users look at the console interface.  But with the current interface it is so easy to either miss the red ball, or see the red ball and not understand its meaning.  Maybe more of a problem than noticing the red ball is knowing how serious a problem it is when the array seems to be working perfectly.

 

I don't have a perfect solution - but do think there needs to be raised awareness of a red ball situation.

What if the internet is down - or if it fails in the middle of the night when no workstations are available to get the udp message.  I'd really like to see more of a product solution.  Plus, emails are somewhat easy to ignore.  unRAID needs to make the user want to FIX the problem.  Getting one email from the server may not be enough.

 

And I can make the same arguments about the "read-only" process... such as if it is in the middle of the night, you aren't writing to the drive and won't notice.....and what if the failed drive contains media only, and is one that is "full" enough that you will never write to it again so as ti "see" this notification?

 

Beep and broadcast message every 2 minutes, with a 12-hour reset button in the UI.  Send an e-mail and a cell-phone text message every hour (multiple addresses).

Both sides of this argument have merit and I understand why.

 

I like the fault tolerance of unRAID in that it simulates the drive and keeps going. That can be invaluable for some people… but for most I think that making the server read-only would be enough.  The problem I see with this though is for the future of unRAID.  If Tom is looking to allow "Third Party Apps" and extensions then we might have a problem with some things.  Case in point: BubbaRaid; I have rtorrent and wtorrent up and running nearly 24x7 and I would not want the array to go to read only if that disk failed, it could completely fubar the download (granted not a huge deal, but still).

 

Some sort of configurable options for this would be great and most probably a necessity.  Some will want the server to keep going, others will want it to go into read-only, while others may (in an extreme case) want it to perform a clean powerdown.

 

I think that ANY and ALL forms of notification need to be worked into unRAID for version 5.  Most other NAS options have some for of e-mail notification.  Also, something that could hopefully be integrated is a heads up type popup (ala Growl).  There is a Growl for Windows (also see here) and Growl for OS X.  Particularly the part about being able to push a notification via javascript from a web browser seems almost perfect for our application.  Being able to display a heads up on the users desktop from the web interface without it being open would be great, in my opinion.  I use growl on my Mac for almost everything and it works great. Obviously all of this would require the user to install growl on there computer which they may not want but… those people can figure something else out.

 

The FEMUR plugin for Firefox is great and I use it in all of my browsers.  It allows me to see the max temp of any given drive and gives a nice brief little overview of the server.

 

That’s just my 2 cents.

What if the internet is down - or if it fails in the middle of the night when no workstations are available to get the udp message.  I'd really like to see more of a product solution.  Plus, emails are somewhat easy to ignore.  unRAID needs to make the user want to FIX the problem.  Getting one email from the server may not be enough.

 

And I can make the same arguments about the "read-only" process... such as if it is in the middle of the night, you aren't writing to the drive and won't notice.....and what if the failed drive contains media only, and is one that is "full" enough that you will never write to it again so as ti "see" this notification?

 

Beep and broadcast message every 2 minutes, with a 12-hour reset button in the UI.  Send an e-mail and a cell-phone text message every hour (multiple addresses).

 

I concur, I want my server to continue, not limit it's ability.

 

Also FWIW, There is the ability to send SMB messages via smbclient.

I also think the FEMUR application is a great way to monitor or get an alert if something is wrong.

 

There is a bit of a chicken and egg situation with some of the suggestions.  Setting up third party or addon solutions require motivation and effort.  Having Tom integrate something into standard product is needed to help people the FIRST time they have a problem, not just the second.

 

Email notifications are an excellent idea, but subject to some uncertainty.  Email addresses can change.  Internet connectivity can be lost.  Server names, IP addresses, and even security needed to send automated emails can change.  So not a perfect solution, certainly a good one.

 

The "squealing server" (beeping continuously until someone notices) is also a good idea.  It requires a functioning speaker (not everyone has this).  Some users have their systems in out-of-the-way locations and hearing the beeping would not be immediate.  But unless the server is in a bomb shelter or something, it will be heard sooner or later (better than today).  I guess the other "problem" might be those that have the server in their bedroom and might not be to happy about the server suddenly starting to beep at 2 in the morning.

 

After considering the ideas expressed here, the read-only simulated disk may not be terribly effective.  It requires that someone write to the disk to be triggered, and as pointed out writes can be pretty infrequent.  Drives do drop off due to write errors but also due to unresponsiveness, with the latter probably the more common occurrence.  I have no issue with the simulated disk being read-only (my writes to the cache disk and are copied to the disk overnight), but now just don't think it would catch enough of the disabled disk situations to be considered a real solution.

 

The idea of making the ENTIRE ARRAY read-only after a disk is disabled is more likely to get noticed.  There could be a button the GUI to make it read/write again.  As a configurable option, this might make sense if several options of notification are made available through unRAID.  I'm sure most everyone responding in this thread would never turn this on, but many users might.

 

Whatever is done, the GUI should make it more obvious that a disk has been disabled and help guide the user to a resolution. 

I think an option for read only on the unaid array is a great idea from the data integrity point of view, but needs to be coupled with email, audible, visible, bug-the-hell-out-of-you until you do something notifications, not as a replacement for such notifications.

 

The matter of interrupting sustained critical writes, like logging, cold be temporarily avoided by a cache drive -- though it requires using user shares, and a large enough cache drive to give you reasonable time to repair the array.  Of course having to write critical data to a cache drive, even when the parity protected array is healthy, may be of concern -- bringing us to mirrored cache drives.

 

I think an option for read only on the unaid array is a great idea from the data integrity point of view....

 

How does switching to readonly add to the integrity of the data?

It doesn't,  it only adds noticeably that there is something wrong.

 

Where do we want limetech to spend it's time.

 

Switching to read only in the hopes that someone will think "hey something is wrong?"

Or

developing a solution that says very clearly... Something is wrong!!!!

 

I would tend to follow the path of other vendors.

GUI notification (switching colors, etc, etc).

Local alerts then Remote alerts.

 

How's about an event handler that is called when a disk goes offline.

Then people can develop their own solution.

 

Maybe the array should just shutdown.

Then there is no way at all it can go unnoticed.

I'm sure the seasoned veterans can do something like this.

 

Where should limetech spend it's time?

 

 

How does switching to readonly add to the integrity of the data?

 

If a drive in an array fails, the less its used the better.  Directing writes to a cache disk, and keeping them from dumping to the degraded array opens up the door for more data recovery options, depending on the type of failure (say, two drives with bad sectors).

 

Where should limetech spend it's time?

 

Here I agree.  Native alerts/notifications, among other items, should be much higher on the priority list than read only for degraded arrays.  Still, I think there is at least some merit to it.  Perhaps "great" was too string of a word, but please read the rest of the post.

 

How does switching to readonly add to the integrity of the data?

 

If a drive in an array fails, the less its used the better.  Directing writes to a cache disk, and keeping them from dumping to the degraded array opens up the door for more data recovery options, depending on the type of failure (say, two drives with bad sectors).

 

You have not really saved much except some I/O, when in fact early warning and preparation is the best scenario.

If a drive fails. know about it and replace it as soon as possible.

 

This could open up the argument, rebuild on a warm or hot spare automatically.

 

 

You have not really saved much except some I/O, when in fact early warning and preparation is the best scenario.

If a drive fails. know about it and replace it as soon as possible.

 

That's sort of the whole point. 

 

This could open up the argument, rebuild on a warm or hot spare automatically.

 

This has come up a few times, and there seemed general agreement that hot spares (automatic rebuild) was not a good idea.  (A user should really look at syslogs and smart reports to figure out if a rebuild should occur.) 

 

Warm spares are better, putting the user in control.  I prefer a cold spare.  But there are no special features needed to support warm or cold spares.

This image illustrates the type of pro-active alert that is possible to send from the unRAID server to client processes running on PCs on your network.

 

I'm running "WhoIsIt" as a client on my PC.  It is one developed as an alternative to a standard "YAC" client.  It responds to the same messages from the LAN.

 

I used netcat in the following example  to send the message to my PC, but it is as easy to use "bash" or "awk" as they both have built in capability to send a message via TCP/IP to port 4550 on my PC.

 

To test, I used this command on my unRAID server:

printf "@CALL\r\nunRAID server needs attention~disk12 has failed" | nc -q 1 -n 192.168.2.10 4550

 

It popped up a semi-transparent notification on my PC that I captured for this example.  It is configurable for size, font, colors, transparency...  When present it is still possible to see and interact with the windows programs on the desktop.   On my HTPC I use a similar alert to show caller ID on the screen in my theater when a call is received.  It very nicely fades in and out on top of the movie.  The duration of the display is configurable. I've got mine set to 15 seconds and made the background color red.  It is impossible to not notice it when it pops up in front of you on the screen.

 

You can download "WhoIsIt21.zip" here: http://forums.snapstream.com/vb/beyond-tv-downloads/23246-whoisit-callerid-display-app.html

 

If there were a "disk-error" event in the new unRAID, we could use the "beep" to draw attention, use the network messaging to send messages every few minutes to the PCs on the LAN.   There is even a YAC client for tivo's so you could see the alert there.

 

All RAID arrays rely on the rapid replacement of a failed drive.  The sooner a failure is noticed, the sooner we can take remedial action. If an alert were sent every 5 minutes or so, it is bound to be noticed eventually.

 

I like this form of notification since no user interaction is needed.  You do not need to open or close the message pop-up.  It does not interfere with the programs on the PC.  Even if you leave the background color white, the large semi-transparent text is still very noticable.

 

The display looks like this on my PC

 

Joe L.

This image illustrates the type of pro-active alert that is possible to send from the unRAID server to client processes running on PCs on your network.

 

I'm running "WhoIsIt" as a client on my PC.  It is one developed as an alternative to a standard "YAC" client.  It responds to the same messages from the LAN.

 

I used netcat in the following example  to send the message to my PC, but it is as easy to use "bash" or "awk" as they both have built in capability to send a message via TCP/IP to port 4550 on my PC.

 

To test, I used this command on my unRAID server:

printf "@CALL\r\nunRAID server needs attention~disk12 has failed" | nc -q 1 -n 192.168.2.10 4550

 

It popped up a semi-transparent notification on my PC that I captured for this example.  It is configurable for size, font, colors, transparency...  When present it is still possible to see and interact with the windows programs on the desktop.   On my HTPC I use a similar alert to show caller ID on the screen in my theater when a call is received.  It very nicely fades in and out on top of the movie.  The duration of the display is configurable. I've got mine set to 15 seconds and made the background color red.  It is impossible to not notice it when it pops up in front of you on the screen.

 

You can download "WhoIsIt21.zip" here: http://forums.snapstream.com/vb/beyond-tv-downloads/23246-whoisit-callerid-display-app.html

 

If there were a "disk-error" event in the new unRAID, we could use the "beep" to draw attention, use the network messaging to send messages every few minutes to the PCs on the LAN.   There is even a YAC client for tivo's so you could see the alert there.

 

All RAID arrays rely on the rapid replacement of a failed drive.  The sooner a failure is noticed, the sooner we can take remedial action. If an alert were sent every 5 minutes or so, it is bound to be noticed eventually.

 

I like this form of notification since no user interaction is needed.  You do not need to open or close the message pop-up.  It does not interfere with the programs on the PC.  Even if you leave the background color white, the large semi-transparent text is still very noticable.

 

The display looks like this on my PC

 

Joe L.

 

There is no doubt that third party tools exist that could be put together into an installable package.  And I think that would be an excellent idea.

 

But I really think that the unRAID product needs to do a better job on this very basic requirements - to communicate that a disk has failed.  The "red ball" on the GUI is just too subtle.

But I really think that the unRAID product needs to do a better job on this very basic requirements - to communicate that a disk has failed.  The "red ball" on the GUI is just too subtle.

We are in complete agreement here...

 

There are too many owners of unRAID servers who post "My server is running perfectly, and I can read and write to all disks,  I "know" that are all working, but one indicator is [glow=red,2,300]red[/glow].   :'( :'( :'( :'(

 

They have no clue they are running with a degraded array, and are not protected from a subsequent concurrent drive failure. The "red ball" on the GUI is way too subtle.

 

Joe L.

You have not really saved much except some I/O, when in fact early warning and preparation is the best scenario.

 

Exactly. However, in a failed array,  not writing to the "good" disks can open up more doors for data recovery should another disk prove to be faulty, in that it keeps a larger portion of the parity data valid relative to the failed disk(s).  The number of cases that this would be needed is admittedly small -- requiring bad sectors on multiple drives simultaneously, yet none of the drives can be completely inaccessible.

 

But, I do not think that read only is a good way to warn of a failed array.  Audible, email and the like are, in my opinion, the best notification options.  You need to be using and writing to the array to notice the read only "notification".  I want my degraded array to track me down and beat me over the head until I fix it.

 

I haven't used it in years, but smbclient can be used to send messages to a Windows PC from Linux over port 139.

You have not really saved much except some I/O, when in fact early warning and preparation is the best scenario.

 

Exactly. However, in a failed array,  not writing to the "good" disks can open up more doors for data recovery should another disk prove to be faulty, in that it keeps a larger portion of the parity data valid relative to the failed disk(s).  The number of cases that this would be needed is admittedly small -- requiring bad sectors on multiple drives simultaneously, yet none of the drives can be completely inaccessible.

 

 

This is not true. It's a false sense of security. If you have a second disk with sector problems. you have bigger issues.

Recovering the failed sectors on a second disk still requires the first failed disk. (i.e. until we have the raid6 solution).

Switching to read only mode only blocks movement of new data to the degraded array.

So any new data would be safer in some queue or cache.

 

Also, bad sectors are re-allocated on write.

Therefore if you are in read-only mode, chances of those sectors being remapped are small unless you do something manually.

 

 

I don't know of any raid array that switches to read only mode because of a partial failure.

The purpose of raid is redundancy.

 

People do have a choice if desired to write a script to turn off their array if a disk fails, then it will surely get noticed.

 

In addition, I would prefer limetech spend time on implementing a RAID6 or hot spare option rather then switching to read only.

 

With limited resources and length of time between upgrades, consider the biggest bang for the buck(time).

But I really think that the unRAID product needs to do a better job on this very basic requirements - to communicate that a disk has failed.  The "red ball" on the GUI is just too subtle.

We are in complete agreement here...

 

There are too many owners of unRAID servers who post "My server is running perfectly, and I can read and write to all disks,  I "know" that are all working, but one indicator is [glow=red,2,300]red[/glow].   :'( :'( :'( :'(

 

They have no clue they are running with a degraded array, and are not protected from a subsequent concurrent drive failure. The "red ball" on the GUI is way too subtle.

 

I reallly think that this is the point.  The product needs a way to tell a user that a drive has failed in a forceful way - without that user being asked to install addons.  The "red ball" in the GUI is way too subtle.

 

I don't know of any raid array that switches to read only mode because of a partial failure.

The purpose of raid is redundancy.

 

People do have a choice if desired to write a script to turn off their array if a disk fails, then it will surely get noticed.

 

In addition, I would prefer limetech spend time on implementing a RAID6 or hot spare option rather then switching to read only.

 

With limited resources and length of time between upgrades, consider the biggest bang for the buck(time).

 

Although there are some features I'd like to see added to unRAID, there are only a few I'd call "very basic" defects that need to be addressed in the base product.  1 - issues in which the base product misleads users (e.g., the restore button), 2 - issues in which the base product does not take adequate precautions to protect user data (e.g., not telling a user when a disk fails).  I think these deserve attention.

 

With the number of disks now possible in an unRAID array, if one fails then the user has to rely on the reliability of up to 19 disks.  That is risky if it is going to take a user days, weeks, or even (for some users) months to notice.  Every 2 days this situation exists is the same risk as a single drive running for more than a month.  And every month the situation exists is the same risk as a single disk running for more than 1.5 years.

 

I don't feel good about the hot spare option.  I won't elaborate here, but such a feature carries its own risks not the mention the need to have an overhead disk as large or larger than the biggest disk installed in the array.

 

Perhaps "read only" is not the right solution, just that this should be treated as a high priority issue within the base product.

Although there are some features I'd like to see added to unRAID, there are only a few I'd call "very basic" defects that need to be addressed in the base product.  1 - issues in which the base product misleads users (e.g., the restore button), 2 - issues in which the base product does not take adequate precautions to protect user data (e.g., not telling a user when a disk fails).  I think these deserve attention.

 

I do agree, there are some potential gotcha's that really need to be resolved.

 

With the number of disks now possible in an unRAID array, if one fails then the user has to rely on the reliability of up to 19 disks.  That is risky if it is going to take a user days, weeks, or even (for some users) months to notice.  Every 2 days this situation exists is the same risk as a single drive running for more than a month.  And every month the situation exists is the same risk as a single disk running for more than 1.5 years.

 

I'm going to say something which is probably going to generate a bit of flame... but if a person has a huge array of disks and some of them older then the warranty or they are junk drives. maybe they are not caring for the data as much.

If a person does have 19 disks and values the data they should have a way of acquiring a disk quickly, or a spare on location.

 

I see people on  AVS forum having these huge multidisk RAID5 arrays.. then spend days on expanding.. when they could loose two drives and loose the whole array.

 

unRAID allows us to loose 2 or more drives and only loose part of the data.

That still doesn't mean you should use drives you do not trust.

 

What the core issue that "Read only" is trying to resolve is notification.

Perhaps a new thread should be started with Suggested ways to improve notification on failed disks.

 

http://lime-technology.com/forum/index.php?topic=3852.msg33893#msg33893

 

 

I don't feel good about the hot spare option.  I won't elaborate here, but such a feature carries its own risks not the mention the need to have an overhead disk as large or larger than the biggest disk installed in the array.

 

Perhaps "read only" is not the right solution, just that this should be treated as a high priority issue within the base product.

 

Many mid based systems have this feature, it's even in the linux kernel for other raid levels.

Did we have a thread on to.. hot spare or not to hot spare???