Suggestions to keep Beta releases from messing up production servers

[lionelhutz]

Just adding this to try and not clutter the release thread.

 

Peter

[prostuff1]

Not to say the obvious but...

 

DON'T USE A BETA ON A PRODUCTION ARRAY!!

 

 

I understand that not that many people have test servers and it may limit the testers but seriously... it is playing with fire.

[SSD]

Not to say the obvious but...

 

DON'T USE A BETA ON A PRODUCTION ARRAY!!

 

I understand that not that many people have test servers and it may limit the testers but seriously... it is playing with fire.

 

If users don't test out betas, they won't get tested.

 

The first round of testers need to be those with test arrays.  A beta should only be used on test arrays while it is going through the initial volley of defect reports (like what we've seen in the 5.0 beta series - b3, b4, b5, b5a, b5b, b6, ...).  We have not reached anything stable yet, so all production arrays should be on 4.7.

 

But once the first testers are satisfied and working well, I believe that EXPERIENCED UNRAID USERS to whom the specific features of the new release are especially valuable (for 5.0b3+, I'd say that's the Apple users), might consider trying to upgrade.  They will be taking a risk, and no one is holding a gun to their heads to upgrade, but some users are willing to take a calculated risk with a beta that is running well on a number of test arrays.  If no one did this, we'd never get a good test, and production releases would be buggy.

 

After the early adopters, other users, maybe not as experienced or more caustious, would need to assess for themselves if/when they want to upgrade.  Again, for users that have no specific need for the functionality, they would wait for a release version, and even then take their time.

 

The problems we have is with inexperienced users with little or no understanding of what can go wrong or even what to do if something does go wrong, decide to upgrade based on the feature list, without reading and comprehending the announcements thread,and within a day of a new beta hitting the forum.  These users are living on the bleeding edge and don't even know it.  It is that population we need to target to not uset betas.

 

[PeterB]

If users don't test out betas, they won't get tested.

 

I agree - without expending a great deal of effort, no one running on a test array is going to expose bugs like a live user will.

 

The first round of testers need to be those with test arrays.  A beta should only be used on test arrays while it is going through the initial volley of defect reports (like what we've seen in the 5.0 beta series - b3, b4, b5, b5a, b5b, b6, ...).  We have not reached anything stable yet, so all production arrays should be on 4.7.

 

I know that is the eminently sensible advice, but I think that it is for each one of us, as individuals, to make their own decision.  Of course, we all need to be aware of the risks involved, and make our judgement individually.  This judgement will be based on reports from other users, together with some knowledge of what has been changed/fixed since the previous (beta?) release, but also taking into account the value we place on our own data and our own capability of dealing with the problems which (may) arise.

 

However, I can see where Tom may be coming from - despite all disclaimers, I'm sure that he will feel some guilt if anyone was to suffer a serious loss of data, and may feel compelled to put in extra effort in order to assist in/prevent such a situation.

 

The problems we have is with inexperienced users with little or no understanding of what can go wrong or even what to do if something does go wrong, decide to upgrade based on the feature list, without reading and comprehending the announcements thread,and within a day of a new beta hitting the forum.  These users are living on the bleeding edge and don't even know it.  It is that population we need to target to not uset betas.

 

Exactly!

[bubbaQ]

My feelings are that general public users should not test betas, but only release candidates.  I like the idea of closed or at least semi-closed betas.

 

Betas are for testing functionality of new features, and catching common bugs.

 

A R/C is useful for getting that wide variety of hardware into the testing/feedback loop, but most common bugs should be caught in the betas, and not survive to a R/C.

[graywolf]

I do not know if it is possible. But could a prompt be added to the betas which would require input (Yes) to proceed...kinda like the preclear.

And have a big disclaimer on it.  You could lose your data?

 

I know it would be a pain for those running headless...but I guess, how many run TEST arrays headless??

 

Since I don't have a Test Array, I keep to the non-betas for that very reason (unRAID and other stuff) no matter what the new bells/whistles/features in the betas.

 

Or another option would be posting a Disclaimer making users type in aknowledgment regarding the dangers/pitfalls of betas.

 

Although common sense would be not running betas on production...but there is a big lack of common sense in the world. Thats why you have warnings on plastic bags, do not put over your head, suffocation may occur. 

[aaronwt]

I do not know if it is possible. But could a prompt be added to the betas which would require input (Yes) to proceed...kinda like the preclear.

And have a big disclaimer on it.  You could lose your data?

 

I know it would be a pain for those running headless...but I guess, how many run TEST arrays headless??

 

Since I don't have a Test Array, I keep to the non-betas for that very reason (unRAID and other stuff) no matter what the new bells/whistles/features in the betas.

 

Or another option would be posting a Disclaimer making users type in aknowledgment regarding the dangers/pitfalls of betas.

 

Although common sense would be not running betas on production...but there is a big lack of common sense in the world. Thats why you have warnings on plastic bags, do not put over your head, suffocation may occur. 

 

It's a Beta. That should be understood. It's up to the individual user whether to run a beta and they use at their own risk. It's not a final product. People aren't babies. So no need to treat them like babies.

[SSD]

It's a Beta. That should be understood. It's up to the individual user whether to run a beta and they use at their own risk. It's not a final product. People aren't babies. So no need to treat them like babies.

 

I think it slightly funny that you post here.  You have 6 posts (including this one), have used unRAID for 1 month, and never had a problem with unRAID.  You fit the critieria of the type of user that would:

 

- decide to load a beta,

- have problems,

- try to fix it yourself,

- have a 30 post thread where users are trying to help you understand what happened / why what you did in response was the wrong thing / and what to do now to try to salvage your data

- have a weeklong recovery extravaganza, fully editorialzed in your thread, complaining all the while that no one told them that they could lose data with the beta.  

 

Forgive me for picking on you, because I don't know you and this may very well not be what would happen to you.  But there are some new users, LIKE you, that will have this experience.  The question is, how to stop it.

[PeterB]

People aren't babies. So no need to treat them like babies.

Unfortunately, I'm not convinced that this is always true!

[S80_UK]

I would also fall into the "new here" group.  As it happens I test beta software in my day job (embedded software in consumer audio devices) so I know that betas can seriously break stuff.

 

I would favour the "semi-closed" beta group approach.  You have to sign up, but the acceptance criteria should not be too strict in order to allow a useful pool of testers.  One of the conditions must of course be to tick one or more boxes to absolve Lime Technology of any liability for anything (hardware, data, software) with appropriate warnings - almost  along the lines of "It's a beta - some of you will lose your data".

[SSD]

My feelings are that general public users should not test betas, but only release candidates.  I like the idea of closed or at least semi-closed betas.

 

Betas are for testing functionality of new features, and catching common bugs.

 

A R/C is useful for getting that wide variety of hardware into the testing/feedback loop, but most common bugs should be caught in the betas, and not survive to a R/C.

 

This would work if every beta resulted in a release. That's not the way Tom works.  There were at least 3 or 4 planned betas for 5.0 - each with new functionality - which could easily take 6 months to fully complete before a release candidate comes out.  Having real users wait until the end of that to test any of it seems like not a good thing.

 

There aren't enough users with test arrays standing by with the time and discipline to do the testing necessary to verify a beta or release.  Only Tom's shop does that (and very well, I might add.  Over the years, the number of serious defects have been very low IMO.)  I tried to create a "UAT" of sorts for 4.7 that I think was helpful in running through a variety of scenarios.  It found zero defects.  It was not until some user with an HPA that the REAL bug in 4.7 became apparent - and that was after release.  Lock down the betas and that will become even more common IMO.  Like it or not, the "ingenious idiot" is needed to really help prove a version is stable.  And that means real users - and lots of them.

[Joe L.]

  Like it or not, the "ingenious idiot" is needed to really help prove a version is stable.  And that means real users - and lots of them.

That is very true. 

 

It apparently is only the user with an HPA that subsequently removed it that hit the bug in 5.0bet6. (at least that is Tom's theory right now)

 

It is absolutely impossible to anyone to duplicate all the hardware/software configurations out there.  The pool of beta testers must be large.

[BRiT]

It apparently is only the user with an HPA that subsequently removed it that hit the bug in 5.0bet6. (at least that is Tom's theory right now)

 

If it's his theory on the MBR not recognized in 5.0 beta 6, then the HPA theory holds no water. I never had HPA on any of my drives and have never removed it on any of the drives.

[Rajahal]

It seems to me that the pool of closed-beta tested would be quite large.  Maybe I'm wrong about that.  I intend to find out with this poll.

[StevenD]

I have to agree with a closed beta.  You experts spent way too much time helping people fix problems when they installed a beta on their production server.

[BRiT]

I have to agree with a closed beta.  You experts spent way too much time helping people fix problems when they installed a beta on their production server.

 

We also spend way too much time helping people fix problems when they installed a stable release. 

[Stokkes]

I agree with one of two methods already proposed

 

1) Closed beta, invite only

2) Semi-closed, ability to sign yourself up after a clear disclaimer, maybe a password protected site or something.

 

I see WAY too many posts from people who install a beta and get upset because things don't work, wiped their data, messed up their array or just generally behaved unexpectedly. Quite a few of these posts are also inflammatory.

 

The comment earlier about people being babies is true. Anyone who's worked in any type of support environment can attest to this. Sorry, but this is the cold hard truth.

 

I think there are quite a few unRAIDers that would be more than willing to be guinea pigs for the greater cause (myself included), but I think it's currently way too easy to install a beta.. Why?

 

1) Posts announcing the new betas are available in the same forum as stable releases

2) Download links for the betas appear on the same page as the stable releases

3) Forums are easily accessible.

 

I think all 3 of these need to be looked at going forward.

 

My $0.02.

[graywolf]

It's a Beta. That should be understood. It's up to the individual user whether to run a beta and they use at their own risk. It's not a final product. People aren't babies. So no need to treat them like babies.

 

Agree - should be up to the individual user

 

Disagree - think a lot (not all) people are babies. Why else would we have so many warnings on packages that should be common sense?  

Do not touch Iron, you could get burned.  

Do not put plastic bag over head, you could suffocate.

Warning: Never iron clothes on the body. (iron)

Warning: Contents may be hot. (coffee)

Never use a lit match or open flame to check fuel level

 

Need I say more?

[Stokkes]

It's a Beta. That should be understood. It's up to the individual user whether to run a beta and they use at their own risk. It's not a final product. People aren't babies. So no need to treat them like babies.

 

Agree - should be up to the individual user

 

Disagree - think a lot (not all) people are babies. Why else would we have so many warnings on packages that should be common sense?  

Do not touch Iron, you could get burned.  

Do not put plastic bag over head, you could suffocate.

Warning: Never iron clothes on the body. (iron)

Warning: Contents may be hot. (coffee)

Never use a lit match or open flame to check fuel level

 

Need I say more?

 

And yet McDonald's has always had "Caution: Hot" and someone in the USA still managed to successfully sue after having burned themselves.

 

Do not underestimate the will of the stupid.

 

[jakebake]

I have to agree with a closed beta.  You experts spent way too much time helping people fix problems when they installed a beta on their production server.

 

We also spend way too much time helping people fix problems when they installed a stable release.  

 

true. But we appreciate it! haha. I have never used linux before, ever! (except for stuff that uses it without my knowledge) So it is a litte tough to figure out any problems without help from experience users (even with the wikis).

[aaronwt]

It's a Beta. That should be understood. It's up to the individual user whether to run a beta and they use at their own risk. It's not a final product. People aren't babies. So no need to treat them like babies.

 

I think it slightly funny that you post here.  You have 6 posts (including this one), have used unRAID for 1 month, and never had a problem with unRAID.  You fit the critieria of the type of user that would:

 

- decide to load a beta,

- have problems,

- try to fix it yourself,

- have a 30 post thread where users are trying to help you understand what happened / why what you did in response was the wrong thing / and what to do now to try to salvage your data

- have a weeklong recovery extravaganza, fully editorialzed in your thread, complaining all the while that no one told them that they could lose data with the beta.  

 

Forgive me for picking on you, because I don't know you and this may very well not be what would happen to you.  But there are some new users, LIKE you, that will have this experience.  The question is, how to stop it.

 

I just set up the unRAID box last week. I already have around seventy terabytes of storage available on my home network and with my unRAId box I will add another fifteen or so with my initial build of thirteen 1.5TB drives. I would never install beta software for my stored content since it is too valuable to me and don't want to risk problems. I want to be able to use it and not have to mess with anything Just like my current boxes run with no issues like my WHS and other NAS devices.

 

If I did use the beta I would take one of my boxes gathering dust and use that to test the bea software. At least it would give me something to do with the dozens of sub 1TB drives I have lying around.

[Rajahal]

That's the sensible approach, but unfortunately not everyone takes it.

[talmania]

Curious:  same aaronwt from AVS?

[nia]

1) Posts announcing the new betas are available in the same forum as stable releases

2) Download links for the betas appear on the same page as the stable releases

3) Forums are easily accessible.

I sort of agree with this. Being a Linux ignorant I know enough not to be part of first or second wave - I do plenty bleeding edge testing with software where I know how to rescue myself...

 

However, as as way to

a) protect the kamikaze users who install while drowning themselves in hot coffee with a plastic bag over their head (!)

and

b) still enabling the large group of sensible users to be educated; not least by following the evolvement and behaviour of a new version, troubleshooting process, workarounds etc.

 

I suggest:

 

• Keep the test forum completely open for all to read (and even write)
  
• Keep the download behind three big warning screens and an e-mail activation link (at the least) or if so decided only accessible by invitation
  

That was my 2 cents...

[aaronwt]

Curious:  same aaronwt from AVS?

yes