Jump to content
fc0712

Let's complain about the LT release process

138 posts in this topic Last Reply

Recommended Posts

How much time does a A4 Blogpost cost? (with pictures) I dont know. 30 min? 1 Hour? a week? Its not much. But it gives much to the community. And it feels more like "somethings happening" xD

Edited by nuhll

Share this post


Link to post
3 hours ago, bonienl said:

 

Nothing of this is really set in stone and some companies use complete different names or strategies for their own purposes.

 

Microsoft ditched the terms "beta" and "rc" altogether, they are now using the terms "insider preview" and  "preview" releases, while a company like ZDnet makes use of "perpetual betas" meaning all their releases are always "beta". Confusing, unless you dive into what they really mean with their release strategy!

 

Professionally I work a lot with Cisco equipment. Their release terminology is completely different. Does that make them wrong? Don't think so, but it requires understanding of what they do.

 

If Limetech feels comfortable with their release strategy, so be it. Just my 2 cents.

 

Read your post again and this time consider what I actually wrote about the meaning of terms.

 

You notice that Microsoft invented own terms - they aren't abusing established terms.

And you seem to imply the same about Cisco.

 

When a user sees something is a "Preview", that user knows that he/she has to read up on what might be meant with that term.

When a user sees  RC, then the user will have an expectation about the meaning of the term.

 

It isn't about what the company feels comfortable with. I would be most comfortable with selling things with no warranty and no support, but that is obviously not a good strategy.

  • Upvote 2

Share this post


Link to post

If LT have their own meaning behind release cycles, they should go ahead and define them. Hijacking established definitions and saying "in our world, up is down" is like trolling.

 

13 hours ago, bonienl said:

Now we come with a 3 wheels transport, and yet it can be called either a car or a motorcycle.

But most often called three-wheeler (car|motorcycle) for obvious reasons.

 

11 hours ago, bonienl said:

I am pretty close to the development cycle, and honesty this never happened. There is enough "internal" debate about what goes into a RC.

Honestly, there are bugs in the latest "stable" that exist in the newest "RC" and thrive from "new feature" components that have not been tested enough. To me it looks like LT is trying to look busy by releasing software as often as possible, no matter the quality.

 

11 hours ago, bonienl said:

This is something we had in the past, but turned out not a good idea. The few independent GUI updates which were released came haunting us down, because people do upgrades in the most unexpected ways. It is much safer to offer a complete upgrade package.

This is where definitions for incompatibility come in place. For this to be successful, there has to be a compatibility window which is defined by breaking changes in the core or other components of the OS. To my knowledge and experience, this is a very good idea.

Edited by realies
  • Upvote 1

Share this post


Link to post
On 3/11/2018 at 11:46 PM, realies said:

To me it looks like LT is trying to look busy by releasing software as often as possible, no matter the quality.

 

I don't think this is the reason. LT don't want to release low quality - it's just an outcome [see note below] of a somewhat broken release process and the size of the company.


LT is a small company and can't have a really large set of own hardware to test on - and can't have a really large set of different customization variants. So LT is depending on lots of people to install beta versions to test. It's just that LT seems to have made the decision that they will get more people to test their beta versions by calling them RC. And I think it's the wrong way to try to get more beta testers by misrepresentation of the term "release candidate".


I'm pretty sure LT would get enough testers even if using the term "beta" - lots of people wants to be a step ahead of everyone else with inside knowledge of new functionality and are dying to test beta software.


The problem with pushing out beta versions that are labeled RC is that there is no room for any true RC at the end of the release cycle for some extended long-time testing (and strict bug-fixing) before final release. The end outcome is that the x.y.0 version must end up becoming the actual release candidate since it's the only new labeling available. And careful generals then have to wait a week or two and possibly pick up x.y.1 or x.y.2.

 

In my book, it's wrong that the users should need to know that:

- "beta" should be read as "alpha"

- "rc" should be read as "beta"

- x.y.0 should be read as "rc"

 

 

 

 

Edit:

Note that "it's an outcome" in my above quote is not relating to "low quality". Because I don't think the releases have low quality.

"it's an outcome" is a reference back to the claim that LT tries to look busy by having many releases.

 

Many releases happens because of multiple ongoing updates and a need for additional people to test on a larger set of installations.


If it would be possible to break down the system into less coupled modules that could be sent out for testing individually then the number of full system releases could be fewer. As of now it's the full package and that gives a need for many test releases.

 

But to break it down into individually testable features would require that all individual parts of unRAID are individually installable, version-controlled, packages complete with dependency information. Slack isn't the best OS to start with. But another issue is the boot-from-flash strategy which means it would take some creative overlay logic to boot from a standard release on flash and then overlay with one or more updated modules at boot time.

 

This is the reason why there can be many quick releases. Not to look busy but because it's much easier to drop off a full build than to get a version-controlled plugin overlay system working well. So the other route is to see if creative use of git branches could help holding back some work-in-progress for later releases. I don't mind many updates but would prefer if a sequence of updates focused on a single feature and then progressed to one or more releases for a different feature. That's what I have been mentioning earlier about more agile deployment. If one feature reaches stable level before the next feature is sent out for test then it's easier to pick a version in the middle of the sometimes very long band of intermediary releases.

Edited by pwm
Clarification of a sentence

Share this post


Link to post

@realies @pwm First let me start by saying that the implications that we are "just trying to look busy" and that we deliberately are trying to "fool" people by using versioning "trickery" is pure rubbish, and I would appreciate and in fact, insist, that you go back and edit your post to strike out those statements.  The only thing you got right was indeed we are a small company without the resources to test a wide range of hardware and configurations for an extremely complex set of subsystems.  However we never publicly release any software where we know we have introduced risk of data loss, except in rare cases where we clearly state the risk in release notes.

 

As for use of "RC" designation, have you ever studied the linux kernel release methodology?  I know your answer is No or else you would know that the "RC" series of a linux kernel is precisely for shaking out bugs and those releases are definitely not ready for any kind of production use.  The choice made to promote a kernel from "RC" to "mainline" is also fairly arbitrary.  Go take a look at any kernel change log and you will see all kinds of crazy bugs fixed by later patch releases.  They typically don't even mark security-related patches as being security-related (because they don't want to give hints to would-be hackers, placing unpatched users at risk).  Unless we are forced to, we also don't use the the ".0" release of a new kernel either because we know there will be wider usage with inevitable patch releases (up to 26 now for 4.14).  We know from monitoring downloads that most of our user community doesn't update to a .0 unRaid release immediately either.  This is the nature of software development and you guys should know better.

 

As for introducing "new features" beyond -rc1 - ok you got me there!  Usually we have one or two truly "new" features which require significant coding in each minor release, and usually most of that coding is in place by -rc1.  Sometimes big chunks might be staged over several RC's.  Sometimes it might take a few RC's to get the UI completely right or handle unexpected cases.  Meanwhile, other developers are contributing code.  I'm not going to tell those people, "Sorry don't give us those changes because we are still testing this other unrelated feature."  Sometimes those features themselves might extend the RC series beyond what we wanted, but that's the price we are willing to pay for now.  In the case of the linux kernel, there are thousands of developers which requires far more discipline and rigor to maintain sanity (hence the "merge window" concept).  In contrast, we are a small team with good communication (usually) and don't need to be so pedantic.

 

Finally, this stuff is hard.  Every user thinks their feature request or bug fix should be the highest priority.  Every user wants to see fast progress.  Every user can find faults and bugs in any software release.  And we have to be as responsive as possible to survive as a company.  So how about you cut us a break?

 

  • Like 10

Share this post


Link to post

Yes, it is hard since you aren't just releasing an application but a complete OS. And it doesn't get easier since you aren't released for a fixed hardware but for more or less arbitrary 64-bit x86-compatible hardware.


But wouldn't it still be better to use the "beta" term and reserve "rc" for software in lockdown state? I still think you would get enough testers.


Beta, by definition, is code where you know you haven't implemented everything. So if you are only able to release a partial update, then that can by definition never be a release candidate. A release candidate contains all changes you think you need for the final release - that's what defines it as a release candidate. Anything else is an intermediate release that could be alpha or beta depending on if it is a rough preview or if it is more or less usable.

 

It's enough that you switch to some arbitrary suffix like "t1", "t2", ... to avoid the issue with rc and the implied meaning of release candidates.

 

When you relate to the Linux kernel, the problem is that it is hard to get every single driver to be in 100% sync release-wise with the rest of the OS. And there is also often lots of functionality released as experimental.

  • Upvote 1

Share this post


Link to post
2 minutes ago, pwm said:

Yes, it is hard since you aren't just releasing an application but a complete OS. And it doesn't get easier since you aren't released for a fixed hardware but for more or less arbitrary 64-bit x86-compatible hardware.


But wouldn't it still be better to use the "beta" term and reserve "rc" for software in lockdown state? I still think you would get enough testers.


Beta, by definition, is code where you know you haven't implemented everything. So if you are only able to release a partial update, then that can by definition never be a release candidate. A release candidate contains all changes you think you need for the final release - that's what defines it as a release candidate. Anything else is an intermediate release that could be alpha or beta depending on if it is a rough preview or if it is more or less usable.

 

It's enough that you switch to some arbitrary suffix like "t1", "t2", ... to avoid the issue with rc and the implied meaning of release candidates.

 

When you relate to the Linux kernel, the problem is that it is hard to get every single driver to be in 100% sync release-wise with the rest of the OS. And there is also often lots of functionality released as experimental.

 

I appreciate your constructive criticism.  Ok maybe we will go back to -beta/-rc designations but here's an example of why that's pretty arbitrary in our case.  In last -rc I outlined use of a new VM Manager config setting called "Upon host shutdown" either hibernate or shutdown VM's.  Ok this is a new feature.  However, it was added because we have been working on VM Manager which necessitates updating the host unRAID OS quite often.  Well this is a pain in the neck for me to fully shutdown all my windows VM's each time I need to test, so what I do is manually patch /etc/rc.d/rc.libvirt to hibernate my VM's instead of shutting them down.  This makes the reboot/test cycle much faster.  But I also got tired of patching that file all the time, and this is a useful feature in cases of power fail where a UPS can trigger a shutdown.  Hence I asked @eschultz to add the setting and a bit of code in rc.libvirt.  So what? Am I to omit this feature or "hide" it because we're at -rc5?  I think not.

Share this post


Link to post

Not sure if "So what?" is an argument or a question. If latter, experimental changes can go within an appropriate branch in the version control environment.

Share this post


Link to post
2 minutes ago, realies said:

Not sure if "So what?" is an argument or a question. If latter, experimental changes can go within an appropriate branch in the version control environment.

 

"experimental" branches tend to proliferate quickly and pretty soon you're not testing with the actual product you're intending to ship.

Share this post


Link to post
50 minutes ago, pwm said:

Beta, by definition, is code where you know you haven't implemented everything. So if you are only able to release a partial update, then that can by definition never be a release candidate.

 

What we are trying to do is generate releases such that any -rc release can be immediately promoted to stable if need be.  For example, the full set of VM Manager changes and features we are working on will take several more weeks to fully implement.  But we will stage these features over several minor/patch releases instead of holding up the entire release, which may have other changes that are necessary to publish (such as security updates).

  • Upvote 1

Share this post


Link to post
2 hours ago, limetech said:

@realies @pwm First let me start by saying that the implications that we are "just trying to look busy" and that we deliberately are trying to "fool" people by using versioning "trickery" is pure rubbish, and I would appreciate and in fact, insist, that you go back and edit your post to strike out those statements.  The only thing you got right was indeed we are a small company without the resources to test a wide range of hardware and configurations for an extremely complex set of subsystems.  However we never publicly release any software where we know we have introduced risk of data loss, except in rare cases where we clearly state the risk in release notes.

I'm very content with the RC releases, incremental releases have been fairly well tested and stable and i have not seen any breaking issues so far.
Normally i'm not a guy that runs release candidates on systems that involve data, since recovery and restoring takes to much of my precious time.
But the quality of the releases in the candidate pipeline are like u said stable, and that's why i haven't missed a release candidate release on unRAID.

We often add new features in our release candidates as well, so i don't mind that either. Mostly when it's needed or demanded by our costumers.
Would the releases go the fully "beta" state, i would certainly not run it. Setting up a extra machine for testing would not be worth the effort for me.


For the size of your company your doing a amazing job, supporting the range of platforms and trying to solve issues on certain hardware platforms.
Surely in my opinion not deserving some of the comments that i have read on the forum, for most because of the complexity of the (sub)systems.


Maybe publishing a issue tracker / roadmap / backlog dashboard would give more insight to people where the development is going towards or what you are working on.
I'm not sure the forums are the right tool for that. But on the other hand managing a system like that would give a lot of extra overhead for a small company like yours.
So i would understand that if it's not a option to maintain.

 


 

Edited by SiNtEnEl
  • Upvote 2

Share this post


Link to post

I equally am happy with the way LT label their RCs i.e. each is good enough to run and the naming convention conveys this viewpoint, and why the releases are not called betas. 

 

Like @SiNtEnEl I wouldn't entrust my whole network (my unRAID server controls literally everything in my household - media, tv, lighting, home security, heating, network, firewall, wifi etc etc), and thanks to VMs and dockers, my only PC - if the releases weren't rock solid - I would stay well away.  I appreciate the 'continuous delivery' approach rather than big infrequent releases.

 

If I was a moderator, I would love to be able to close this thread as it's probably the only negative thread in what's a very positive community.  So what if the naming convention doesn't match other products?  No one is going to get hurt by installing a RC, because 9/10 they work - if you're not sure, then don't download or read the bloody release thread first!

 

41 minutes ago, SiNtEnEl said:

Maybe publishing a issue tracker / roadmap / backlog dashboard would give more insight to people where the development is going towards or what you are working on.
I'm not sure the forums are the right tool for that. But on the other hand managing a system like that would give a lot of extra overhead for a small company like yours.
So i would understand that if it's not a option to maintain.

 


 

I would love to see some sort of tracker or a better way to provide limetech with feedback on what features the community really need e.g. like this https://feathub.com/Radarr/Radarr.  Random 'feature request' posts just don't cut it for me, and provides no feedback on what's been added to the backlog or not

Edited by DZMM
  • Upvote 1

Share this post


Link to post

My opinion is that this discussion has been had ad infinitum over the years, we all know by now that LT's definition of RC is perhaps different to what most people would expect.  Just accept it.

 

In LT's defence though, yes things might break, or there may be new bugs to elements, but AFAIK there's never been a bug with core NAS functionality or one that has caused data loss.

 

So, if an upgrade breaks the VM or docker subsystem, just downgrade again, easy as pie.  New functionality has been developed by @bonienl to make this easy.

  • Like 3
  • Upvote 1

Share this post


Link to post
On 3/12/2018 at 6:22 AM, pwm said:

Edit:

Note that "it's an outcome" in my above quote is not relating to "low quality". Because I don't think the releases have low quality.

"it's an outcome" is a reference back to the claim that LT tries to look busy by having many releases.

 

Many releases happens because of multiple ongoing updates and a need for additional people to test on a larger set of installations.


If it would be possible to break down the system into less coupled modules that could be sent out for testing individually then the number of full system releases could be fewer. As of now it's the full package and that gives a need for many test releases.

 

But to break it down into individually testable features would require that all individual parts of unRAID are individually installable, version-controlled, packages complete with dependency information. Slack isn't the best OS to start with. But another issue is the boot-from-flash strategy which means it would take some creative overlay logic to boot from a standard release on flash and then overlay with one or more updated modules at boot time.

 

This is the reason why there can be many quick releases. Not to look busy but because it's much easier to drop off a full build than to get a version-controlled plugin overlay system working well. So the other route is to see if creative use of git branches could help holding back some work-in-progress for later releases. I don't mind many updates but would prefer if a sequence of updates focused on a single feature and then progressed to one or more releases for a different feature. That's what I have been mentioning earlier about more agile deployment. If one feature reaches stable level before the next feature is sent out for test then it's easier to pick a version in the middle of the sometimes very long band of intermediary releases.

 

I updated an earlier post this morning but didn't had the time to finish before I had to run for a business meeting.

 

I think much of the issues in this thread could be handled by switching to a classical tripple-branched distribution model.

  • testing
  • rc
  • stable

 

Testing

x.y.z-beta<num>

Testing would contain all the new functionality.

Always invite only - experienced users who have been running unRAID for a reasonable amount of time and/or shown good tech skills.

The majority of issues could be found here and probably cut in half the number of support threads under general support - for most issues, the affected user are likely to be able to give good feedback and possibly even suggested workarounds/fixes.

Any testing release could contain one or more concurrently developed new features. Actual usability (guestimated readiness level) as described in the release thread.

 

Release Candidate

x.y.z-rc<num>

Individual features from Testing that are felt to be release-ready.

Also potential CVE that needs to be rushed out.

RC branch available to all unRAID users - probably also people running on test license.

Most incompatibility issues from old configuration cruft etc catched during the testing stage.

Any issues in RC branch individually decided if they will be fixed "inline" or if the fix will need to wait for a turn back to the Testing branch.

 

Stable

x.y.z

Only functionality that has been 1-2 weeks with good outcome as RC, or critical CVE that needs to be rushed out.

 

 

Assuming good use of Git, the above should not cost any significant difference in administrative costs.
But to be practical, it requires that all functional changes are handled as individual feature branches in the source code repository so that the individual feature branches can be copied from Testing -> RC -> Stable when appropriate.

 

It should also give a significant protection for inexperienced users - especially since it's quite common that people update without reading release notes and it's also quite common that the general support threads suggests that people should try the latest "RC" - in this case it would be true release candidates the users would be suggested to test.

 

Having the individual changes as feature branches makes it easy to add/remove a feature change depending on test result or quick-rush a change without being too affected by concurrent work-in-progress changes.

 

It would not really matter much if there drops out a new RC every week and possibly even a Stable release every week - because every single release is most likely good enough for production use.

 

The user would not have to spend time trying to figure out if they have a release candidate or some intermediate release because the RC and Stable branches would not contain any half-finished intermediate releases and no test code.

 

I'm pretty sure that more than enough users would want an invite to Testing that there would be large enough pool of testers to catch the majority of issues before pushing a release candidate.

  • Upvote 1

Share this post


Link to post

wait, are we still still talking about this???

Share this post


Link to post
49 minutes ago, pwm said:

cut in half the number of support threads under general support

I think most support threads are not due to issues with the product, they are due to people not knowing how to use the product.

Share this post


Link to post
Just now, trurl said:

I think most support threads are not due to issues with the product, they are due to people not knowing how to use the product.

Yes, of course. And the majority of data losses are caused by people making wild assumptions. Humans are almost always the weakest link. Which is why a number of users have to be protected from themselves. Even the significant percentage that refuses to read the release notes.

 

Having a firewall between inexperienced users and new features will give a bit of additional time to find the sharp edges.

 

What I would like is for the update procedure to include a pre-install script that the older unRAID version can download and run before performing the actual update. The Testing branch would have caught a number of people with their cache partition incorrectly aligned, issues with incompatible plugins etc while testing with more experienced users. There would then be time to have prepare a pre-install script that might refuse to update because some of the pre-conditions hasn't been fulfilled. And the pre-install script could pop up a message with important guidelines for the user to accept before updating.

  • Upvote 1

Share this post


Link to post
42 minutes ago, pwm said:

pre-install script that the older unRAID version can download and run before performing the actual update.

There is a discussion happening elsewhere regarding this that leverages the CA/FCP datafiles that are continually updated.

Share this post


Link to post
1 minute ago, Squid said:

There is a discussion happening elsewhere regarding this that leverages the CA/FCP datafiles that are continually updated.

Great.

Share this post


Link to post
On 12.3.2018 at 8:12 AM, limetech said:

@realies @pwm First let me start by saying that the implications that we are "just trying to look busy" and that we deliberately are trying to "fool" people by using versioning "trickery" is pure rubbish, and I would appreciate and in fact, insist, that you go back and edit your post to strike out those statements.  The only thing you got right was indeed we are a small company without the resources to test a wide range of hardware and configurations for an extremely complex set of subsystems.  However we never publicly release any software where we know we have introduced risk of data loss, except in rare cases where we clearly state the risk in release notes.

 

As for use of "RC" designation, have you ever studied the linux kernel release methodology?  I know your answer is No or else you would know that the "RC" series of a linux kernel is precisely for shaking out bugs and those releases are definitely not ready for any kind of production use.  The choice made to promote a kernel from "RC" to "mainline" is also fairly arbitrary.  Go take a look at any kernel change log and you will see all kinds of crazy bugs fixed by later patch releases.  They typically don't even mark security-related patches as being security-related (because they don't want to give hints to would-be hackers, placing unpatched users at risk).  Unless we are forced to, we also don't use the the ".0" release of a new kernel either because we know there will be wider usage with inevitable patch releases (up to 26 now for 4.14).  We know from monitoring downloads that most of our user community doesn't update to a .0 unRaid release immediately either.  This is the nature of software development and you guys should know better.

 

As for introducing "new features" beyond -rc1 - ok you got me there!  Usually we have one or two truly "new" features which require significant coding in each minor release, and usually most of that coding is in place by -rc1.  Sometimes big chunks might be staged over several RC's.  Sometimes it might take a few RC's to get the UI completely right or handle unexpected cases.  Meanwhile, other developers are contributing code.  I'm not going to tell those people, "Sorry don't give us those changes because we are still testing this other unrelated feature."  Sometimes those features themselves might extend the RC series beyond what we wanted, but that's the price we are willing to pay for now.  In the case of the linux kernel, there are thousands of developers which requires far more discipline and rigor to maintain sanity (hence the "merge window" concept).  In contrast, we are a small team with good communication (usually) and don't need to be so pedantic.

 

Finally, this stuff is hard.  Every user thinks their feature request or bug fix should be the highest priority.  Every user wants to see fast progress.  Every user can find faults and bugs in any software release.  And we have to be as responsive as possible to survive as a company.  So how about you cut us a break?

 

You should really invest your time, instead for such posts, in a weekly blog post about what you do, what you plan to do, about whats on your mind. Mix it with some pictures and you havee more time for real coding :) ... you get more visitors.... your clients know what you are doin... like the guys from factorio do since YEARS.

Edited by nuhll
  • Like 1

Share this post


Link to post
On 3/16/2018 at 3:35 PM, nuhll said:

You should really invest your time, instead for such posts, in a weekly blog post about what you do, what you plan to do, about whats on your mind. Mix it with some pictures and you havee more time for real coding :) ... you get more visitors.... your clients know what you are doin... like the guys from factorio do since YEARS.

 

I can't agree more, what is the direction of unraid, what features have the team in mind, what will be included in 6.6, etc. the community is a bit blind in this aspect, a monthly post in the blog would be enough. I just purchased unraid and I would like to be better informed.

 

In addition a platform like this: https://ideas.sophos.com/forums/330219-xg-firewall/filters/top

for unraid would be quite interesting as well to keep the community more engaged.

Share this post


Link to post

Why should LT do a weekly post telling us what they're up to?

Because Factorio do?

Microsoft doesn't.

Let's be realistic here, LT is a small team, they get a lot done with the team they have, and for those of you that haven't been around as long as some of us, the release tempo has increased dramatically.

Do I necessarily agree with everything LT do? No.

Do I feel the need for this discussion to keep going on? No

I do not buy into the argument that LT informing the community would cut support requests that's just not the case, of all the posts I've answered I can't recall ever thinking "If only Tom/Eric/Jon had written a weekly blog post, we'd have avoided this issue."

LT is a business not a community cooperative nor opensource collaboration.

Sent from my LG-H815 using Tapatalk

  • Upvote 1

Share this post


Link to post

I don't see the need for monthly/weekly/periodic blogposts. They don't serve any real purpose.

 

I see the need for an overall product development roadmap that's kept fairly up to date.

Share this post


Link to post
1 hour ago, CHBMB said:

I do not buy into the argument that LT informing the community would cut support requests that's just not the case, of all the posts I've answered I can't recall ever thinking "If only Tom/Eric/Jon had written a weekly blog post, we'd have avoided this issue."

I'm not sure anyone in this thread has claimed that weekly blog posts would cut support requests.

 

What I have been saying is that a different release model would cut support requests by catching lots of the sharp corners before the beginners gets access to new functionality.

  • Upvote 1

Share this post


Link to post
37 minutes ago, pwm said:

 

What I have been saying is that a different release model would cut support requests by catching lots of the sharp corners before the beginners gets access to new functionality.

 

Sorry, I clearly got some posts mixed up, you're quite right.

 

I'm still not sure it would catch the sharp corners, imho, the biggest problem people run into is the fact Unraid is an immensely complex os, with a deceptively easy webui.  This has had the effect of lowering the bar to accessibility, people are running Unraid who would probably never dream of running a comparable headless Ubuntu server for arguments sake, not a bad thing in and of itself, but does bring some issues. 

 

Roll back to v4.7 days, when there was just basic NAS functionality and far less options, you'd have less issues.  Add KVM, Docker and a huge increase in the number of plugins and as one might predict the number of support requests have gone through the roof.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now