So, what was that when the forums were broken?

[BRiT]

So, what happened earlier today, where the forums were completely broken? Was it another "nothing was changed" event? 🤣

[JonathanM]

Yeah, about that.

[SpencerJ]

tech debt getting paid 💸

[JonathanM]

Still massively messed up here. Only marginally useable in dark mode, light mode is completely screwed.

@SpencerJ, any timeline for getting this fixed?

[ConnerVT]

At least we can post again.  Earlier I could browse, but was unable to enter text in the Reply box.

[SpencerJ]

25 minutes ago, JonathanM said:

Still massively messed up here. Only marginally useable in dark mode, light mode is completely screwed.

@SpencerJ, any timeline for getting this fixed?

What is still messed up? 
 

cc @elibosley

[JonathanM]

3 minutes ago, SpencerJ said:

What is still messed up? 
 

cc @elibosley

Looks fixed now. When I posted that, the text editor buttons were missing in the dark theme, and the light theme was completely blown out with text overlapping, icons full screen, nothing working.

[elibosley]

Hi Y’all - I’m primarily responsible for the outage today, and I’ll give you a rundown on what happened and why it happened: 

 

So, right now the forums run on a VM, which has worked pretty well for a long time, but has caused some pain points around upgrades. In order to begin moving the forums to a system that allows easier scaling, upgrades, and rollbacks, we needed to move uploads and other content that is stored on the VM to a storage provider than can scale outside of just a disk.

 

IPS provides tools to change the storage location for files, so we figured that we could just swap over the storage to S3 and it would migrate. We tested this on our staging environment to great success, so we figured it would be fine to swap the main server over. 
 

Over the past couple days, IPS moved all the content that has been uploaded over the last 10+ years onto S3 and it went off without a hitch… At least until we tried to move the theme files. When this occurred, it failed to move the vast majority of those files, and the files it did move were not properly created. After reverting this, IPS moved most of the files back, but began to move the files between the two storage servers seemingly randomly - causing the first outage this morning. 
 

We then swapped the setting again to store on S3, which seemed to work great at first, but then IPS started throwing errors for missing JavaScript - turns out their mover is pretty insufficient when your forum has been running for as long as ours, and we were forced to move files manually to patch the missing JavaScript.

 

We think that we finally caught all of the missing JS / CSS files, and we do see this improving performance by quite a bit, especially under load - due to offloading of static file serving to S3. 
 

Let me know if you have any other questions, and sorry about the delays. We’re addressing years of technical debt doing this kind of maintenance so these things can be tricky, so we really appreciate the understanding.

 

TLDR: IPS has some issues and our forums have been around for a long time. 

[SpencerJ]

Thanks Eli. 
 

For those wondering, IPS = Invision community. 
 

Have a great weekend all

[JonathanM]

2 hours ago, elibosley said:

I’m primarily responsible for the outage today

30 lashes with a wet noodle!🤣

 

Thanks for getting us moved to the next step in the seemingly explosive growth of our little community!