So, what was that when the forums were broken?

Recommended Posts

Hi Y’all - I’m primarily responsible for the outage today, and I’ll give you a rundown on what happened and why it happened: 


So, right now the forums run on a VM, which has worked pretty well for a long time, but has caused some pain points around upgrades. In order to begin moving the forums to a system that allows easier scaling, upgrades, and rollbacks, we needed to move uploads and other content that is stored on the VM to a storage provider than can scale outside of just a disk.


IPS provides tools to change the storage location for files, so we figured that we could just swap over the storage to S3 and it would migrate. We tested this on our staging environment to great success, so we figured it would be fine to swap the main server over. 

Over the past couple days, IPS moved all the content that has been uploaded over the last 10+ years onto S3 and it went off without a hitch… At least until we tried to move the theme files. When this occurred, it failed to move the vast majority of those files, and the files it did move were not properly created. After reverting this, IPS moved most of the files back, but began to move the files between the two storage servers seemingly randomly - causing the first outage this morning. 

We then swapped the setting again to store on S3, which seemed to work great at first, but then IPS started throwing errors for missing JavaScript - turns out their mover is pretty insufficient when your forum has been running for as long as ours, and we were forced to move files manually to patch the missing JavaScript.


We think that we finally caught all of the missing JS / CSS files, and we do see this improving performance by quite a bit, especially under load - due to offloading of static file serving to S3. 

Let me know if you have any other questions, and sorry about the delays. We’re addressing years of technical debt doing this kind of maintenance so these things can be tricky, so we really appreciate the understanding.


TLDR: IPS has some issues and our forums have been around for a long time. 

  • Thanks 2
  • Upvote 3
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.