limetech Posted June 30, 2013 Share Posted June 30, 2013 Our web host provider, vps.net, suffered multiple power failures and backup generator failures. That's their story and they're sticking with it. Looks like the hunt for a new hosting service is back on (but vps.net website is so slick and modern looking, they must be good - right). Here's their status log: Saturday, June 29, 2013 at 8:09 PM Atlanta Zone D and E Outage Sunday, June 30, 2013 at 12:35 AM (EDT) 16:35 GMT The portable backup generator has arrived and Cloud Servers are being brought back online now. Sunday, June 30, 2013 at 11:20 AM (EDT) 15:20 GMT Power for this datacenter should be restored soon. The utilty company has Electricians on-site to fix the outage, and a portable backup generator is being delivered soon to provide power to the entire datacenter. Extra VPS.NET Engineers will continue to be available today so we restore our services to operation as quickly as possible. Sunday, June 30, 2013 at 10:00 AM (EDT) 14:00 GMT Both the Atlanta-D and Atlanta-E cloud are offline once again now due to a power outage, and the backup power generator did not work correctly. Sunday, June 30, 2013 at 9:31 AM (EDT) 13:31 GMT Our Engineers have completed the repairs for Atlanta-D and are now starting up cloud servers. Sunday, June 30, 2013 at 7:59 AM (EDT) 11:59 GMT This remaining outage for Atlanta-D is also affecting many websites on our Cloud Sites service in Atlanta. Sunday, June 30, 2013 at 7:01 AM (EDT) 11:02 GMT The storage repairs are almost completed now. VPS.NET Engineers are continuing to complete all the needed work to startup the affected cloud servers. Sunday, June 30, 2013 at 6:36 AM (EDT) 10:36 GMT There have been some delays with the repairs in Atlanta-D, but we are still working to get it online as quickly as possible. Sunday, June 30, 2013 at 4:36 AM(EDT) 08:37 GMT VPS.NET Engineers are continuing to complete repairs to get Atlanta-D available again soon. Sunday, June 30, 2013 at 3:34 AM (EDT) / 07:34 GMT Most servers in Atlanta-E are now online. Cloud servers in Atlanta-D are not yet started due to a storage issue that we are working to resolve now. Sunday, June 30, 2013 at 2:51 AM (EDT) / 06:51 GMT We are beginning to startup cloud servers in both clouds: Atlanta-D and also Atlanta-E. Sunday, June 30, 2013 at 2:21 AM (EDT) / 06:21 GMT Our network engineers have successfully mitigated the hardware failure caused by the power outage in Atlanta. We are now verifying the integrity of the remaining hardware, we will update this post as we progress.” Sunday, June 30, 2013 at 1:21 AM (EDT) / 05:21 GMT The power outage in Atlanta has caused a component failure in one of our network switches, our engineers are working on mitigating this failure. We will update this post as we progress. Sunday, June 30, 2013 12:40 AM (EDT) :We are continuing to power up and verify integrity of our hardware. We will update this post as we progress. Saturday, June 29, 2013 11:40 PM (EDT) : Power has been restored and we are working on getting our equipment back online. Saturday, June 29, 2013 11:18 PM (EDT) : Our datacenter provider in Atlanta suffered a power outage and subsequent failure of their backup generator which has affected Atlanta Zones E and D. We are working with them on restoring power to our equipment, once that is done we can begin the process of bringing VPS’s online. As always we will continue to update this post with more information as it becomes available Saturday, June 29, 2013 11:00 PM (EDT) : Our engineers are still working with datacenter staff to resolve the issues affecting Atlanta Zones D and E. Thank you for your patience while we resolve this issue. Saturday, June 29, 2013 10:00 PM (EDT) : Our engineers are still working with datacenter staff to resolve the issues affecting Atlanta Zones D and E. We will update this page with more info as it becomes available. Saturday, June 29, 2013 8:09 PM (EDT) : The network connection is down at the data center. Our engineers are working to get this resolved as soon as possible with the data center. We do not have an estimated time of resolution. Please check here for updates. close Sunday, June 30, 2013 at 12:33 PM Atlanta Cloud Sites outage Sunday, June 30, 2013 at 12:38 PM (EDT) The portable backup generator has arrived and all Cloud Sites servers are back online. Sunday, June 30, 2013 at 11:29 AM EDT The Atlanta Cloud-site services are within our Atlanta-D cloud which is offline. Please view our other outage update for more details. [...] More Details Saturday, June 29, 2013 at 8:09 PM Atlanta Zone D and E Outage Sunday, June 30, 2013 at 12:35 AM (EDT) 16:35 GMT The portable backup generator has arrived and Cloud Servers are being brought back online now. Sunday, June 30, 2013 at 11:20 AM (EDT) 15:20 GMT Power for this datacenter should be restored soon. The utilty company has Electricians on-site to fix [...] More Details Quote Link to comment
garycase Posted June 30, 2013 Share Posted June 30, 2013 Hopefully they've learned their lesson and this won't be a recurring issue. It seems they've been fairly reliable except for this extended outage ... although I may have just missed some of the other outages. I wouldn't rush into switching ... you often just exchange one set of problems for another when you do that Quote Link to comment
JackBauer Posted June 30, 2013 Share Posted June 30, 2013 Yeah - and they may have learned their lesson that a "weekly test" of the generator isn't enough to ensure it really is ready to support the full load of the data center. (If a weekly test really was being done) Quote Link to comment
Ice_Black Posted June 30, 2013 Share Posted June 30, 2013 I really recommend you go with linode.com Top service and excellent network! Quote Link to comment
speeding_ant Posted June 30, 2013 Share Posted June 30, 2013 pair.com, great service too. Quote Link to comment
rick.p Posted June 30, 2013 Share Posted June 30, 2013 SquareSpace, you can't kill them. Period. http://www.businessinsider.com/squarespace-data-center-hurricane-sandy-2012-10 Quote Link to comment
Necrotic Posted June 30, 2013 Share Posted June 30, 2013 Yeah - and they may have learned their lesson that a "weekly test" of the generator isn't enough to ensure it really is ready to support the full load of the data center. (If a weekly test really was being done) Somehow I doubt they have control over the generator. They most likely rent space from the datacenter owner, which are also responsible for the generators as part of their sold service. Quote Link to comment
JackBauer Posted June 30, 2013 Share Posted June 30, 2013 Yeah - and they may have learned their lesson that a "weekly test" of the generator isn't enough to ensure it really is ready to support the full load of the data center. (If a weekly test really was being done) Somehow I doubt they have control over the generator. They most likely rent space from the datacenter owner, which are also responsible for the generators as part of their sold service. Well it's more complicated than that - the facility of VPS.net in Atlanta seems to have a lot of data center clients... My point being that the vendor providing the space to all these providers is going to be getting pressure from them to make sure this never recurs. Quote Link to comment
limetech Posted July 1, 2013 Author Share Posted July 1, 2013 Yeah - and they may have learned their lesson that a "weekly test" of the generator isn't enough to ensure it really is ready to support the full load of the data center. (If a weekly test really was being done) Somehow I doubt they have control over the generator. They most likely rent space from the datacenter owner, which are also responsible for the generators as part of their sold service. Well it's more complicated than that - the facility of VPS.net in Atlanta seems to have a lot of data center clients... My point being that the vendor providing the space to all these providers is going to be getting pressure from them to make sure this never recurs. Are you familiar with this data center and/or which other hosting provider is using it? I was looking at a Georgia Electric power outage map and saw very few outages in Atlanta in the last 24 hours. Quote Link to comment
JackBauer Posted July 1, 2013 Share Posted July 1, 2013 http://www.webhostingtalk.com/showthread.php?t=1280424 Quote Link to comment
limetech Posted July 1, 2013 Author Share Posted July 1, 2013 http://www.webhostingtalk.com/showthread.php?t=1280424 Nice, thank you for posting that; I thought I had people on my back, I'd hate to be in those guys' shoes. That guy UNIXy does seem familiar though.... Quote Link to comment
Necrotic Posted July 1, 2013 Share Posted July 1, 2013 http://blog.totalserversolutions.com/press-releases/colo-announces-extensive-infrastructure-expansion-details-for-34-peachtree-facility-in-atlanta/ Interesting that there was an announcement of upgrades 2 days before the outage. I doubt they would've done much though, normally you announce before you actually do the investment to enhance the impact of the marketing. Though I do feel bad for them, these DC consume a lot of power and its very hard to stress test generators without actually doing the power transfer (which exposes the UPS and switches to the extreme loads, lowering the expected life). Then again, some say Murphy was an optimist... Quote Link to comment
NAS Posted July 1, 2013 Share Posted July 1, 2013 Unless you have serious money to spend you cant get 100% uptime from a single provider. Previous track record is a good indicator of performance but it is not definative. Things break. I woul reocmmend getting a second backup web host and using DNSMADEEEASY servcie to flip on outage. You can keep the backup only informational and not have things like the forum on it. This way you dont have the headache of data sync and split brain but you still have an almost 100% internet presence for only an exta $100 bucks PA Quote Link to comment
Pauven Posted July 1, 2013 Share Posted July 1, 2013 I was looking at a Georgia Electric power outage map and saw very few outages in Atlanta in the last 24 hours. Tom, I'm in the Atlanta area and can confirm that we have occasional localized power outages, mostly due to stormy weather knocking out power lines. We have a lot of trees, and a lot of above ground power distribution. Add in lots of rain, winds from hurricane remnants, and surprisingly frequent tornadoes, and downed power lines are all but a given. I used to work in a healthcare data center in Atlanta, and the facility we used was chosen specifically for its dual-homed electricity providers, that way if power was interrupted from one provider, chances were the other provider was still providing electricity to the facility. Even then, I recall once occasion we had to go on generator power after a tornado wreaked havoc. Sorry, I don't have any specific recommendations hosting providers, just thought you may want to add dual-homed power grids to your consideration list, regardless of geographic location. -Paul Quote Link to comment
nars Posted July 1, 2013 Share Posted July 1, 2013 I can tell you that I have a very good experience with 'Leaseweb' for dedicated servers (I do manage some few servers - used as mirrors to a downloads site - hosted with them, on both NL (for 6 years now) and US datacenters (for 2 years now)), so far without any significant problems, other than a problem on a switch on US datacenter sometime ago - quickly sorted - and one or two very small (quickly sorted) network glitches in years... From my experience their service, mainly in Europe, is very good and with good and fast support... not sure though about their vps's. Also note that a big part of European traffic of Wikipedia, as well as some other big sites, seems handled by them. Did also heard good things about 'Pair', from some friends using it to host relatively large sites, but no experience myself. Quote Link to comment
demonmaestro Posted September 8, 2014 Share Posted September 8, 2014 Not to drag up an old topic but instead of paying sub company's for there servers why wont you go and get a server directly from the data center them self? I have used Softlayer(ThePlanet) for a very long time and never had any issues even during hurricane Ike or any issues that has came into Texas for that matter. Quote Link to comment
jonp Posted September 8, 2014 Share Posted September 8, 2014 Not to drag up an old topic but instead of paying sub company's for there servers why wont you go and get a server directly from the data center them self? I have used Softlayer(ThePlanet) for a very long time and never had any issues even during hurricane Ike or any issues that has came into Texas for that matter. Changing our web infrastructure is something on the never-ending list of to-do's that we will have to address after we release 6.0. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.