FormSpring Blog » Maintenance http://blog.formspring.com Tips and information for using web forms Thu, 11 Mar 2010 19:51:10 +0000 http://wordpress.org/?v=2.9.1 en hourly 1 Service Update http://blog.formspring.com/2010/02/13/service-update-4/ http://blog.formspring.com/2010/02/13/service-update-4/#comments Sat, 13 Feb 2010 12:37:12 +0000 chris http://blog.formspring.com/?p=1968

FormSpring experienced an outage this morning for 9 minutes from approximately 6:33 AM to 6:42 AM EST.  We experienced no data loss.  Our main database server experienced a RAID failure, cutting off traffic to our main site and our form builder. The server technicians at our hosting company were able to quickly rebuild and restore our RAID array.  As of 6:42 AM EST all forms are back up and operating normally.

We apologize for the outage and know you rely on your forms for a wide variety of business purposes . We are working with our hosting company to see what the exact problem was and why the failure occurred. We are also working with them to see if there are any precautions that we can take to ensure that this does not happen again.

If you have any questions or would like more information about a particular issue, please contact our support team and we will answer any questions.Recent Posts:

Other Related Posts

]]> http://blog.formspring.com/2010/02/13/service-update-4/feed/ 0 Service Update http://blog.formspring.com/2010/01/21/service-update-3/ http://blog.formspring.com/2010/01/21/service-update-3/#comments Thu, 21 Jan 2010 14:10:00 +0000 Erica http://blog.formspring.com/?p=1899

FormSpring experienced an outage this morning for 33 minutes from approximately 5:38 AM to 6:11 AM EST.  We experienced no data loss.  Our load balancers went down, cutting off traffic, so forms could not be built, viewed or submitted.  As of 6:11 AM EST all forms are back up and operating normally.

We are waiting on more info from our data center as to what exactly happened to cause the outage.  We will update this post when we have more to report.

Recent Posts:

Other Interesting Posts

]]> http://blog.formspring.com/2010/01/21/service-update-3/feed/ 2 Service Update http://blog.formspring.com/2009/10/21/service-update-2/ http://blog.formspring.com/2009/10/21/service-update-2/#comments Wed, 21 Oct 2009 13:28:13 +0000 ryan http://blog.formspring.com/?p=1756

FormSpring experienced an outage this morning from approximately 7:48 AM to 8:05 AM EST.  We experienced no data loss.  Our load balancers went down, cutting off traffic, so forms could not be built, viewed or submitted.  As of 8:05 AM EST all forms are back up and operating normally.

We are waiting on more info from our data center, Peer 1, as to what exactly happened to cause the outage.  We will update this post when we have more to report.Recent Posts:

Other Interesting Posts

]]> http://blog.formspring.com/2009/10/21/service-update-2/feed/ 0 Out of Office Friday, July 3rd http://blog.formspring.com/2009/07/02/out-of-office-friday-july-3rd/ http://blog.formspring.com/2009/07/02/out-of-office-friday-july-3rd/#comments Thu, 02 Jul 2009 13:07:39 +0000 ade http://blog.formspring.com/?p=1334

Just a quick head’s up that the FormSpring office will be closed on Friday, July 3rd as we celebrate Independence Day.

If you have any questions, please send us a message or leave a voicemail at 800.845.6697.  We’ll try to respond as soon as possible Monday morning.  Have a great weekend!Recent Posts:

Other Interesting Posts

]]> http://blog.formspring.com/2009/07/02/out-of-office-friday-july-3rd/feed/ 0 June 21st Maintenance Notice http://blog.formspring.com/2009/06/18/june-21st-maintenance-notice/ http://blog.formspring.com/2009/06/18/june-21st-maintenance-notice/#comments Thu, 18 Jun 2009 15:07:10 +0000 chris http://blog.formspring.com/?p=1297

We will be migrating the FormSpring site and application to a new hosting platform on Sunday, June 21st from 1:00am – 2:00am Eastern time. During this window we expect less than 5 minutes of actual downtime. With any downtime a maintenance message will be displayed when viewing a form or using the application.

Some Background

We’ve been on our current cluster of servers for a little over a year, but as we’ve continued to grow we see a need to preemptively expand the hosting architecture to accommodate future growth and employ additional high-availability technology.

A few weeks ago we attempted a migration that resulted in unexpected performance issues leading to a two-hour outage, ultimately causing us to revert back to our old servers. We’ve since started fresh with a new data center, new servers, and very different infrastructure.

We feel very confident that this migration will be a success. We’ve assaulted the servers with a battery of new tests to try to pinpoint any potential performance issues and have found the new infrastructure gives us a minimum of 3x more capacity than we have today.

What do I need to do?

In short, nothing. The migration will be very much behind-the-scenes, and you shouldn’t notice anything different in the new environment.

However, there are a couple things you should look out for. These issues are only applicable to a few people so if none of this seems familiar, you can safely ignore this.

  • If you’ve hard-coded FormSpring’s IP addresses in your computer’s hosts file or a router, please remove these entries. It’s important to use our normal DNS servers to obtain IP addresses in real-time.
  • Similarly, if you’ve white-listed FormSpring’s IP addresses in your mail server, please remove these entries and white-list messages coming from any formspring.com server. Messages may come from one of several servers whose IP addresses are subject to change without notice!

Please feel free to contact us if you have any questions about the migration. We will be updating our blog and Twitter stream once the migration is complete or with any additional information.

UPDATE 6/21 1:30am – The maintenance went smoothly and we’re up on the new environment.  If you experience any issues please file a support request.Recent Posts:

Other Interesting Posts

]]> http://blog.formspring.com/2009/06/18/june-21st-maintenance-notice/feed/ 0 Service Update http://blog.formspring.com/2009/05/26/service-update/ http://blog.formspring.com/2009/05/26/service-update/#comments Tue, 26 May 2009 20:03:14 +0000 chris http://blog.formspring.com/?p=1241

We wanted to provide a full update on the service problems that FormSpring experienced today.  On Tuesday, May 26, 2009 we experienced a number of outages over the course of approximately 2 hours that led to downtime of the FormSpring site, affecting your ability to build, view or submit forms. The first outage happened at 10:39am EST and intermittent outages or slowness continued until 12:42pm EST.

What happened

The root of the outage was our migration to a new hosting environment that took place shortly before 2am. The migration itself went smoothly and there was barely a blip of downtime as we made the switch. The new environment had been running in parallel with the old for over a month, and we had run it through a barrage of tests and trial runs to help ensure that the migration would go unnoticed. The migration was one of many steps we had planned to expand FormSpring’s underlying server architecture.

We watched the new environment closely after the migration, and around 10:30am started to see elevated levels of CPU usage on the primary web server. While the level of traffic on the site wasn’t above norms, it appeared that the new environment reached a tipping point where each new connection to the site slowed down the server and new connections queued up behind others causing the server to become even slower. In just a few minutes the site went from normal to slow beyond usable.

We started working with our hosting provider to troubleshoot the issue. Over the course of the next hour we tried a combination of configuration tweaks, failed over to a backup web server, and increased the capacity of the primary web server from 2 CPUs to 4, all without much success of solving the problem. We also shut down the admin side of the application so that the servers could focus on just serving forms and collecting submissions. We finally failed back to the original hosting environment, and continue to run there right now.

Status of form submissions

Fortunately, we did not see any data loss during the outages. Our primary database server continued to operate as expected, and we had multiple backups on hand in case of data corruption. However, users likely would have had trouble viewing or submitting forms, depending on the time of submission. In that case they would have seen either an error message from their browser or a FormSpring error message indicating that the site was down or a connection was lost. If that happened then the submission was never saved to the database and the form will need to be re-submitted.

Going forward

We’re fully back on the old environment and can continue to run there for weeks or months if necessary. However, we do feel it’s important to eventually continue to expand our architecture and are assessing our original plans for doing so.

The first step is to work with engineers at our hosting provider to figure out what happened with the new environment and either find a fix or start from scratch elsewhere. It’s apparent that our tests did not sufficiently simulate real world usage. Various benchmarks and load testing showed performance improvements of 50%-100% per server, but this number was obviously misleading. While it’s notoriously hard to simulate real world web traffic, we’ll be trying some new things to help make sure that we’re not surprised again by performance problems.

We know that you depend on us for many of your business transactions and we want to ensure you that we take this very seriously. We’ll keep you posted and let you know what our next steps will be.  If you would like a refund or would like to discuss any specific problems you incurred please contact our support team and we will work with you to resolve any problems promptly.  We thank everyone for being patient with us, whether following our updates here on the blog, or via Twitter, and appreciate the support as we worked to get everything working properly.Recent Posts:

Other Interesting Posts

]]> http://blog.formspring.com/2009/05/26/service-update/feed/ 0 Current Technical Problems http://blog.formspring.com/2009/05/26/current-technical-problems/ http://blog.formspring.com/2009/05/26/current-technical-problems/#comments Tue, 26 May 2009 15:41:59 +0000 chris http://blog.formspring.com/?p=1236

Update 12:40 (Eastern) You should be able to access your forms and the admin area. We reverted back to our old servers,  if you are having issues accessing the site use this IP Address http://66.135.60.102/. There might be a slight delay in everyone accessing the site due to DNS issues

Update 11:50 AM (Eastern): We have failed over to our back up servers. Your forms should be viewable by your customers/users but the admin area is still unavailable. We are working to get that up and should have more updates shortly.

We moved our servers over the weekend and are currently having some downtime issues. We are working quickly to resolve the problem and get your forms up and running. We know this is a huge inconvenience for you but we hope to have an update very soon.

Thanks for your patience as we get this all sorted out.Recent Posts:

Other Interesting Posts

]]> http://blog.formspring.com/2009/05/26/current-technical-problems/feed/ 3 FormSpring Outages http://blog.formspring.com/2008/01/16/formspring-outages/ http://blog.formspring.com/2008/01/16/formspring-outages/#comments Wed, 16 Jan 2008 19:57:29 +0000 chris http://www.formspring.com/blog/2008/01/16/formspring-outages/

We have experienced some outages today. You can read about them in the Forums and post any questions or comments you might have.

We are up and running and are looking at ways to solve the problem to make sure that they don’t happen again.Recent Posts:

Other Interesting Posts

]]> http://blog.formspring.com/2008/01/16/formspring-outages/feed/ 0 Maintenance window Sunday 11pm EST http://blog.formspring.com/2007/07/27/maintenance-window-sunday-11pm-est/ http://blog.formspring.com/2007/07/27/maintenance-window-sunday-11pm-est/#comments Sat, 28 Jul 2007 02:05:01 +0000 ade http://www.formspring.com/blog/2007/07/27/maintenance-window-sunday-11pm-est/

We’ll be working on the server on Sunday July 29th between 11pm EST and 11:30pm EST in order to upgrade to the new version we’ve been talking about for a couple weeks now.

There will be some downtime during the upgrade, but we’ll work hard at keeping it to a minimum. Forms will be inaccessible for about 10-15 minutes during that maintenance window, while the administration tool will be down for about the full 30 minutes.

Thanks for your patience. You can check back here for updates or contact us for more information.Recent Posts:

Other Interesting Posts

]]> http://blog.formspring.com/2007/07/27/maintenance-window-sunday-11pm-est/feed/ 0 Downtime http://blog.formspring.com/2007/04/03/downtime/ http://blog.formspring.com/2007/04/03/downtime/#comments Tue, 03 Apr 2007 18:17:23 +0000 admin http://www.formspring.com/blog/2007/04/03/downtime/

The upgrades went smoothly last night with < 5 minutes of total downtime.  However, we just experienced an unrelated problem with the network at our data center which left the site down this afternoon for about an hour roughly between 12:30pm and 1:45pm EST.  The source of the problem is unknown, but we’re working on trying to trace the problem and help ensure that it won’t happen again.

We’re terribly sorry for the problems this causes for you.  If you’ve been significantly affected by this issue, please contact us.

Check back here for updates on the situation.Recent Posts:

Other Interesting Posts

]]> http://blog.formspring.com/2007/04/03/downtime/feed/ 0