Amazon Outage Causing Failures (Resolved)

Update at 5:05 PM ET: As promised, an "all clear" update regarding this Amazon S3 downtime event. Amazon has been hard at work at a fix for their S3 problems and although they have not yet announced a fix, our internal monitoring shows our customers having success with all operation types.

Feel free to connect to BrickFTP to perform any operation at this point.

We are still slightly throttling connections on a per-IP basis while the system recovers, but these limits should not affect most normal users and we intend to remove them all entirely as soon as the Amazon S3 system can handle the volume.

With the engineering challenges behind us, we will be "all hands on deck" responding to emails and voicemails so if you had sent us a message, you should expect a response soon.

We will send out a post-mortem to our full list later this week after we've had some time to discuss internally the lessons learned by this event. Hopefully Amazon will complete an incident report as well and we will share all the relevant details with you.

Back to work!

___________________________________________________________________________________________________

Update at 4:12 PM ET: Amazon has been working hard to resolve this issue. At this time, all operations except for uploads are now succeeding reliably. That includes download, list, and delete.

Uploads are still mostly failing at this time and Amazon estimates an additional 1 hour ETA on a fix for uploads.

You are free to continue to try to upload during this time, but we are limiting FTP, SFTP, and WebDAV connections to 10 per IP address at this time so as to not further overload Amazon's systems.

We will let you know once uploads are operating reliably again.

___________________________________________________________________________________________________

The BrickFTP team is aware of an Amazon outage that is causing failures for our BrickFTP customers. As you probably know, BrickFTP currently stores hundreds of terabytes of data, (i.e. hundreds and hundreds of physical disk drives worth) and in order to do that we use Amazon.com as our backend storage vendor.

Amazon is the largest vendor of cloud services in the world, and thousands of popular websites are offline during this outage. They claim to be aware of the issue and to be working on it. Here's a post from Amazon:

We've identified the issue as high error rates with S3 in US-EAST-1, which is also impacting applications and services dependent on S3. We are actively working on remediating the issue.

Unfortunately, we don't have an ETA I can share at this time, however, our full staff is in and working on this. We will mail the full list again once we are operational.

We are doing everything we can, however our system is ultimately built on top of Amazon S3 and there isn't much we can do until they're up.