A couple of weeks ago, an article was posted on medium describing how an empty S3 bucket ran up a massive AWS bill in less than two days.
In the article, the author describes creating an empty S3 bucket in the eu-west-1 region and uploading some files there for testing. Two days later, he checked the billing page to find over 100 million put requests and a bill of over $1300 dollars.
The article… kinda went viral and a bunch of folks in the AWS community were pretty pissed that something like this could happen.
But why would a private S3 bucket that nobody should know about in the first place start receiving millions of requests almost overnight? The answer is a bit of an interesting one.
Remember that s3 bucket names are globally unique meaning that there can only be a single instance of a bucket with a specific name. It turns out that the author of the article created an S3 bucket with a name that was being used under the default configuration of a popular open source tool. This means as soon as he created the bucket, every instance of this open source tool was trying to store files in it.
Of course, the requests lacked the right permissions and were rightfully rejected by S3. Herein lies the interesting bit that was the root cause of the massive bill. It turns out that S3 charges you even when the request fails with an Unauthorized or AccessDenied error. To make this even worse, you don’t even need an AWS account to attempt to write a file to a random bucket. Clearly something aint right.
The article got a lot of attention by the AWS higher ups including a well known Vice President Jeff Bar. A day after the article was posted, he acknowledged that customers should not have to pay for unauthorized requests and that the S3 team would be working on a fix.
On March 14th, AWS released a blog post stating that unauthorized requests that customers did not initiate would be free of charge. This means that bucket owners will never receive request of bandwidth charges for requests that return a 403 or Access Denied error response code that originated outside of their AWS account or organization.
I for one was delighted to see the quick turnaround from AWS and the S3 team to resolve this issue quickly.
Over to you – a) what do you think of being charged for requests to private buckets? and b) AWS’ response to the fiasco.
