Well, this is a post that I did not expect to ever be writing, but here it is… Like the good DIY software engineer that I am, I was running my blog on AWS, with S3 for storage behind CloudFront CDN. The total weight of the Jekyll’s _site folder is 3.5Mb and my monthly bill was always below $3. What could go wrong? Cue AWS Billing alert.

Alert

Seeing the email subject line on my mobile phone screen, at first I discarded it, thinking it is an alert for a work account, which I expected to increase a bit this month. The next day, however, rummaging through my inbox I notice it again and with a raised eyebrow, try to log in to see my billing dashboard. I find a month-to-date bill of $75 with a forecast of $111. That’s not an exceptionally large sum for me but still - a stark jump (that grew even more - read on).

My first thought? Someone hacked my account. Although I do have free AWS security alerts setup amongst other things, but this is not infallible. As my eyes wander towards the breakdown by service the CloudFront CDN cost was around $60.

Next thought: did one of my posts get very popular? I thought the “AWS Like Service On boto3” was a neat idea worth a few more eyeballs enjoying it. Am I on HackerNews? A quick check of Google Analytics fizzle out this hope:

My blog is still a regular Joe. No new post = no visitors. So what is going on?

Forensics

First, what are the data transfer patterns? Maybe it’s just a temporary spike for whatever reason? Not really. After a slow ramp-up, it’s been 140GB/hour consistently for almost 24 hours.

Next up, where is it coming from? Virtually all requests were from France. I do love brie and baguettes, but I don’t think the French love my blog that much.

The average viewer is, supposedly, on a Desktop Windows machine and uses Firefox. As for more in-depth logs about viewers - my CloudFront distribution did not have any logging turned on. Probably for the best, because it would’ve racked up a bigger bill faster.

As unlikely as it is, maybe I too had forgotten some large file on S3, as happened to Chris Short in July. However, opening the popular objects page presented a different picture. Each URL on the website - HTML, CSS, JS & images - had been opened more than a million times, accumulating gigabytes of total data transfer, contributing to my monthly bill.

CloudFront tracks referrers, but referrer is notoriously unreliable as is the case this time - the referrer is my website or not-specified at all.

This is all I have picked up. As for whether this is malicious or crawler-that-went-rogue, I lean towards malicious. It is hard to believe something of such scale would’ve been able to continue for 24 hours and more so accidentally.

AWS Support

So the initial bill of $75 did throw me off, althought it would be okay if it were caused by legitimate traffic. However, it looks like a DDoS attack, and since CloudFront advertises basic coverage against DDoS, this is just something they have not caught up to yet.

I’ll just open a support ticket with billing to bring attention and move on, right? Well, not quite. In retrospect, this might have been naive, but “Amazon CloudFront, AWS Shield, WAF, and Route 53 work seamlessly to create <…> security perimeter against <…> application layer DDoS attacks” not of the goodness of their hearts. As noted more accurately in DDoS resiliency whitepaper - out of the box it only protects against transport and network layer attacks. This makes me more worried and I check the billing dashboard again.

So within hours things have escalated quickly to $375. Dreading the idea of having to pay this, I disable my CloudFront distribution, contemplate for a bit, and decide that email support is probably not going to cut it. It is time for a serious 1:1 chat with support.

Andrew from AWS answers within minutes, asks whether I expect(ed) that usage, checks in with an internal engineer, and confirms that AWS will cover the bill with credits. In the interim, I get pep talked by AWS Support engineer. :)

Conclusions

I have three takeaways from this incident.

First, this snippet has probably saved my eyes from popping out at the end of the month:

AWSTemplateFormatVersion: '2010-09-09'

Resources:

  Budget:
    Type: AWS::Budgets::Budget
    Properties:
      NotificationsWithSubscribers:
      - Subscribers:
        - SubscriptionType: SNS
          Address: !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:MySNSTopicForBudgetAlerts
        Notification:
          ComparisonOperator: GREATER_THAN
          NotificationType: ACTUAL
          Threshold: 80
          ThresholdType: PERCENTAGE
      - Subscribers:
        - SubscriptionType: SNS
          Address: !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:MySNSTopicForBudgetAlerts
        Notification:
          ComparisonOperator: GREATER_THAN
          NotificationType: FORECASTED
          Threshold: 100
          ThresholdType: PERCENTAGE
      Budget:
        BudgetLimit:
          Amount: 25
          Unit: USD
        TimeUnit: MONTHLY
        CostTypes:
          IncludeSupport: true
          IncludeOtherSubscription: true
          IncludeTax: true
          IncludeSubscription: true
          UseBlended: false
          IncludeUpfront: true
          IncludeDiscount: true
          IncludeCredit: false
          UseAmortized: false
          IncludeRefund: false
        BudgetType: COST

Second, in case of emergency, and for a private AWS account, choosing the chat option is best when creating a support case.

Third, the reasons why someone decided to do this remain unknown to me. I am still curious as an attack like this must have cost money. A quick google search leads to “The Cost Of Launching a DDoS Attack” on Securelist:

The cost of a five-minute attack on a large online store is about $5

Attack on this blog took at least 24 hours before I turned the domain off for a day, and generated a transfer rate of 140GB/hour. It definitely must have cost well above $100, if not more. And if that is true, someone must have been very proud of taking a $2/mo website down paying orders of magnitude more. Bravo! Unless the target was not me directly, but then again - who knows?