QUESTIONS?

Updates & insights on all things CDN

BootstrapCDN Security Post-Mortem

A very unfortunate security event happened last month, which affected folks using BootstrapCDN. We at NetDNA want to share an open, detailed report in this blog post, and continue to answer questions that may not have been addressed. Incident Timeline On May 24th at 7:19pm, I got a tweet: Something seemed to be up. I checked the origin and it was clean: no files were changed, there was no activity in /var/log/btmp, etc. It seemed like there was no unauthorized access. But the next day May 25th  at 9:17am I got another tweet: Screenshot_5_28_13_2_59_PM I jumped on my laptop and logged into the Control Panel to purge the cache. Here's what I saw: Screenshot_5_28_13_3_24_PM "Wait, what!? How was the origin not the BootstrapCDN origin URL?". I quickly reverted the origin server to the original (origin.bootstrapcdn.com) and purged the cache. Now, I was seeing 502 errors: not only did the attacker change the origin server, but the port (to 81). After reverting the port back to 80, and letting the change propagate, I purged the cache again. In hindsight, I should have checked the file hashes on the CDN; I had only audited our origin server to ensure there was no unauthorized access on the origin.

What happened?

From what we've discovered, it appears one of our vendors laid off a support engineer. Their credentials weren't revoked, and that person sold or otherwise transferred them to the attackers. The attackers then rebooted a server into single-user mode, changed the root password, and ssh'd into the  server. From there, they accessed the BootstrapCDN API credentials, updated the origin and port, and purged specific file caches. They began sending files from another hacked web server with a malicious javascript payload, likely an exploit toolkit based on CVE-2012-0507 (details). The attack was executed when the visitor's user-agent was "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)" and had Java installed; other user-agents returned a 404 to mask the attack.

What was in the malicious files?

The malicious javascript files contained the following:Which translates to:The iframe ran the following:The iframe appears to exploit the Java vulnerability mentioned above.

Actions Taken

At NetDNA, we've responded to the incident by:
  • Strengthen our vendor relationships to ensure strict security policies
  • Working with the appropriate authorities
  • We've released numerous security features (including limited access API keys, 2-Step Authentication and Control Panel IP whitelisting) to prevent, detect and mitigate future attacks
And on the BootstrapCDN side we have:
  • Replaced all API keys to be report/purge-only (so origin and port information cannot be modified)
  • Utilized the new NetDNA security features
  • Written a script to verify the MD5 checksums on the uploaded files, and alert us by email if there's ever a mismatch
BootstrapCDN_Security Unfortunately, this type of attack is a potential threat to any free public CDN because of the reach and diversity of website visitors. We encourage other free public CDN providers to learn from our experiences by continuously monitoring the files they serve, and communicating openly about security breaches that may arise. I'd like to personally offer thanks to the following people for their help during this unfortunate incident: Dealing with the aftermath of these events is never easy, but we're continuing to improve our service as a result. Thanks for your support.
  • http://opensourcehacker.com Mikko Ohtamaa

    Very nice and honest post-mortem!

Categories

Recent Posts

  • Now Serving: Hong Kong China
  • 5 Things you can do with our Raw Logs API
  • Happy 2nd Birthday BootstrapCDN
  • Now Shipping: MaxCLI > maxcurl
  • Now Serving: Tokyo Japan

Compare the speed of two websites in 5 seconds.

100% Free

Start Testing
TRUSTED BY