We take security matters extremely seriously here at Clearspring. I wanted to provide a more detailed followup to Saturday morning’s “bin.clearspring.com malware” issue that affected latent users of our deprecated Launchpad Widget Platform for a few hours. Note: This issue did not affect current AddThis installations.
What Happened on Saturday, and Why
We, like most of you, use a Content Delivery Network (CDN) to serve static files as quickly as possible to clients around the world. We actually have several domains set up on CDNs, the one we’re talking about now is used to serve bin.clearspring.com. The assets at this hostname are related to our deprecated Launchpad Widget Platform, as well as some other legacy Clearspring offerings.
From time to time, we allow our CDN provider to create an account with access to our assets to help us debug or otherwise analyze the production system. One such account, which was used for production support some time ago and not deleted as it should have been, was compromised on July 20th, outside of our facility and network, and used to upload a redirecting index.html file (among a few others). This meant that a request to the root of the host would result in a redirect to a spam/malware site. Given that users do not directly visit this URL under normal circumstances (http://bin.clearspring.com is usually a 404), it had no actual immediate negative consequence. That changed in the middle of the night on July 31st (early Saturday AM Eastern time), when Google’s malware classification system located it and, because of a request to http://bin.clearspring.com/ that redirected to this malware site, classified bin.clearspring.com as a malware site. This became visible to users, even those not visiting bin.clearspring.com directly, because the Chrome browser actually blocks sites that include loads of other malware-classified sites. Sites that included legacy Launchpad widgets, therefore, would have been blocked in Chrome.
We began getting user reports early Saturday morning, and quickly identified and removed the offending assets and the access that had been used to place them there. This removed the immediate problem, but left Google Chrome users in the same state as it takes some time for Google to reclassify. We initiated the Google re-review process, essentially requesting Google to recheck the site and declassify it as malware (upon finding no more). This process took a few hours, and by Saturday afternoon the issue was resolved both in terms of the existence of the bad files, as well as in terms of the malware classification and resultant Google Chrome block.
What We’re Doing to Prevent This
We’ve taken and are taking several additional measures to prevent a similar occurrence in the future. We’ve revised our access control policy for our core CDN to disallow access from external locations entirely, are resetting all existing access controls, and are putting in place an additional layer of access monitoring on our CDN provider accounts. We’ve also extended our operational monitoring system to verify the integrity of our CDN roots, not just the assets we actually serve to our products, from an end user perspective. Finally, as it relates to malware classifications specifically, we’ve also claimed all CDN properties (not just our own public web properties) with Google to ensure we have a better view of its own crawling analysis on an ongoing basis.
What Products did this Affect?
This issue affected legacy Launchpad, Widget Promotion Channel, and Widget Ad Network offerings. It did not affect any standard-install AddThis installation. It may have affected AddThis installations for publishers that have cached the AddThis code on their own domains and not updated it since March (our caching policy indicates that publishers will update every 24 hours).
Concluding
This was the result of a security breach of our CDN. We apologize for the problems the event caused for you or your visitors on Saturday morning. We believe we acted as quickly as possible to resolve the issue, and have taken the correct preventative steps moving forward. Thanks for your support, and don’t hesitate to let us know if you have any questions at all about how this was handled or what we’re doing about it moving forward.