Cloudflare chief technology officer, John Graham-Cumming, reported yesterday in a blog post a major security incident caused by a Cloudflare parser bug. The incident was responsible for revealing some private information from the company's customers, such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. Also, to worsen the situation, the leaked memory had already been cached by search engines by the time the company was made aware of the issue.
Fortunately, Cloudflare customer SSL private keys were not leaked and the firm has not discovered any evidence of malicious exploits of the bug or other reports of its existence yet. Also, Cloudflare worked quickly to circumvent the problem after Tavis Ormandy, from Google’s Project Zero, contacted the company to report the security issue.
First, Cloudflare disabled three minor features: Email Obfuscation, Server-side Excludes, and Automatic HTTPS Rewrites. Because only those three features were using the bogus HTML parser, the memory leak was immediately retained.
After that, with the help of Google, Yahoo, Bing and other search engines, Cloudflare identified 770 unique URIs, from 161 unique domains, that had been cached containing the leaked memory. All the identified cached data was then purged by the search engines.
You can follow the main events of the response of the company to the bug in the timeline below, provided by Graham-Cumming:
All times are UTC.
2017-02-18 0011 Tweet from Tavis Ormandy asking for Cloudflare contact information
2017-02-18 0032 Cloudflare receives details of bug from Google
2017-02-18 0040 Cross functional team assembles in San Francisco
2017-02-18 0119 Email Obfuscation disabled worldwide
2017-02-18 0122 London team joins
2017-02-18 0424 Automatic HTTPS Rewrites disabled worldwide
2017-02-18 0722 Patch implementing kill switch for cf-html parser deployed worldwide2017-02-20 2159 SAFE_CHAR fix deployed globally
2017-02-21 1803 Automatic HTTPS Rewrites, Server-Side Excludes and Email Obfuscation re-enabled worldwide
This was a pretty quick response, but Cloudflare also stated that the memory could have been leaking since September 22, 2016, the day Automatic HTTP Rewrites, one of the minor features responsible for the issue, was enabled. Furthermore, the company informed that the period of four days starting on February 13, 2017, the day Email Obfuscation was partially migrated to the new parser, saw the greatest potential impact, because this feature was the primary cause of the leaked memory.
Finally, we would like to inform that Neowin users are completely safe, since our website doesn't use the Cloudflare services. Also, Cloudflare is now reviewing older software to look for potential other security problems.
Source: Cloudflare Blog via TechCrunch
6 Comments - Add comment