"No matter what happens in life, be good to people. Being good to people is a wonderful legacy to leave behind." - Taylor Swift (whose website has been IPv6 enabled for many many years)

Starting today with free domains, IPv6 is no longer something you can toggle on and off, it’s always just on.

Cloudflare has always been a gateway for visitors on IPv6 connections to access sites and applications hosted on legacy IPv4-only infrastructure. Connections to Cloudflare are terminated on either IP version and then proxied to the backend over whichever IP version the backend infrastructure can accept.

That means that a v6-only mobile phone (looking at you, T-Mobile users) can establish a clean path to any site or mobile app behind Cloudflare instead of doing an expensive 464XLAT protocol translation as part of the connection (shaving milliseconds and conserving very precious battery life).

That IPv6 gateway is set by a simple toggle that for a while now has been default-on. And to make up for the time lost before the toggle was default on, in August 2016 we went back retroactively and enabled IPv6 for those millions of domains that joined before IPv6 was the default. Over those next few months, we enabled IPv6 for nearly four million domains –– you can see Cloudflare’s dent in the IPv6 universe below –– and by the time we were done, 98.1% of all of our domains had IPv6 connectivity.

As an interim step, we added an extra feature –– when you turn off IPv6 in our dashboard, we remind you just how archaic we think that is.

With close to 100% IPv6 enablement, it no longer makes sense to offer an IPv6 toggle. Instead, Cloudflare is offering IPv6 always on, with no off-switch. We’re starting with free domains, and over time we’ll change the toggle on the rest of Cloudflare paid-plan domains.

In November we published stats about the IPv6 usage we see on the Cloudflare network in an attempt to answer who and what is pushing IPv6. The top operating systems by percent IPv6 traffic are iOS, ChromeOS, and MacOS respectively. These operating systems push significantly more IPv6 traffic than their peers because they use a routing choice algorithm called Happy Eyeballs. Happy Eyeballs opportunistically chooses IPv6 when available by doing two DNS lookups –– one for an IPv6 address (this IPv6 address is stored in the DNS AAAA record - pronounced quad-A) and then one for the IPv4 address (stored in the DNS A record). Both DNS queries are flying over the Internet at the same time and the client chooses the address that comes back first. The client even gives IPv6 a few milliseconds head start (iOS and MacOS give IPv6 lookups a 25ms head start for example) so that IPv6 may be chosen more often. This works and has fueled some of IPv6’s growth. But it has fallen short of the goal of a 100% IPv6 world.

While there are perfectly good historical reasons why IPv6 and IPv4 addresses are stored in separate DNS types, today clients are IP version agnostic and it no longer makes sense for it to require two separate round trips to learn what addresses are available to fetch a resource from.

Alongside OpenDNS, we are testing a new idea - what if you could ask for all the addresses in just one DNS query?

With OpenDNS, we are prototyping and testing just that –– a new DNS metatype that returns all available addresses in one DNS answer –– A records and AAAA records in one response. (A metatype is a query type in DNS that end users can’t add into their DNS zone file, it’s assembled dynamically by the authoritative nameserver.)

What this means is that in the future if a client like an iPhone wants to access a mobile app that uses Cloudflare DNS or using another DNS provider that supports the spec, the iPhone DNS client would only need to do one DNS lookup to find where the app’s API server is located, cutting the number of necessary round trips in half.

This reduces the amount of bandwidth on the DNS system, and pre-populates global DNS caches with IPv6 addresses, making IPv6 lookups faster in the future, with the side benefit that Happy Eyeballs clients prefer IPv6 when they can get the address quickly, which increases the amount of IPv6 traffic that flows through the Internet.

We have the metaquery working in code with the reserved TYPE65535 querytype. You can ask a Cloudflare nameserver for TYPE65535 of any domain on Cloudflare and get back all available addresses for that name.

$ dig cloudflare.com @ns1.cloudflare.com -t TYPE65535 +short
198.41.215.162
198.41.214.162
2400:cb00:2048:1::c629:d6a2
2400:cb00:2048:1::c629:d7a2
$

Did we mention Taylor Swift earlier?

$ dig taylorswift.com @ns1.cloudflare.com -t TYPE65535 +short
104.16.193.61
104.16.194.61
104.16.191.61
104.16.192.61
104.16.195.61
2400:cb00:2048:1::6810:c33d
2400:cb00:2048:1::6810:c13d
2400:cb00:2048:1::6810:bf3d
2400:cb00:2048:1::6810:c23d
2400:cb00:2048:1::6810:c03d
$

We believe in proving concepts in code and through the IETF standards process. We’re currently working on an experiment with OpenDNS and will translate our learnings to an Internet Draft we will submit to the IETF to become an RFC. We’re sure this is just the beginning to faster, better deployed IPv6.

A buffer overflow error in GNU libc DNS stub resolver code was announced last week as CVE-2015-7547. While it doesn't have any nickname yet (last year's Ghost was more catchy), it is potentially disastrous as it affects any platform with recent GNU libc—CPEs, load balancers, servers and personal computers alike. The big question is: how exploitable is it in the real world?

It turns out that the only mitigation that works is patching. Please patch your systems now, then come back and read this blog post to understand why attempting to mitigate this attack by limiting DNS response sizes does not work.

But first, patch!

Let's start with the PoC from Google, it uses the first attack vector described in the vulnerability announcement. First, a 2048-byte UDP response forces buffer allocation, then a failure response forces a retry, and finally the last two answers smash the stack.

$ echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf
$ sudo python poc.py &
$ valgrind curl http://foo.bar.google.com
==17897== Invalid read of size 1
==17897==    at 0x59F9C55: __libc_res_nquery (res_query.c:264)
==17897==    by 0x59FA20F: __libc_res_nquerydomain (res_query.c:591)
==17897==    by 0x59FA7A8: __libc_res_nsearch (res_query.c:381)
==17897==    by 0x57EEAAA: _nss_dns_gethostbyname4_r (dns-host.c:315)
==17897==    by 0x4242424242424241: ???
==17897==  Address 0x4242424242424245 is not stack'd, malloc'd or (recently) free'd
Segmentation fault

This proof of concept requires attacker talking with glibc stub resolver code either directly or through a simple forwarder. This situation happens when your DNS traffic is intercepted or when you’re using an untrusted network.

One of the suggested mitigations in the announcement was to limit UDP response size to 2048 bytes, 1024 in case of TCP. Limiting UDP is, with all due respect, completely ineffective and only forces legitimate queries to retry over TCP. Limiting TCP answers is a plain protocol violation that cripples legitimate answers:

$ dig @b.gtld-servers.net +tcp +dnssec NS root-servers.net | grep "MSG SIZE"
;; MSG SIZE  rcvd: 1254

Regardless, let's see if response size clipping is effective at all. When calculating size limits, we have to take IP4 headers into account (20 octets), and also the UDP header overhead (8 octets), leading to a maximum allowed datagram size of 2076 octets. DNS/TCP may arrive fragmented—for the sake of argument, let's drop DNS/TCP altogether.

$ sudo iptables -I INPUT -p udp --sport 53 -m length --length 2077:65535 -j DROP
$ sudo iptables -I INPUT -p tcp --sport 53 -j DROP
$ valgrind curl http://foo.bar.google.com
curl: (6) Could not resolve host: foo.bar.google.com

Looks like we've mitigated the first attack method, albeit with collateral damage. But what about the UDP-only proof of concept?

$ echo "nameserver 127.0.0.10" | sudo tee /etc/resolv.conf
$ sudo python poc-udponly.py &
$ valgrind curl http://foo.bar.google.com
==18293== Syscall param socketcall.recvfrom(buf) points to unaddressable byte(s)
==18293==    at 0x4F1E8C3: __recvfrom_nocancel (syscall-template.S:81)
==18293==    by 0x59FBFD0: send_dg (res_send.c:1259)
==18293==    by 0x59FBFD0: __libc_res_nsend (res_send.c:557)
==18293==    by 0x59F9C0B: __libc_res_nquery (res_query.c:227)
==18293==    by 0x59FA20F: __libc_res_nquerydomain (res_query.c:591)
==18293==    by 0x59FA7A8: __libc_res_nsearch (res_query.c:381)
==18293==    by 0x57EEAAA: _nss_dns_gethostbyname4_r (dns-host.c:315)
==18293==    by 0x4F08AA0: gaih_inet (getaddrinfo.c:862)
==18293==    by 0x4F0AC4C: getaddrinfo (getaddrinfo.c:2418)
==18293==  Address 0xfff001000 is not stack'd, malloc'd or (recently) free'd
*** Error in `curl': double free or corruption (out): 0x00007fe7331b2e00 ***
Aborted

While it's not possible to ship a whole attack payload in 2048 UDP response size, it still leads to memory corruption. When the announcement suggested blocking DNS UDP responses larger than 2048 bytes as a viable mitigation, it confused a lot of people, including other DNS vendors and ourselves. This, and the following proof of concept show that it's not only futile, but harmful in long term if these rules are left enabled.

So far, the presented attacks required a MitM scenario, where the attacker talks to a glibc resolver directly. A "good enough" mitigation is to run a local caching resolver, to isolate glibc code from the attacker. In fact, doing so not only improves the Internet performance with a local cache, but also prevents past and possibly future security vulnerabilities.

Unfortunately, no. A local stub resolver such as dnsmasq alone is not sufficient to defuse this attack. It's easy to traverse, as it doesn't scrub upstream answers—let's see if the attack goes through with a modified proof of concept that uses only well-formed answers and zero time-to-live (TTL) for cache traversal.

$ echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf
$ sudo dnsmasq -d -a 127.0.0.1 -R -S 127.0.0.10 -z &
$ sudo python poc-dnsmasq.py &
$ valgrind curl http://foo.bar.google.com
==20866== Invalid read of size 1
==20866==    at 0x8617C55: __libc_res_nquery (res_query.c:264)
==20866==    by 0x861820F: __libc_res_nquerydomain (res_query.c:591)
==20866==    by 0x86187A8: __libc_res_nsearch (res_query.c:381)
==20866==    by 0xA0C6AAA: _nss_dns_gethostbyname4_r (dns-host.c:315)
==20866==    by 0x1C000CC04D4D4D4C: ???
Killed

The big question is—now that we've seen that the mitigation strategies for MitM attacks are provably ineffective, can we exploit the flaw off-path through a caching DNS resolver?

Let's start with the first phase of the attack—a compliant resolver is never going to give out a response larger than 512 bytes over UDP to a client that doesn't support EDNS0. Since the glibc resolver doesn't do that by default, we have to escalate to TCP and perform the whole attack there. Also, the client should have at least two nameservers, otherwise it complicates a successful attack.

$ echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf
$ echo "nameserver 127.0.0.1" | sudo tee -a /etc/resolv.conf
$ sudo iptables -F INPUT
$ sudo iptables -I INPUT -p udp --sport 53 -m length --length 2077:65535 -j DROP

Let's try it with a proof of concept that merges both the DNS proxy and the attacker.

1. The DNS proxy on localhost is going to ask the attacker both queries over UDP, and the attacker responds with a TC flag to force client to retry over TCP.

2. The attacker responds once with a TCP response of 2049 bytes or longer, then forces the proxy to close the TCP connection to glibc resolver code. This is a critical step with no reliable way to achieve that.

3. The attacker sends back a full attack payload, which the proxy happily forwards to the glibc resolver client.

$ sudo python poc-tcponly.py &
$ valgrind curl http://foo.bar.google.com
==18497== Invalid read of size 1
==18497==    at 0x59F9C55: __libc_res_nquery (res_query.c:264)
==18497==    by 0x59FA20F: __libc_res_nquerydomain (res_query.c:591)
==18497==    by 0x59FA7A8: __libc_res_nsearch (res_query.c:381)
==18497==    by 0x57EEAAA: _nss_dns_gethostbyname4_r (dns-host.c:315)
==18497==    by 0x1C000CC04D4D4D4C: ???
==18497==  Address 0x1000000000000103 is not stack'd, malloc'd or (recently) free'd
Killed

The key factor to a real world non-MitM cache resolver attack is to control the messages between the resolver and the client indirectly. We came to the conclusion that djbdns’ dnscache was the best target for attempting to illustrate an actual cache traversal.

In order to fend off DoS attack vectors like slowloris, which makes numerous simultaneous TCP connections and holds them open to clog up a service, DNS resolvers have a finite pool of parallel TCP connections. This is usually achieved by limiting these parallel TCP connections and closing the oldest or least-recently active one. For example—djbdns (dnscache) holds up to 20 parallel TCP connections, then starts dropping them, starting from the oldest one. Knowing this, we realised that we were able to terminate TCP connections with ease. Thus, one security fix becomes another bug’s treasure.

In order to exploit this, the attacker can send a truncated UDP A+AAAA query, which triggers the necessary retry over TCP. The attacker responds with a valid answer with a TTL of 0 and dnscache sends the glibc client a truncated UDP response. At this point, the glibc function send_vc() retries with dnscache over TCP and since the previous answer's TTL was 0, dnscache asks the attacker’s server for the A+AAAA query again. The attacker responds to the A query with an answer larger than 2000 to induce glibc's buffer mismanagement, and dnscache then forwards it to the client. Now the attacker can either wait out the AAAA query while other clients are making perfectly legitimate requests or instead make 20 TCP connections back to dnscache, until dnscache terminates the attacker's connection.

Now that we’ve met all the conditions to trigger another retry, the attacker sends back any valid A response and a valid, oversized AAAA that carries the payload (either in CNAME or AAAA RDATA), dnscache tosses this back to the client, triggering the overflow.

It seems like a complicated process, but it really is not. Let’s have a look at our proof-of-concept:

$ echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf
$ echo "nameserver 127.0.0.1" | sudo tee -a /etc/resolv.conf
$ sudo python poc-dnscache.py
[TCP] Sending back first big answer with TTL=0
[TCP] Sending back second big answer with TTL=0
[TCP] Preparing the attack with an answer >2k
[TCP] Connecting back to caller to force it close original connection('127.0.0.1', 53)
[TCP] Original connection was terminated, expecting to see requery...
[TCP] Sending back a valid answer in A
[TCP] Sending back attack payload in AAAA

Client:

$ valgrind curl https://www.cloudflare.com/
==6025== Process terminating with default action of signal 11 (SIGSEGV)
==6025==  General Protection Fault
==6025==    at 0x8617C55: __libc_res_nquery (res_query.c:264)
==6025==    by 0x861820F: __libc_res_nquerydomain (res_query.c:591)
==6025==    by 0x86187A8: __libc_res_nsearch (res_query.c:381)
==6025==    by 0xA0C6AAA: _nss_dns_gethostbyname4_r (dns-host.c:315)
==6025==    by 0x1C000CC04D4D4D4C: ???
Killed

This PoC was made to simply illustrate that it’s not only probable, but possible that a remote code execution via DNS resolver cache traversal can and may be happening. So, patch. Now.

We reached out to OpenDNS, knowing they had used djbdns as part of their codebase. They investigated and verified this particular attack does not affect their resolvers.

Dan Kaminsky wrote a thoughtful blog post about scoping this issue. He argues:

I’m just going to state outright: Nobody has gotten this glibc flaw to workthrough caches yet. So we just don’t know if that’s possible. Actualexploit chains are subject to what I call the MacGyver effect.

Current resolvers scrub and sanitize final answers, so the attack payload must be encoded in a well-formed DNS answer to survive a pass through the resolver. In addition, only some record types are safely left intact—as the attack payload is carried in AAAA query, only AAAA records in the answer section are safe from being scrubbed, thus forcing the attacker to encode the payload in these. One way to circumvent this limitation is to use a CNAME record, where the attack payload may be encoded in a CNAME target (maximum of 255 octets).

The only good mitigation is to run a DNS resolver on localhost where the attacker can't introduce resource exhaustion, or at least enforce minimum cache TTL to defuse the waiting game attack.

You might think it's unlikely that you could become a MitM target, but the fact is that you already are. If you ever used a public Wi-Fi in an airport, hotel or maybe in a café, you may have noticed being redirected to a captcha portal for authentication purposes. This is a temporary DNS hijacking redirecting you to an internal portal until you agree with the terms and conditions. What's even worse is a permanent DNS interception that you don't notice until you look at the actual answers. This happens on a daily basis and takes only a single name lookup to trigger the flaw.

Neither DNSSEC nor independent public resolvers prevent it, as the attack happens between stub and the recursor on the last mile. The recent flaws highlight the fragility of not only legacy glibc code, but also stubs in general. DNS is deceptively complicated protocol and should be treated carefully. A generally good mitigation is to shield yourself with a local caching DNS resolver¹, or at least a DNSCrypt tunnel. Arguably, there might be a vulnerability in the resolver as well, but it is contained to the daemon itself—not to everything using the C library (e.g., sudo).

If you're running GNU libc between 2.9 and 2.22 then yes. Below is an informative list of several major platforms affected.

Platform

Notice

Status

Debian

CVE-2015-7547

Patched packages available (squeeze and newer)

Ubuntu

USN-2900-1

Patched packages available (14.04 and newer)

RHEL

KB2161461

Patched packages available (RHEL 6-7)

SUSE

SUSE-SU-2016:0472-1

Patched packages available (latest)

Network devices & CPEs

Updated list of affected platforms

The toughest problem with this issue is the long tail of custom CPEs and IoT devices, which can't be really enumerated. Consult the manufacturer's website for vulnerability disclosure. Keep in mind that if your CPE is affected by remote code execution, its network cannot be treated as safe anymore.

If you're running OS X, iOS, Android or any BSD flavour², you're not affected.

• [1] Take a look at Unbound, PowerDNS Recursor or Knot DNS Resolver for a compliant validating resolver.

• [2] Applications running under Linux emulation and using glibc may be affected, make sure to update ports.

This has been a rough week in the security industry with big attacks and compromises reported at companies from Facebook to Apple. We're therefore happy to end the week with some good news: the web's open resolvers, one of the sources of the biggest DDoS attacks, are getting closed.

Last October, we wrote a blog post about DDoS amplification attacks. This type of attack makes up some of the largest DDoSs CloudFlare sees, sometimes exceeding 100 gigabits per second (100Gbps). The attacks use DNS resolvers that haven't been properly secured in order to "amplify" the resources of the attacker. An attacker can achieve more than a 50x amplification, meaning that for every byte they are able to generate themselves they can pummel a victim with 50 bytes of garbage data.

The problem stems from misconfigured DNS resolver software (e.g., BIND) that is set up to respond to a query from any IP address. Since DNS requests typically are sent over UDP, which, unlike TCP, does not require a handshake, an attacker can spoof a victim's IP address as the source address in a packet and a misconfigured DNS resolver will happily bombard the victim with responses.

While CloudFlare's network is very good at absorbing even these large attacks, the long term solution for the web is for providers to clean up the open resolvers running on their networks. We wanted to help with that, so we engaged in a bit of name-and-shame at the end of the last blog post, listing the networks with the largest number of open resolvers. The good news is it worked: almost four months later our tests show that the number of open resolvers across the Internet is down more than 30%. The chart below shows the progress individual networks have made in cleaning up the problem.

A few other organizations deserve a special shout out for helping with this effort. The great folks at Team Cymru have been tracking open resolvers and other badness online since before CloudFlare was even an idea. Their consistent efforts in this area have been awesome, and we're in the process of partnering with them to help get the word out.

In addition, SoftLayer has been especially vocal and active in spearheading clean up efforts on its network. As they pointed out in a great blog post, because of the size and nature of their network, it's often difficult for them to police the configuration of software their customers run. Even so, they are actively reaching out to customers to educate them about the dangers of running open resolvers on their networks.

We greatly appreciate country CERTs/CSIRTs and various Information Sharing and Analysis Centers (ISACs) reaching out to us offering to get in touch with some of the less responsive network providers.

Going forward, we are happy to provide the IP addresses running open resolvers directly to any network provider that is interested in cleaning up their networks. If you're running a network on the list above, please don't hesitate to reach out to us, and we'll get you the data you need to help with cleanup.

Just a quick update on the initiative between CloudFlare, OpenDNS, and the DCWG to clean up the DNSChanger malware. In the last week, just over 11,000 websites enabled the Visitor DNSChanger Detector App through CloudFlare. Since then, those sites have collectively served more than 56 million page views. Just over 12,000 visitors to those websites have seen the warning about the DNSChanger virus and clicked on the link to learn more and clean up their infection. In just the first week, that's nearly 4% of the total number of estimated infected computersthat the CloudFlare community has already helped notify and get cleaned up.

While hundreds of thousands of computers are still infected and risk losing access to the Internet on July 9, 2012, we're proud of the strong start to this effort by the CloudFlare community along with OpenDNS and the DCWG.

If you haven't yet enabled the Visitor DNSChanger Detector App for your sites on CloudFlare, you can do so by following this link.

Thanks for helping us #savetheweb.

Several years ago, some suspected cyber criminals on the Internet wrote a family of malware dubbed DNSChanger. About a year ago, law enforcement tracked down the suspected cyber criminals behind this malware, arrested them, and took over the servers they were using to redirect customers to rogue sites.

As a result of a court order, the Internet Systems Consortium (ISC) under the direction of the FBI, has continued to run the DNS servers used by the malware for the last year. However, the court order will soon expire and those servers are scheduled to be shut down on July 9, 2012. When that happens, hundreds of thousands of Internet users whose systems are still infected and/or affected could lose access to the web, email, and anything else that depends on DNS. This is the story of how two Internet infrastructure startups — CloudFlare and OpenDNS — are playing a small part to help solve the problem.

Up front, in order to understand this story, you need to understand there are two types of DNS servers: recursive and authoritative. Everyone who uses the Internet needs a recursive DNS server. Your ISP usually provides these types of services or you can use a provider like OpenDNS, Google, DNSAdvantage, other public resolvers, or even run a server yourself to handle your recursive DNS queries.

On the other hand, every domain needs at least one authoritative DNS server. Authoritative servers are where a particular domain's records are hosted and published. Many domain registrars provide authoritative DNS servers, or you can use a service like CloudFlare and we provide authoritative DNS. When an Internet user types a Universal Resource Identifier (URI) aka Universal Resource Locator (URL) into their browser, clicks on a link, or sends an email, their computer queries their recursive DNS provider. If the recursive DNS provider has the answer cached then it responds. If it doesn't have the answer cached, or if the answer it has is stale, then the recursive DNS server queries the authoritative DNS server.

As mentioned above, OpenDNS provides recursive DNS. Their customers are web surfers and they provide a terrific service that helps speed up Internet browsing and protect people on the web from malware. CloudFlare provides authoritative DNS. Our customers are websites and we make those sites faster and protect sites from attacks directed at them. While we're often asked if OpenDNS and CloudFlare are competitive, in reality both services are complementary just using different parts of DNS (recursive and authoritative) to achieve a similar mission: a faster, safer, better Internet.

The DNSChanger malware family was designed to change the recursive DNS server that Internet users' computers queries. Instead of directing DNS queries at the recursive server you or your ISP configured, the malware modified computer settings to route queries to recursive DNS servers controlled by the suspected cyber criminals.

The job of DNS is to translate a domain name such as dcwg.org, which humans prefer, into an IP address, like 108.162.205.64, which servers and routers can use. If you are a cyber criminal and you can gain control over someone's recursive DNS then you can direct traffic to certain sites to a fake version of the site. Once DNSChanger had web surfers querying rogue recursive DNS servers, all requests for legitimate websites could be directed to a fake website. For example, even if you typed your bank's domain name into your browser, if the suspected cyber criminals control recursive DNS then they can send you to a malicious site and steal your information.

Over the years DNSChanger operated unchecked, more than a million computers and home routers had their DNS configurations modified. Thankfully, law enforcement was able to track down the suspected cyber criminals behind the malware, arrest them, and seize control of the rogue recursive DNS servers. Unfortunately, hundreds of thousands of computers are still using the formerly rogue recursive DNS servers. On July 9, 2012 the court order directing ISC to operate the servers expires and those servers are scheduled to be shut down. On that date, all systems which still have their DNS settings modified by DNSChanger will effectively be cut off from the Internet.

The DNSChanger Working Group (DCWG), a loosely affiliated organization comprised of some of the world's largest and most competent ISPs, search engines vendors, software vendors, security companies, and others, has been working to get the word out about the problem and reduce the impact of the shutdown of the DNSChanger recursive servers. The DCWG launched a website (dcwg.org) to provide information about the malware, let people test whether they are infected, and provide recommendations on how to fix their systems. CloudFlare first became involved when the folks at dcwg.org reached out to us because their site was under heavy load after attention from major media outlets. CloudFlare helped keep the dcwg.org website online under the load caused by media attention over the last 10 days. We offloaded more than 95% of the traffic to the site, ensuring the site ran fast and stable even when it was being featured on the front page of cnn.com.

Unfortunately, one of the challenges in trying to address situations like DNSChanger is that you only know to go to the dcwg.org website if you already know about it. What you needed was something akin to an emergency broadcast system that would inform people who were infected that they had a problem as they surfed the web. In the process of working with the DCWG, we realized we might be able to help.

Some of our engineers created an app named Visitor DNSChanger Detector App. Any website on CloudFlare can enable the app with a single click from our apps marketplace. The app installs a small bit of Javascript on the page that tests visitors to see if they're infected. If the tests do not detect anything, nothing happens. If the tests indicate that the DNSChanger recursive servers are being used, then a banner is displayed across the top of the page and visitors are directed to instructions on how to clean up the infection (more on that in a second).

More than 470 million people pass through CloudFlare's network on a monthly basis. Our data suggest that more than half of the people infected with DNSChanger would visit at least one site on CloudFlare per month. Thepower of the Visitor DNSChanger Detector App is that as CloudFlare publishers enable it then there is an increasing likelihood that people who are infected will get information about their infection before they are no longer able to use the Internet on July 9, 2012.

While we've made it extremely easy for publishers on CloudFlare's network to help get the word out, we didn't want to restrict participation to only those sites using our service. We therefore decided to release the code for the checks publicly and as open source so anyone who can install a few lines of Javascript on their web pages will be able to install it on their own sites to inform their potentially infected users. You can access the code from the following GitHub Repo. We're hopeful that sites both large and small will take the time to install the code in order to help inform their visitors who may be infected.

While CloudFlare is able to assist with informing web surfers they have an infection, we aren't particularly well situated to actually fix the problem. After all, it isn't our customers that are directly impacted,but rather the customers of our customers. Many of the folks infected can get help from their ISPs, but for some this might not be an option. CloudFlare reached out to David Ulevitch, the CEO of OpenDNS and he saw this as a great opportunity to further OpenDNS's mission of helping build a better Internet. We added OpenDNS as aresource for publishers to display to their customers when the Javascript detects the use of the DNSChanger recursive servers.

This incident illustrates to me the importance and power of the DNS system that underpins the Internet. The suspected cyber criminals were able to modify DNS settings to steal advertising revenue and perform other illegal activities. CloudFlare uses authoritative DNS in order to provision powerful tools to make sites faster and even help create a sort of emergency warning system for the Internet. OpenDNS provides high performance recursive DNS caching services for their customers. Combined, we hope to help the DCWG get the word out so the hundreds of thousands of Internet users still impacted by the DNSChanger malware will be able to take steps to ensure they'll be able to use the Internet on July 10, 2012 and beyond.