On 4 October 2023, Cloudflare experienced DNS resolution problems starting at 07:00 UTC and ending at 11:00 UTC. Some users of 1.1.1.1 or products like WARP, Zero Trust, or third party DNS resolvers which use 1.1.1.1 may have received SERVFAIL DNS responses to valid queries. We’re very sorry for this outage. This outage was an internal software error and not the result of an attack. In this blog, we’re going to talk about what the failure was, why it occurred, and what we’re doing to make sure this doesn’t happen again.

In the Domain Name System (DNS), every domain name exists within a DNS zone. The zone is a collection of domain names and host names that are controlled together. For example, Cloudflare is responsible for the domain name cloudflare.com, which we say is in the “cloudflare.com” zone. The .com top-level domain (TLD) is owned by a third party and is in the “com” zone. It gives directions on how to reach cloudflare.com. Above all of the TLDs is the root zone, which gives directions on how to reach TLDs. This means that the root zone is important in being able to resolve all other domain names. Like other important parts of the DNS, the root zone is signed with DNSSEC, which means the root zone itself contains cryptographic signatures.

The root zone is published on the root servers, but it is also common for DNS operators to retrieve and retain a copy of the root zone automatically so that in the event that the root servers cannot be reached, the information in the root zone is still available. Cloudflare’s recursive DNS infrastructure takes this approach as it also makes the resolution process faster. New versions of the root zone are normally published twice a day. 1.1.1.1 has a WebAssembly app called static_zone running on top of the main DNS logic that serves those new versions when they are available.

On 21 September, as part of a known and planned change in root zone management, a new resource record type was included in the root zones for the first time. The new resource record is named ZONEMD, and is in effect a checksum for the contents of the root zone.

The root zone is retrieved by software running in Cloudflare’s core network. It is subsequently redistributed to Cloudflare’s data centers around the world. After the change, the root zone containing the ZONEMD record continued to be retrieved and distributed as normal. However, the 1.1.1.1 resolver systems that make use of that data had problems parsing the ZONEMD record. Because zones must be loaded and served in their entirety, the system’s failure to parse ZONEMD meant the new versions of the root zone were not used in Cloudflare’s resolver systems. Some of the servers hosting Cloudflare's resolver infrastructure failed over to querying the DNS root servers directly on a request-by-request basis when they did not receive the new root zone. However, others continued to rely on the known working version of the root zone still available in their memory cache, which was the version pulled on 21 September before the change.

On 4 October 2023 at 07:00 UTC, the DNSSEC signatures in the version of the root zone from 21 September expired. Because there was no newer version that the Cloudflare resolver systems were able to use, some of Cloudflare’s resolver systems stopped being able to validate DNSSEC signatures and as a result started sending error responses (SERVFAIL). The rate at which Cloudflare resolvers generated SERVFAIL responses grew by 12%. The diagrams below illustrate the progression of the failure and how it became visible to users.

21 September 6:30 UTC: Last successful pull of the root zone.4 October 7:00 UTC: DNSSEC signatures in the root zone obtained on 21 September expired causing an increase in SERVFAIL responses to client queries.7:57: First external reports of unexpected SERVFAILs started coming in.8:03: Internal Cloudflare incident declared.8:50: Initial attempt made at stopping 1.1.1.1 from serving responses using the stale root zone file with an override rule.10:30: Stopped 1.1.1.1 from preloading the root zone file entirely.10:32: Responses returned to normal.11:02: Incident closed.

This below chart shows the timeline of impact along with the percentage of DNS queries that returned with a SERVFAIL error:

We expect a baseline volume of SERVFAIL errors for regular traffic during normal operation. Usually that percentage sits at around 3%. These SERVFAILs can be caused by legitimate issues in the DNSSEC chain, failures to connect to authoritative servers, authoritative servers taking too long to respond, and many others. During the incident the amount of SERVFAILs peaked at 15% of total queries, although the impact was not evenly distributed around the world and was mainly concentrated in our larger data centers like Ashburn, Virginia; Frankfurt, Germany; and Singapore.

DNS has a binary format for storing resource records. In this binary format the type of the resource record (TYPE)  is stored as a 16-bit integer. The type of resource record determines how the resource data (RDATA) is parsed. When the record type is 1, this means it is an A record, and the RDATA can be parsed as an IPv4 address. Record type 28 is an AAAA record, whose RDATA can be parsed as an IPv6 address instead. When a parser runs into an unknown resource type it won’t know how to parse its RDATA, but fortunately it doesn’t have to: the RDLENGTH field indicates how long the RDATA field is, allowing the parser to treat it as an opaque data element.

                                   1  1  1  1  1  1
      0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                                               |
    /                                               /
    /                      NAME                     /
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                      TYPE                     |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     CLASS                     |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                      TTL                      |
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                   RDLENGTH                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--|
    /                     RDATA                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

RFC 1035

The reason static_zone didn’t support the new ZONEMD record is because up until now we had chosen to distribute the root zone internally in its presentation format, rather than in the binary format. When looking at the text representation for a few resource records we can see there is a lot more variation in how different records are presented.

.			86400	IN	SOA	a.root-servers.net. nstld.verisign-grs.com. 2023100400 1800 900 604800 86400
.			86400	IN	RRSIG	SOA 8 0 86400 20231017050000 20231004040000 46780 . J5lVTygIkJHDBt6HHm1QLx7S0EItynbBijgNlcKs/W8FIkPBfCQmw5BsUTZAPVxKj7r2iNLRddwRcM/1sL49jV9Jtctn8OLLc9wtouBmg3LH94M0utW86dKSGEKtzGzWbi5hjVBlkroB8XVQxBphAUqGxNDxdE6AIAvh/eSSb3uSQrarxLnKWvHIHm5PORIOftkIRZ2kcA7Qtou9NqPCSE8fOM5EdXxussKChGthmN5AR5S2EruXIGGRd1vvEYBrRPv55BAWKKRERkaXhgAp7VikYzXesiRLdqVlTQd+fwy2tm/MTw+v3Un48wXPg1lRPlQXmQsuBwqg74Ts5r8w8w==
.			518400	IN	NS	a.root-servers.net.
.			86400	IN	ZONEMD	2023100400 1 241 E375B158DAEE6141E1F784FDB66620CC4412EDE47C8892B975C90C6A102E97443678CCA4115E27195B468E33ABD9F78C

Example records taken from https://www.internic.net/domain/root.zone

When we run into an unknown resource record it’s not always easy to know how to handle it. Because of this, the library we use to parse the root zone at the edge does not make an attempt at doing so, and instead returns a parser error.

The static_zone app, tasked with loading and parsing the root zone for the purpose of serving the root zone locally (RFC 7706), stores the latest version in memory. When a new version is published it parses it and, when successfully done so, drops the old version. However, as parsing failed the static_zone app never switched to a newer version, and instead continued using the old version indefinitely. When the 1.1.1.1 service is first started the static_zone app does not have an existing version in memory. When it tries to parse the root zone it fails in doing so, but because it does not have an older version of the root zone to fall back on, it falls back on querying the root servers directly for incoming requests.

Initially we tried to disable the static_zone app through override rules, a mechanism that allows us to programmatically change some behavior of 1.1.1.1. The rule we deployed was:

phase = pre-cache set-tag rec_disable_static

For any incoming request this rule adds the tag rec_disable_static to the request. Inside the static_zone app we check for this tag and, if it’s set, we do not return a response from the cached, static root zone. However, to improve cache performance queries are sometimes forwarded to another node if the current node can’t find the response in its own cache. Unfortunately, the rec_disable_static tag is not included in the queries being forwarded to other nodes, which caused the static_zone app to continue replying with stale information until we eventually disabled the app entirely.

Cloudflare regularly performs rolling reboots of the servers that host our services for tasks like kernel updates that can only take effect after a full system restart. At the time of this outage, resolver server instances that were restarted between the ZONEMD change and the DNSSEC invalidation did not contribute to impact. If they had restarted during this two-week period, they would have failed to load the root zone on startup and fallen back to resolving by sending DNS queries to root servers instead. In addition, the resolver uses a technique called serve stale (RFC 8767) with the purpose of being able to continue to serve popular records from a potentially stale cache to limit the impact. A record is considered to be stale once the TTL amount of seconds has passed since the record was retrieved from upstream.  This prevented a total outage; impact was mainly felt in our largest data centers which had many servers that had not restarted the 1.1.1.1 service in that timeframe.

This incident had widespread impact, and we take the availability of our services very seriously. We have identified several areas of improvement and will continue to work on uncovering any other gaps that could cause a recurrence.

Here is what we are working on immediately:

Visibility: We’re adding alerts to notify when static_zone serves a stale root zone file. It should not have been the case that serving a stale root zone file went unnoticed for as long as it did. If we had been monitoring this better, with the caching that exists, there would have been no impact. It is our goal to protect our customers and their users from upstream changes.

Resilience: We will re-evaluate how we ingest and distribute the root zone internally. Our ingestion and distribution pipelines should handle new RRTYPEs seamlessly, and any brief interruption to the pipeline should be invisible to end users.

Testing: Despite having tests in place around this problem, including tests related to unreleased changes in parsing the new ZONEMD records, we did not adequately test what happens when the root zone fails to parse. We will improve our test coverage and the related processes.

Architecture: We should not use stale copies of the root zone past a certain point. While it’s certainly possible to continue to use stale root zone data for a limited amount of time, past a certain point there are unacceptable operational risks. We will take measures to ensure that the lifetime of cached root zone data is better managed as described in RFC 8806: Running a Root Server Local to a Resolver.

We are deeply sorry that this incident happened. There is one clear message from this incident: do not ever assume that something is not going to change!  Many modern systems are built with a long chain of libraries that are pulled into the final executable, each one of those may have bugs or may not be updated early enough for programs to operate correctly when changes in input happen. We understand how important it is to have good testing in place that allows detection of regressions and systems and components that fail gracefully on changes to input. We understand that we need to always assume that “format” changes in the most critical systems of the internet (DNS and BGP) are going to have an impact.

We have a lot to follow up on internally and are working around the clock to make sure something like this does not happen again.

We’re using the following IPv4 addresses for our resolver: 1.1.1.1 and 1.0.0.1. Easy to remember. These addresses have been provided to Cloudflare by APNIC for both joint research and this service. You can read more about their work via the APNIC blog.

DNS resolver, 1.1.1.1, is served by Cloudflare’s Global Anycast Network.

Our friends at DNSimple have made this amazing DNS Tutorial for anyone to fill in their gaps on how DNS works. They explain all about resolvers, root name servers, and much more in a very informative way.

When resolving a domain name, a query travels from your end system (i.e. a web browser) to a recursive DNS service. If the DNS record is not in the service’s local cache, the recursor will query the authoritative DNS hierarchy to find the IP address information you are looking for. The recursor is the part that DNS resolver, 1.1.1.1 plays. It must be fast and these days it must be secure!

Our goals with the public resolver are simple: Cloudflare wants to operate the fastest public resolver on the planet while raising the standard of privacy protections for users. To make the Internet faster, we are already building data centers all over the globe to reduce the distance (i.e. latency) from users to content. Eventually we want everyone to be within 10 milliseconds of at least one of our locations.

In March alone, we enabled thirty-one new data centers globally (Istanbul, Reykjavík, Riyadh, Macau, Baghdad, Houston, Indianapolis, Montgomery, Pittsburgh, Sacramento, Mexico City, Tel Aviv, Durban, Port Louis, Cebu City, Edinburgh, Riga, Tallinn, Vilnius, Calgary, Saskatoon, Winnipeg, Jacksonville, Memphis, Tallahassee, Bogotá, Luxembourg City, Chișinău) and just like every other city in our network, new sites run DNS Resolver, 1.1.1.1 on day-one!

Our fast and highly distributed network is built to serve any protocol and we are currently the fastest authoritative DNS provider on the Internet, a capability enjoyed by over seven million Internet properties. Plus, we already provide an anycast service to two of the thirteen root nameservers. The next logical step was to provide faster recursive DNS service for users. Our recursor can take advantage of the authoritative servers that are co-located with us, resulting in faster lookups for all domain names.

While DNSSEC ensures integrity of data between a resolver and an authoritative server, it does not protect the privacy of the “last mile” towards you. DNS resolver, 1.1.1.1, supports both emerging DNS privacy standards - DNS-over-TLS, and DNS-over-HTTPS, which both provide last mile encryption to keep your DNS queries private and free from tampering.

Historically, recursor sends the full domain name to any intermediary as it finds its way to the root or authoritative DNS. This meant that if you were going to www.cloudflare.com, the root server and the .com server would both be queried with the full domain name (i.e. the www, the cloudflare, and the com parts), even though the root servers just need to redirect the recursive to dot com (independent of anything else in the fully qualified domain name). This ease of access to all this personal browsing information via DNS presents a grave privacy concern to many. This has been addressed by several resolvers’ software packages, though not all solutions have been widely adapted or deployed.

The DNS resolver, 1.1.1.1, provides, on day-one, all defined and proposed DNS privacy-protection mechanisms for use between the stub resolver and recursive resolver. For those not familiar, a stub resolver is a component of your operating system that talks to the recursive resolver. By only using DNS Query Name Minimisation defined in RFC7816, DNS resolver, 1.1.1.1, reduces the information leaked to intermediary DNS servers, like the root and TLDs. That means that DNS resolver, 1.1.1.1, only sends just enough of the name for the authority to tell the resolver where to ask the next question.

The DNS resolver, 1.1.1.1, is also supporting privacy-enabled TLS queries on port 853 (DNS over TLS), so we can keep queries hidden from snooping networks. Furthermore, by offering the experimental DoH (DNS over HTTPS) protocol, we improve both privacy and a number of future speedups for end users, as browsers and other applications can now mix DNS and HTTPS traffic into one single connection.

With DNS aggressive negative caching, as described in RFC8198, we can further decrease the load on the global DNS system. This technique first tries to use the existing resolvers negative cache which keeps negative (or non-existent) information around for a period of time. For zones signed with DNSSEC and from the NSEC records in cache, the resolver can figure out if the requested name does NOT exist without doing any further query. So if you type wwwwwww dot something and then wwww dot something, the second query could well be answered with a very quick “no” (NXDOMAIN in the DNS world). Aggressive negative caching works only with DNSSEC signed zones, which includes both the root and a 1400 out of 1544 TLDs are signed today.

We use DNSSEC validation when possible, as that allows us to be sure the answers are accurate and untampered with. The cost of signature verifications is low, and the potential savings we get from aggressive negative caching more than make up for that. We want our users to trust the answers we give out, and thus perform all possible checks to avoid giving bad answers to the clients.

However, DNSSEC is very unforgiving. Errors in DNSSEC configuration by authoritative DNS operators can make such misconfigured domains unresolvable. To work around this problem, Cloudflare will configure "Negative Trust Anchors" on domains with detected and vetted DNSSEC errors and remove them once the configuration is rectified by authoritative operators. This limits the impact of broken DNSSEC domains by temporarily disabling DNSSEC validation for a specific misconfigured domain, restoring access to end consumers.

Initially, we thought about building our own resolver, but rejected that approach due to complexity and go-to-market considerations. Then we looked at all open source resolvers on the market; from this long list we narrowed our choices down to two or three that would be suitable to meet most of the project goals. In the end, we decided to build the system around the Knot Resolver from CZ NIC. This is a modern resolver that was originally released about two and a half years ago. By selecting the Knot Resolver, we also increase software diversity. The tipping point was that it had more of the core features we wanted, with a modular architecture similar to OpenResty. The Knot Resolver is in active use and development.

The recent advanced features we wanted were:

• Query Minimization RFC7816,

• DNS-over-TLS (Transport Layer Security) RFC7858,

• DNS-over-HTTPS protocol DoH,

• Aggressive negative answers RFC8198,

Small disclaimer: the original main developer of Knot Resolver, Marek Vavruša, has been working on the Cloudflare DNS team for over two years.

There are many factors that affect how fast a resolver is. The first and foremost is: can it answer from cache? If it can, then the time to answer is only the round-trip time for a packet from the client to the resolver.

When a resolver needs to get an answer from an authority, things get a bit more complicated. A resolver needs to follow the DNS hierarchy to resolve a name, which means it has to talk to multiple authoritative servers starting at the root. For example, our resolver in Buenos Aires, Argentina will take longer to follow a DNS hierarchy than our resolver in Frankfurt, Germany because of its proximity to the authoritative servers. In order to get around this issue we prefill our cache, out-of-band, for popular names, which means when an actual query comes in, responses can be fetched from cache which is much faster. Over the next few weeks we will post blogs about some of the other things we are doing to make the resolver faster and better, Including our fast caching.

One issue with our expansive network is that the cache hit ratio is inversely proportional to the number of nodes configured in each data center. If there was only one node in a data center that’s nearest to you, you could be sure that if you ask the same query twice, you would get a cached answer the second time. However, as there’s hundreds of nodes in each of our data centers, you might get an uncached response, paying the latency-price for each request. One common solution is to put a caching load balancer in front of all your resolvers, which unfortunately introduces a single-point-of-failure. We don’t do single-point-of-failures.

Instead of relying on a centralized cache, DNS resolver, 1.1.1.1, uses an innovative distributed cache, which we will talk about in a later blog.

Here’s the deal - we don’t store client IP addresses never, ever, and we only use query names for things that improve DNS resolver performance (such as prefill all caches based on popular domains in a region and/or after obfuscation, APNIC research).

Cloudflare will never store any information in our logs that identifies an end user, and all logs collected by our public resolver will be deleted within 24 hours. We will continue to abide by our privacy policy and ensure that no user data is sold to advertisers or used to target consumers.

See https://1.1.1.1/ because it's that simple!

We are grateful to APNIC, our partner for the IPv4 addresses 1.0.0.1 and 1.1.1.1 (which everyone agrees is insanely easy to remember). Without their years of research and testing, these addresses would be impossible to bring into production. Yet, we still have a way to go with that. Stay tuned to hear about our adventures with those IPs in future blogs.

For IPv6, we have chosen 2606:4700:4700::1111 and 2606:4700:4700::1001 for our service. It’s not as easy to get cool IPv6 addresses; however, we’ve picked an address that only uses digits.

But why use easy to remember addresses? What’s special about public resolvers? While we use names for nearly everything we do; however, there needs to be that first step in the process and that’s where these number come in. We need a number entered into whatever computer or connected device you’re using in order to find a resolver service.

Anyone on the internet can use our public resolver and you can see how to do that by visiting https://1.1.1.1/ and clicking on GET STARTED.

For most of the world, Sunday is 1/4/2018 (in America the day/month is reversed as-in 4/1/2018). Do you see the 4 and the 1? We did and that’s why we are announcing 1.1.1.1 today. Four ones! If it helps you remember 1.1.1.1, then that’s a good thing!

Sure, It’s also April Fools' Day and for a good portion of people it’s a day for jokes, foolishness, or harmless pranks. This is no joke, this is no prank, this is no foolish act. This is DNS Resolver, 1.1.1.1 ! Follow it at #1dot1dot1dot1

Photo by Niko Soikkeli / Unsplash

The root of the DNS tree has been using DNSSEC to protect the zone content since 2010. DNSSEC is simply a mechanism to provide cryptographic signatures alongside DNS records that can be validated, i.e. prove the answer is correct and has not been tampered with. To learn more about why DNSSEC is important, you can read our earlier blog post.

Today, the root zone is signed with a 2048 bit RSA “Trust Anchor” key. This key is used to sign further keys and is used to establish the Chain of trust that exists in the public DNS at the moment.

With access to this root Trust Anchor, it would be possible to re-sign the DNS tree and tamper with the content of DNS records on any domain, implementing an on-path DNS attack… without causing recursors and resolvers to consider the data invalid.

As explained in this blog the key is very well protected with eye scanners and fingerprint readers and fire-breathing dragons patrolling the gate (okay, maybe not dragons). Operationally though, the root zone uses two different keys, the mentioned Trust Anchor key (that is called the Key Signing Key or KSK for short) and the Zone Signing Key (ZSK).

The ZSK (Zone Signing Key) is used to generate signatures for all of the Resource Records (RRs) in a zone.

You can query for the DNSSEC signature (the RRSIG record) of “www.cloudflare.com” using your friendly dig command.

$ dig www.cloudflare.com +dnssec

;; QUESTION SECTION:
;www.cloudflare.com.		IN	A
;; ANSWER SECTION:
www.cloudflare.com.	4	IN	A	198.41.215.162
www.cloudflare.com.	4	IN	A	198.41.214.162
www.cloudflare.com.	4	IN	RRSIG	A 13 3 5 20180207170906 20180205150906 35273 cloudflare.com. 4W4mJXJRnd/wHnDyNo5minGvZY6hVNSXITnUI+pO6fzhnkpsEp1ko8K7 1PQ6r0s9SwLgrgfneqXyPs4b5X0YDw==

The two A records shown here can be cryptographically verified using the RRSIG and ZSK in the zone. The ZSK can itself be verified using the KSK, and so on… this continues upwards following the “chain of trust” until the root KSK is found.

The http://dnsviz.net/ tool can be used to help visualize how this verification can be done for any domain on the internet, for example here is the trust chain for “www.cloudflare.com”.

To verify the RRSIG on “www.cloudflare.com” we would need to cryptographically verify the signatures in reverse order on the diagram. First “cloudflare.com”, then “com”, and finally “.” – the root zone.

If you are able to access the secret key that’s used to sign the root, it’s possible to trick resolvers into verifying a "forged" answer.

While this DNSSEC signing has been deployed on the root zone, for over seven years, there is one operation that has never been attempted: rolling the Key Signing Key. This means to generate a new key and update every part of DNS infrastructure on the internet that needs it, retiring the old one completely.

The ZSK (Zone Signing Key) has been rolled religiously every quarter since 2010, however rolling the Key Signing Key is a much scarier operation. If it goes wrong it could leave the root zone signing invalid, meaning a large part of the internet would not trust any of the content, effectively knocking DNS offline for validating resolvers. After DNSSEC was designed, a mechanism was devised for rolling out a new Key Signing Key in RFC5011, this operation is commonly known as the 5011 roll-over.

All cryptographic keys have a life cycle that can represented by states:Generated == the key is created but only the “owner” knows of its properties.Published == the key has been made public either as a public key or a hash of it.Active == the key is in useRetired == the has been withdrawn from service but is still publishedRevoked == they key has been marked as not to be trusted ever again.Removed == taken out of publication

Different keys move through the states in different ways depending on the usage, for example some keys are never revoked, just removing them is sufficient, for example the root ZSK’s are never revoked. When rolled, the root KSK will pass through all states.

For most keys used in DNS the trust is derived by a relationship between the parent zone and the child zone. The parent publishes a special record, the DS (Delegation Signer), that contains cryptographically strong binding to the actual key, a hash. The child has a DNSKEY RRset at the top of its zone that has at least one key that matches one of the DS records in the parent. To complete the chain of trust the DNSKEY RRset MUST be signed by that key.

The root zone has no parent, thus trust cannot be derived in the same way. Instead, validating resolvers must be configured with the root Trust Anchor. This anchor must be refreshed during a key rollover or the validating resolver will not trust anything it sees in the root zone after the old KSK (from 2010) is retired from service. The Trust Anchors can be updated in a number of ways, such as a manual update, a software update, or an in-band update. The preferred update mechanism is the previously mentioned in-band update mechanism RFC5011-roll.

The process outlined in RFC 5011 relies on two factors, first that the new key is published in the DNSKEY RRset – which is signed by the old KSK, and is kept there for at least a hold-down period of 30 days. Validating resolvers that follow the procedure will check frequently to see if there is a new KSK in the DNSKEY set. The new key can be trusted because it has been signed with a key that is already in service. When there is new key, it is placed in PendingAddition state If at any point one of the key’s in PendingAddition is removed from the DNSKEY set, the resolver will forget about it. This means that if the key were to appear again, it would start a new 30 day hold-down period.

After the key has been in PendingAddtional for 30 consecutive days it is accepted into Active state and will be trusted to sign the DNSKEY set for the root. From this point onwards, the new key can be used to sign the Zone Signing Key, and in turn the root zone content itself.

There are two main reasons;

1. The community wants to be a sure that the RFC5011 mechanism works in practice. Knowing this makes future rollovers possible, and less risky. Regular rollovers are something to be done as a matter of good key hygiene, like changing your password regularly

2. Enables thinking about switching to different algorithms. RSA with a large key size is a strong algorithm, but using it causes DNS packets to be larger. There are other algorithms like the ones that Cloudflare uses that are based on elliptic curves have smaller keys but increased safety per bit. To switch to a new algorithm would require a new key.

Some people advocated rolling the key and changing the algorithm at the same time but that was deemed too risky. The right time to start talking about that is after the current roll concludes successfully.

ICANN started the rollover process last year. The new keys has been created and replicated to all the HSM’s (Hardware Security Modules) in the two facilities that ICANN operates. From now on we will use the terms KSK2010 (the old key) and KSK2017 (the new key).

Before starting the roll-over process, testing of RFC5011 implementations took place and most implementations reported success.

The new key was published in DNS on July 11th 2017, thus the DNSKEY set now contains two KSKs. At that point the new key/KSK2017 has entered Published state. It was scheduled to become Active on October 11th 2017. Any validating resolver that has been operating for at least 30 days during the July 11-October 11 window should have placed the new Trust Anchor in “Active” state before October 11th. But sometimes things do not go according to plan.

One of the things that was put in place before the rollover was a way for resolvers to signal to authoritative servers what trust anchor the resolver trusts RFC8145. RFC8145 was only published in April 2017, thus during the KSK2017 key publication phase, only the latest version of Bind-9 supported it by default.

The mechanism works by resolvers periodically sending a query to the root nodes, with a query name formatted like “_ta-4a5c” or “_ta-4a5c-4f66”. The name contains HEX encoded versions of the Trust Anchor identifiers, 19036 and 20326 respectively. This at least allows root operators to estimate the % of resolvers that have implemented RFC8145 AND are aware of each Trust Anchor.

On September 29 ICANN postponed the roll based on evidence from the resolvers that sent in reports.It was concerning that the latest and greatest version of Bind-9 in 4% of cases did not pick up the new Trust Anchor, this was explained in more detail in a DNS-OARC presentation. But this still leaves us with the question, why?

It is also important to note that although other implementations of RFC8145 did not enable it by default thus most of the reports were by Bind-9.

Rolling the KSK at this point would have resulted in the remaining resolvers not trusting the content of the root zone, ultimately breaking all DNS resolution through them.

At Cloudflare we operate validating resolvers in all of our >120 data centers, and we monitored the adoption of trust anchors on a weekly basis, expecting everything to work correctly. After 6 weeks we noticed that things were not going right, some of the resolvers had picked up the new trust anchor and others had not accepted the new trust anchor even though more than enough time had passed.

First let’s look at the assumptions that RFC5011 makes.

• The resolver is a long running process that understands time and and can keep state

• The resolver has access to persistent writeable storage that will work across reboots.

In the protocol community we had worried about the first one a lot, for the second one we had identified two failure cases: machine configured from old read-only medium, and new machine takes over. Both were considered rare enough and operators would know to deal with those exceptions.

Turns out the second assumption in RFC5011 had more failure modes than the community expected.

For example in Bind-9, it originally had a hardcoded list of “trusted-keys”. Later when RFC5011 support was added the configuration option “managed-keys” was added. It looks like some installations while religiously updating the software never changed from the fixed configuration to the RFC5011 managed one. In this case the only recovery is to change the configuration, and in some cases the operator selected this operating mode assuming he/she would distribute a new configuration file during rollover, but the person may have left or forgotten.

Software that uses managed-keys operations (Bind-9, Unbound, Knot-resolver) uses a file to maintain state between restarts. BUT it is possible that the file is read-only and in that case managed-keys works just like trusted-keys. Why anyone would have a configuration like that is a good question? The interesting obersevation is that unless the implementation complains loudly about the read-only state, the operator is not likely to notice. The only recovery option here is to change the configuration so the trust anchor file can be written.

Software upgrades are another possible reason for not picking up the new trust anchor, but only if the file containing the Trust Anchor state is overwritten or lost. This can happen if the resolver machine has a disk replacement/reformat etc. but in this case the net effect is only slowing down the acceptance of the new trust anchor. This failure is visible as as KSK2017 spends more than 30 days in state “PendingAddition” but that is only visible if someone is looking.

Modern operating practices use “containers” that are spun up and down, in those cases there is no “persistent” storage. To avoid validation errors in this case the software installed must know about the new key or perform a key discovery upon startup like the unbound-anchor program performs for Unbound.

There are probably few other reasons where operations may cause the errors seen by the Trust Anchor Signaling.

Back to what happened at Cloudflare? In our case the issue was a combination of upgrades and container issues. We were upgrading software on all our nodes and our resolver processes were allocated to different computers. Our fix was to quickly upgrade to a software version that knew about the new trust anchor, so future restarts/migrations would not cause loss in trust.

ICANN has just asked for comments on restarting the rollover process and perform the roll on October 11th 2018.

What can you do to prepare for the key rollover?If you operate a validating resolver, make sure you have the latest version of your vendors software, audit the configuration files and file permissions and check that your software supports both KSK2010 (key tag 19036) and KSK2017 (key tag 20326).

If you are a concerned end user right now there is nothing you can do, but the IETF is considering a proposal to allow remote trust anchor checking via queries. Hopefully this will be standardized soon and DNS resolver software vendors add support, but until then there is no testing possible by you.

If you speak languages other than English and you worry about your local operators should know about the DNSSEC Key Rollover failure modes, feel free to republish this blog or parts of it in your language.

Every quarter there is a new KSK signing ceremony where signatures for 3 months of use of the KSK are generated. February 6th 2018 is the next one and it will sign a DNSKEY set with both KSKs but only signed by KSK2010 . You can see the script for the ceremony here and you can even watch it online. But the fun part of this particular ceremony is the destruction of old HSM (Hardware Security Module), via some fancy contraption.

An HSM is a special kind of equipment that can store private keys and never leak them, and protects its secrets by erasing them when someone tries to access/tamper with the equipment. The secrets remain in the HSM as long as a non-replaceable battery lasts. The old KSK HSMs have a lifetime of 10 years and were made in late 2009 or early 2010 thus the battery is not designed to last much longer. Last year the private keys were safely and securly moved to newer models and the new machines have been in use for about a year. The final step of retiring the old machines is to destroy them during the ceremony, tune in to see how that is done.

Excited by working on cutting edge stuff? Or building systems at a scale where once-in-a-decade problems can get triggered every day? Then join our team.

1. The lack of legitimate ANY use.

2. The abundance of malicious ANY use.

3. The constant use of ANY queries in large DNS amplification DDoS attacks.

Additionally, we were about to launch Universal DNSSEC, and we could foresee the high cost of assembling ANY answers and providing DNSSEC-on-the-fly for those answers, especially when most of the time, those ANY answers were for malicious, illegitimate, clients.

Although we usually make a tremendous effort to maintain backwards compatibility across Internet protocols (recently, for example, continuing to support SHA-1-based SSL certificates), it was clear to us that the DNS ANY query was something that was better removed from the Internet than maintained for general use.

Our proposal at the time was to return an ERROR code to the querier telling them that ANY was not supported, and this sparked a robust discussion in the DNS protocol community. In this blog post, we’ll cover what has happened and what our final plan is.

Just before we published our blog a popular software started using ANY queries, to get all address records for a name -- something that ANY isn’t actually designed to do. The effect of this software was that our steady ANY query load had grown from few hundred per second to tens of thousands in a matter of days. Luckily, the software in question issued a revised version that did not use ANY, and our steady ANY query load returned to the old level.

CC BY-SA 2.0 image by Liam Quinn

As that was happening, a lively discussion started to form among those in the DNS community. The first fundamental question that had to be answered was: “Does ANY mean ALL?”. That is, was an ANY query meant to be a way to receive all of the records in a zone for the query name?

Different people had different interpretations of ANY depending on the kind of DNS service they were providing. For example, it is nice as an operator of a DNS resolver to be able to ask your own resolver, “what is stored in the resolver memory/cache for a particular name?”. An ANY query with the right query flags can help answer that. On the other hand, an operator of an authoritative server does not need that functionality, as AXFR provides another reliable way to get that information. In short, the ANY query is a nice tool for people that are trying to understand what is going on in DNS resolution. Some community members argued that ANY should be a restricted query to a privileged few that have the need to know.

While ANY can be a nice tool to debug and expose inconsistencies in resolver caches, it’s still not a great tool: with more and more Anycast resolver clusters, there is no guarantee that two subsequent queries will hit the same resolver.

Our in-house DNS server is optimized to provide dynamic answers to questions. For example, depending on how a CNAME is configured, our server may return the CNAME, fetch the real answer from the target of the CNAME (what we call CNAME Flattening), or provide CloudFlare addresses as answers. Thus for us to answer an ANY query, we need to compute all of the possible combinations just to know what to return.

Beyond that, our DNSSEC implementation signs answers on-the-fly. Thus, returning an answer with many different types of DNS records in the answer requires signing all of them at the edge. Thus providing support for ANY in the “traditional” sense had serious computational and response time implications. This is not unique to CloudFlare; we are not the only DNS implementation that has this high cost factor in answering ANY queries.

In a recent paper by Akamai the authors draw the conclusion that DNSSEC is the main cause of large answers used for DDoS attacks. But looking at the packet capture they included in the paper, it’s clear the real cause of the large answers is that the attackers use ANY queries to maximize the amplification factor.

CC BY-SA 2.0 image by Vladimer Shioshvili

We regularly see attacks that attempt to use our powerful DNS system as a source of reflection. We have in response to this created sophisticated systems to detect the attacks and mitigate them in a responsible manner. Our deprecation of ANY is a key part of those protections. One of our main mantra is “do not return larger than needed answers”, exactly to help protect others on the Internet from amplification attacks.

Soon after we announced our planned deprecation of ANY, we submitted an Internet Draft to the IETF proposing to restrict ANY queries to authorized parties only. In the resulting discussion, it became clear that what mattered here was how we deprecated ANY. To make sure no system was adversely affected, we had to take into account how various applications and DNS implementations were using ANY.

In short, there are two main uses of ANY queries:

1. Some programs use ANY as a probabilistic optimization attempt to get the answers they need.

2. Some use ANY to debug DNS resolvers when things go wrong.

However, neither of those use cases are satisfied by the current ANY landscape. There is a lack of common behavior among resolvers as to how they treat answers that are cached as a result of an ANY query. Some will return this data when a more specific query matches, while other resolvers will fetch the exact requested data, even though they already have it in cache from an ANY query. This is because resolvers interpret RFC2181 section 5.4.1 (Ranking Data) in differing ways, and some resolvers were written without applying the rules from RFC2181 at all. Some resolvers will not forward ANY queries to authoritative servers if there exists a single RRset for that name in the resolver’s cache, and others will forward it unless they have a prior ANY answer in the cache. Furthermore, some resolvers will only reuse the results of an ANY query to answer other ANY queries, queries for all other types will result in a direct query for the type requested.

In many cases, the ANY query results in an answer that is too large to fit in the UDP packet size requested, resulting in a truncated answer, leading to a follow up via TCP. For a while, the DNS community believed that returning truncated answers would stop attacks, but in reality, that will only mitigate simple attacks using forged packets. In attacks that are reflected via open resolvers, returning truncated packets will not work because the open resolvers are happy to fall back to TCP if the UDP answer sets the truncate bit.

So, there is no common understanding of how the ANY query should be treated to minimize its amplification potential. To be fair, the ANY query is a special type of query called a meta-query i.e. it is not an actual type. Nevertheless, the community was divided into two camps: “ANY == ALL” and “ANY != ALL”.

The community was also further divided into groups of “ANY is ok for everyone” and “ANY should be restricted to “good” clients”. Our position from the beginning was that “ANY != ALL” and we were looking for a way to help curb the number of large amplified attacks on the Internet that used ANY.

Over a few months, we engaged in a number of experiments to see how different DNS systems reacted to different non-ANY responses to the ANY query. After a fair amount of experimentation and discussions with colleagues around the world, we decided on an approach that is recursive resolver centric. What we wanted to do was to give answers that are friendly to recursive resolvers, i.e. we give them something small that they can cache and return to repeated ANY queries. Returning an error to a recursive resolver was not a good option, as the resolver will just ask the next authoritative server and visit all the authoritative servers before giving up.

We also wanted to avoid guessing the intention of the originator of the query, which is why we did not follow one proposal to give out the A+AAAA+MX records or a CNAME if one existed. We do not like that, as the answer is bigger than it has to be and there is more data in the answer than the originator wanted.

For example, consider an email server that wants an MX record if one exists, but will fallback to an IP address if the MX does not exist. Instead, we decided to return what we call a “harmless” answer––an answer that is not useful for any application on the Internet. We selected an old DNS record type that is not used much anymore, but has the nice property that all test tools display as text: HINFO. This approach is documented in the current Internet Draft Refuse ANY draft that was adopted by the DNSOP working group of the IETF that handles DNS protocol issues.

As you can see below, when asked for ANY, we only return one HINFO record and the optional RRSIG that is only needed when the zone is signed. This record can be cached, and has the added benefit of being small and therefore reducing the amplification factor the attacker expected.

; <<>> DiG 9.8.3-P1 <<>> @ns2.p31.dynect.net. amazon.com. any +dnssec +norec
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29671
;; flags: qr aa; QUERY: 1, ANSWER: 16, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;amazon.com.			IN	ANY

;; ANSWER SECTION:
amazon.com.		900	IN	SOA	dns-    external-master.amazon.com. root.amazon.com. 2010113317 180 60 3024000 60
amazon.com.		3600	IN	NS	pdns6.ultradns.co.uk.
amazon.com.		3600	IN	NS	pdns1.ultradns.net.
amazon.com.		3600	IN	NS	ns2.p31.dynect.net.
amazon.com.		3600	IN	NS	ns3.p31.dynect.net.
amazon.com.		3600	IN	NS	ns1.p31.dynect.net.
amazon.com.		3600	IN	NS	ns4.p31.dynect.net.
amazon.com.		60	IN	A	54.239.26.128
amazon.com.		60	IN	A	54.239.25.208
amazon.com.		60	IN	A	54.239.25.200
amazon.com.		60	IN	A	54.239.17.7
amazon.com.		60	IN	A	54.239.17.6
amazon.com.		60	IN	A	54.239.25.192
amazon.com.		900	IN	MX	5 amazon-smtp.amazon.com.
amazon.com.		900	IN	TXT	"spf2.0/pra include:spf1.amazon.com include:spf2.amazon.com include:amazonses.com -all"
amazon.com.		900	IN	TXT	"v=spf1 include:spf1.amazon.com include:spf2.amazon.com include:amazonses.com -all"

;; Query time: 30 msec
;; SERVER: 204.13.250.31#53(204.13.250.31)
;; WHEN: Wed Apr 13 09:57:59 2016
;; MSG SIZE  rcvd: 565

Unsigned answer from large internet company is 565 bytes long.

In contrast a signed answer from CloudFlare.com is only 224 bytes long or less than ½ the unsigned answer above.

; <<>> DiG 9.8.3-P1 <<>> cloudflare.com any @ns3.cloudflare.com +dnssec +norec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36238
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;cloudflare.com.			IN	ANY

;; ANSWER SECTION:
cloudflare.com.		3789	IN	HINFO	"Please stop asking for ANY" "See draft-ietf-dnsop-refuse-any"
cloudflare.com.		3789	IN	RRSIG	HINFO 13 2 3789 20160414100147 20160412080147 35273 cloudflare.com. lGyCY7IC5sgHBfE95IJXDUS4diFjE5kq4vNMhhqP6+2+NyTQh1zAh1qw 3C710mFvvuCWe4VyRiqlu1jUzMnuLg==

;; Query time: 80 msec
;; SERVER: 2400:cb00:2049:1::a29f:21#53(2400:cb00:2049:1::a29f:21)
;; WHEN: Wed Apr 13 10:01:47 2016
;; MSG SIZE  rcvd: 224

We have been returning the HINFO answer for ANY queries since October 2015, with very few reports of problems in the field, besides a single Twitter rant about us not understanding DNS.

A few other DNS server vendors and DNS operators have followed our lead or adopted a similar line of defense. The latest example is the University of Cambridge which modified their BIND implementation to only return a single RRset for an ANY query. A common DNS server (NSD) has also been limiting ANY responses to only A+AAAA+MX types and/or CNAME, and a patch to return an even smaller answer has been proposed.

The moral is that the ANY query is not a useful tool for most DNS operators, but it is a wonderful tool if one is in the business of generating attacks against anyone else on the Internet. Answering ANY queries with a giant answer is not helping mitigate the plague of DoS volume attacks.

CloudFlare has taken a step to make the Internet a less hostile place and is leading by example. We strongly urge others to follow in our steps and neuter the amplification factor that a single DNS query can achieve. We all want to build a safer Internet, and neutering ANY query is one small step that everyone can take.

CC BY-ND 2.0 image by Antarctica Bound

New DNS Resource Records types are being added all the time. As the Internet evolves, new RR’s gain traction while the usage of some old record types decreases. Did you know you can use DNS to express the location of your server on the planet's surface?

Today, we are announcing that we are deprecating the DNS ANY meta-query. In a few weeks we'll be responding to those queries with rcode 4 / Not Implemented.

“ANY” is one of the special “magic” types in DNS. Instead of being a query for a single type like A , AAAA or MX, ANY retrieves all the available types for a given name. Over the years there have been many arguments over the semantics of ANY with some people arguing it really means ALL. Answers to ANY queries are among the biggest that DNS servers give out. The original reason for adding the ANY to DNS was to aid in debugging and testing and there is no real reason why a normal application would ever issue a ANY query.

ANY is not the only operation that lists records in DNS. The AXFR operation could be used to transfer all data for a DNS zone between two servers. A frequently recommended option is to restrict the scope of IP addresses that can perform this action and return REFUSED or drop the query for all others. The reason for this is security: people can easily learn the entire DNS zone with one command. In many case it is not okay for anyone to list all entries in your DNS zone. We believe the same holds for ANY and that an authoritative server should be allowed to refuse to answer it.

ANY queries are not widely used by any real world software. We aware of only two programs that issue ANY queries:

• Un-patched versions qmaild

• Firefox version 36.0 to 36.0.1

We strongly believe using ANY query type in both those cases was a mistake and should not be endorsed.

The usage of ANY by a Firefox caught us by surprise and increased the ANY queries load 8 fold:

Since Firefox reverted the use of ANY we expect this load to drop back to the level of a background noise. Unfortunately the most common users of ANY queries in practice are people trying to perform DNS reflection attacks, exploiting the unusual length of the ANY responses.

Disabling or throttling ANY is not unprecedented. UltraDNS disabled them briefly in 2013 with little impact visible to Internet users. A number of operators have refused to answer ANY queries over UDP, forcing the traffic to TCP, with the side effect that forged ANY queries are not amplified. Similarly a number of DNS operators use QoS techniques to limit how many ANY questions they will answer per second.

Attempting to handle ANY queries creates enormous complexity in our DNS server code base. It's almost impossible to generate a proper response, anyway. Consider load-balancing, geoip, CNAME flattening features, and on-the-fly answer generation.

Due to the lack of justified uses and to avoid code complexity, we have decided to completely phase out attempting to answer meaningfully the ANY qtype. In the near future we plan on returning NOTIMP return code in response to all ANY queries to our authoritative servers. This is the most truthful answer we can give as the code to process ANY type will be removed.

If you are aware of any software that relies solely on ANY queries, now is the time to fix it.

In this blog, we’ll introduce a significant problem that DNS operators like CloudFlare face when trying to provide the best possible experience to our customers. If you are a CloudFlare customer, you’ll remember during the sign up process you were asked to login to your registrar account in order to change your nameservers (NS). The absence of an automated process for changing NS records not only makes our signup process one step longer than we’d like, it also prevents CloudFlare, and other 3rd party DNS operators, from doing a slew of other things that would benefit customers and the Internet as a whole.

Note: In this blog we’ll use the term DNS Operator mainly in the context of operators that provide Authoritative DNS service. This is sometimes called Managed DNS service.

For those who are not yet CloudFlare customers, let’s run through the sign up process:

When CloudFlare customers enable our DNS services for their domain, we allocate and provide them with nameservers. After the customer configures various records within their domain (e.g., A, AAAA, MX, CNAME, etc. records) on the CloudFlare system, customer’s then need to go back to their Domain Registrar and manually update their NS records so they match the NS information provided by CloudFlare. Once the NS records have been changed, CloudFlare becomes the authoritative DNS operator for that zone.

This manual process is outdated, and there is only one thing standing in the way of an automated process—the current domain industry registration model.

The domain registration system includes Registrants (Resellers and Registrars), Registries, and ICANN, and there are strict rules about how information flows through it:

Note: A full glossary of the terms used in this document is available from the ICANN website.

Notice that DNS Operators are not included in the ICANN diagram above. When this model was created, no one one thought that DNS service might be provided by someone other than Registrant and Registrar. Things have changed, but unfortunately, the system and its rules haven’t.

In nearly all cases, CloudFlare customers are using a Registrar where CloudFlare’s relationship with the customer is not explicitly expressed. The relationship can only be inferred by the NS records that point towards CloudFlare, or by the fact that nameserver addresses that are in CloudFlare address spaces. This omission of 3rd party DNS operators in the ICANN model causes operational difficulties.

Operational difficulties arise currently because the only way to find the Registrar of a domain name is to query for the “whois” information. Below is an selected output from a successful whois query:

Domain Name: CLOUDFLARE.COM ... Registrar WHOIS Server: whois.networksolutions.com Registrar URL: http://networksolutions.com ... Registrar: NETWORK SOLUTIONS, LLC. Registrar IANA ID: 2 ... Registrant Name: CloudFlare, Inc.

All domain updates (such as postal address or name server changes) go from the customer (or Registrant) to the Registrar before being sent to the Registry. As a DNS Operator, CloudFlare is separate from these R-named entities so we don’t show up in a whois query.

The difficulty is that whois information, which may contain postal addresses, telephone numbers, and/or email addresses, is designed for human consumption and human action. Because this information has historically been changed by people, there isn’t a protocol specified regarding how a DNS operator’s system can ask a registrar for changes in delegation information in an authenticated automated way.

In the current ICANN system there are roughly three classes of DNS operators. These classes are based on the ability of DNS operators to make changes in the delegating parent zone:

Registrars/Resellers—Have direct interface to the registry database, and can change information at will and instantly.

Registrant—Have a User Interface (usually web) to update information

Third Party DNS Operator—Need either the registrant to update information on its behalf, or have access to the registrants account with registrar (which is a bad security practice). In reality, DNS operators can only expect registrants to log into the account on two occasions: a. when service is moved to Operator, or b. when service is taken away from operator.

CloudFlare is advocating to gain the ability to update NS records for our customers and address records associated with them using automated channels. Our goal is to be able to add and remove nameservers from customer domains without the customer being involved.

Creating an automated process for updating NS (and DS) records would help solve the operational difficulties to providing DNS service, and would open up new possibilities for the Internet as a whole. If DNS operators had control of NS and DS records they could re-balance nameservers for stability, quickly change nameservers that go bad, and even better protect customers against DDoS attacks. Most people don’t know that when certain NS records come under heavy DDoS attack, all customers that share that nameserver might also experience a degradation in service. If those NS records could be changed quickly, we could lessen the impact on domains that are not being targeted.

Perhaps the most important reason for automating DNS operator’s access customer NS and DS records (Delegated Signer Record) is that this change would pave the way for DNSSEC implementation. DNSSEC requires maintaining DS records because they have to be inserted into the parent domain and potentially updated on a regular basis. If this record is not properly maintained, then DNSSEC validation fails, making the domain inaccessible. Presently, this is done through a web interface at the Registrar by the Registrant.

In theory, the Registrant could designate the DNS Operator as Technical Contact, but that doesn’t help unless the Technical Contact is given full access to the Registrant’s account since most Registrars don’t provide role-based access to accounts. In any case, asking our customers to update their delegation information in order to reflect the DNSSEC trust chain is problematic. One issue with this approach is that if CloudFlare were to ask hundreds of thousands of customers that own millions of domains to make manual updates to their records, there would be a huge chance of error. If records are updated incorrectly, it would not only cause frustration, it might cause the site to go down due to DNSSEC errors.

CloudFlare could try to minimize these errors by publishing CDS and or CDNSKEY records in each zone, that the registrar can pick up via DNS query and apply. But the long term solution is full automation with authorized updates to delegation information.

Some will say that the current system will work if the DNS operator is designated as Technical Contact (one of the roles defined in the ICANN model) but almost no Registrar offers a role based accounts for customers. All that does is to decrease the probability that phone call or email from Technical contact is dismissed as social engineering attack.

CloudFlare wants to team up with Registrars, Registries, and other DNS Operators to define and deploy more reliable methods for updating NS and DS records. We think this would be a big win for our customers, and, ultimately, for the internet as a whole. If you’re interested in participating in this process, you can sign up for this mailing list: dnssec-auto-ds@elists.isoc.org

This blog post is probably more personal than the usual posts here. It’s about why I joined CloudFlare.

I’ve been working on DNSSEC evolution for a long time as implementor, IETF working group chair, protocol experimenter, DNS operator, consultant, and evangelist. These different perspectives allow me to look at the protocol in a holistic way.

First and foremost, it’s important to realize the exact role of DNSSEC. DNSSEC is actually a misnomer: it’s from an era when the understanding of different security technologies, and what role each plays, was not as good as today. Today, this protocol would be called DNSAUTH. This is because all it does is to provide integrity protection to the answers from authoritative servers.

Over the years, the design of DNSSEC has changed. A number of people working on early versions of DNSSEC (myself included) didn’t know DNS all that well. Similarly, many DNS people at the time didn’t understand security, and in particular, cryptography all that well. To make things even more complex, general understanding of the DNS protocol was lacking in certain areas and needed to be clarified in order to do DNSSEC properly. This has led to three major versions of the protocol. The first two were not deployable for various reasons. Some of the decisions made, in hindsight, were sub-optimal. They were artifacts of constraints placed on the design by the DNS protocol itself, understanding of DNS, and various operational realities. DNSSECv3 [RFC403x], however, is deployable.

Today, we have wide spread deployment of the crucial building blocks for DNSSEC:

• Root and TLDs: over 2/3’rds of the TLDs are signed

• Most DNS software is DNSSEC enabled

• Many registrars offer their customers DNSSEC support either by signing the customers zones or by adding DS records into the parent zones.

What’s missing is having Enterprise zones signed, and turning on validation in resolvers and clients. It’s estimated that over 10% of all user DNS answers today are validated, but how much is validated in data centers is unknown.

DNSSEC deployment has been what I call a game of “excuse elimination”. First it was “com will never be signed”, then “the root will never be signed”, then “the answers will be too big”, and so on. Right now, the main excuse is, “this important domain is not signed”. Getting CDNs to sign is a great step towards getting the important domains signed. This is because CDNs frequently act as DNS operators for such domains.

So what does all of this have to do with me joining CloudFlare? Well, when a friend mentioned to me that CloudFlare was looking for a DNS person I checked them out. CloudFlare impressed me by wanting to do things correctly right from the beginning, and people here are not afraid to do things differently if it’s better. CloudFlare had been thinking about doing DNSSEC, and wanted me to help them implement and deploy it. This is turning out to be a fun project not just because of the scale, but also because of the ability to take a fresh perspective and questioning all prior assumptions.

CloudFlare’s DNS servers provide answers from over 30 anycasted data centers all over the world. We operate lots of DNS servers—authoritative for millions of zones. Not all the servers return the same answers to all clients because of policies and locations. Furthermore, much of the DNS data we serve has geographical bias. Thus, some data centers never see a query for that data. In this environment the only realistic way to answer the question is to generate the signatures at the edge on demand. This is a radical departure from most DNSSEC implementations, but there are few implementations like PowerDNS that have this capability. What online signing does is significantly reduce the volume of data that has to be transferred to the edge. We’ve designed our systems to only transfer signed DNSKEY (and CDS) records to the edges, while everything else is signed there. This requires transferring the zone signing keys to the edge.

CloudFlare is a frequent target of DNS attacks, both against our customers and as an reflector/amplifier. For that reason, we are fanatical about keeping DNS answers as small as possible to minimize the damage our systems can do to others when used as a reflector. This has directed us in a number of choices on how we do DNSSEC.

First, we use the Elliptic Curve algorithm ECDSA P-256. A ECDSA key is stronger than most RSA keys used today and the signatures are much smaller. Also, it takes fewer CPU cycles to generate the signatures than with RSA making this is a double win for us. When we started on the project, we found only one Validating Resolver implementation that did not support ECDSA. We reached out to them and now Google Public DNS correctly validates ECDSA!

Second, we do negative answers in a special way. Negative answers in DNSSEC can get large. For zones signed with NSEC, it’s not uncommon to have SOA + RRSIG(SOA) + 2 NSEC records + 2 RRSIG(NSEC) records in the negative answers. Even for the weakest RSA keys allowed, this results in an answer that is at least 635 bytes. NSEC3 signed answers require, in most cases, 3 NSEC3 and 3 RRSIG (NSEC3) records to deny the existence of the item asked for—that’s at least 1000 bytes. So we selected NSEC as our negative answer and use ECC keys. But the biggest saving comes from not having to prove that the covering wildcard exists at all, which is the role of the second NSEC record. We return an answer that says, “sure, the name exists, but the type you asked for does not”. This allows us to return only one NSEC record in negative answers!

In the past, NSEC records have been criticized for leaking information about the zone contents. Our implementation of negative answers allows us to provide negative answer with no value for a zone walker. Thus, our customers will gain the best possible defense against zone walking.The net result of our careful engineering of DNS answers is that we are able to keep most of our signed answers under 512 bytes. There are, however, exceptions, like when customers have large answers or long names but that is unavoidable.

Today's announcement regarding CloudFlare's alpha DNSSEC support is the first step towards providing a comprehensive DNSSEC offerings to our customers. We plan on offering DNSSEC to all our customers soon.