A string of DNS protocol bugs.

I went to turn on DNSSEC for cqx.ltd.uk today - the server that signed it broken right before my Christmas busy period so I disabled DNSSEC on that zone until I got round to fixing it.

I've encountered three different apparent protocol implementation bugs in the space of a few hours:

• Andrews and Arnold's web based control panel accepts DS records as generated by BIND's dnssec-keygen tool but then throws a complicated looking error when talking to Nominet, the UK domain registry, to put those records where they need to be. As far as I can tell, this is because the BIND output has whitespace in the middle of a hex string, something RFC 4034 s5.3 seems to think is acceptable. Why is installing crypto keys always so hard?
• For a while, Hetzner's recursive resolvers were unable to verify (and therefore refused to answer) results for my zone. I have a suspicion (but I don't have much to go on other than a hunch) that this was something to do with DS records and the actual zone having some kind of mismatch - although Google Public DNS at 8.8.8.8, and Verisign's DNSSEC checker both worked ok.
• I discovered an implementation quirk in the Haskell dns library, which I use inside a debugging tool I'm slowly building. This is to do with the mechanism which DNS uses to compress replies: where a domain name would be repeated in a response, it can be replaced by a pointer to another occurence of that name in the reply. It looks like in this case that the dns library will only accept those pointers if they point to regions of the reply that have specifically already been parsed by the domain name parsing code, rather than pointers to arbitrary bytes in the reply. This is frustratingly familiar to another bug I encountered (at Campus London) where their (not-so) transparent firewall was reordering DNS replies; giving a bug that only manifested when I was sitting in their cafe. (github issue #103)

dnssec revisited

well I was pleased that my dnssec notes remained sufficiently intelligble after a few months to set up a new zone. but the machine I was just using took almost 5h to generate keys for a new zone. oh the entropy.

sshfp dns

I've set up DNSSEC, so I'm on the path to trusting DNS more. I can put SSH key fingerprints for my hosts into DNS, and SSH clients can check those.

Even with insecure DNS, this is probably better than what you do now, which is to just to choose 'yes' to the following prompt without actually checking: (seriously, do you ever bother?)

The authenticity of host 's0.barwen.ch (192.168.55.55)' can't be established.
RSA key fingerprint is 9e:81:ab:cb:2a:ad:26:2f:10:ed:dd:5c:55:dd:ea:58.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? 

SSH can check s0's DNS record to see if a fingerprint is stored there, and tell you if it matches. So lets set that up.

I need to add an SSHFP record to the DNS for s0.barwen.ch.

On host fubar (without needing to be root):

$ ssh-keygen -r s0.barwen.ch
s0.barwen.ch IN SSHFP 1 1 560f08c1687a60e62a65ef427e63698ae1797d6f
s0.barwen.ch IN SSHFP 2 1 4ef38fd457d0afec50ca21eacb771f724e6d7236

So those are the records to add to barwen.ch's DNS.
(btw, vim on my machine doesn't like SSHFP records and highlights everythign red - eww)

Now wait for DNS to settle, and when I connect for the first time, I get a different message (my emphasis).

The authenticity of host 's0.barwen.ch (192.168.55.55)' can't be established.
RSA key fingerprint is 9e:81:ab:cb:2a:ad:26:2f:10:ed:dd:5c:55:dd:ea:58.
Matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? 

Cool.

You might need to set the client option VerifyHostKeyDNS ask in your ~/.ssh/config - if you really trust DNS, you can set it to yes instead, and it won't even ask you when there's an SSHFP record present.

You can try this yourself, even without a user account, because host key verification happens before user authentication: ssh -o 'VerifyHostKeyDNS ask' s0.barwen.ch

dnssec: automated re-signing of hawaga.org.uk

In a previous post on DNSSEC-signing my zone hawaga.org.uk, I mentioned that signatures will expire after 30 days, and so I (or rather one of my computers) will need to re-sign the zone at least every month.

Basically I need to run the dnssec-signzone command again, but there is some dancing around that needs to happen.

The most awkward was that I need to increment the zone serial number in the SOA record of my zone. Previously I've maintained this by hand, keeping it in format YYYYMMDDNN (year, month, day, sequence-number-on-that-day). That format is quite appealing because even if I forget what number I got up to, I can wait a day and know that I have a number in sequence.

dnssec-signzone offers a couple of options for doing things to serial numbers, but neither was what I wanted: one will increment the input SOA by one, but I want to maintain a pristine source zone file; another will set the SOA to the number of seconds since the unix epoch. This changes the format away from what I want.

So I wrote a quick utility, soatick, to generate zone serial numbers based on the current time and a state file, so that each invocation will generate a new serial number matching the format that I want:

$ ./soatick
2011010901
$ ./soatick
2011010902
$ ./soatick
2011010903

Now I'll use the m4 macro processor to put this in place before signing the zone:

export NEWSERIAL=$(/home/benc/src/soatick/soatick )
m4 -D___SERIAL___=$NEWSERIAL < db.hawaga.template > db.hawaga.generated
/usr/sbin/dnssec-signzone -S -t -a -l dlv.isc.org -f db.hawaga.signed -o hawaga.org.uk db.hawaga.generated

I put the above in a script called from cron, and set it to run every week.

Now a weakness here is that I have to keep my signing key unpassworded and on a system connected to the internet. The zone-signing and key-signing key separation should help here, by allowing me to keep a more important key offline and a less important key online, but I haven't investigated it in any greater depth - perhaps I should...

dnssec: signing hawaga.org.uk

In this post I will sign my main DNS zone hawaga.org.uk using DNSSEC so that other resolvers can verify the contents when they make a DNS lookup.

The parent domain org.uk does not do DNSSEC at the moment so I am going to use DLV (DNSSEC Lookaside Validation). DLV will let me publish my zone public keys in DLVs registry instead of in the parent org.uk zone, and my zones will be validated by anyone who has configured DLV in their resolver, rather than by all DNSSEC users (pretty much the first is a strict subset of the second). This is something of a hack - hopefully in a few months org.uk will support DNSSEC and I'll be able to directly publish my keys there instead.

Most of what follows works for both normal DNSSEC and DNSSEC-with-DLV.

I need to generate two public/private keypairs: a zone signing key, and a key signing key. The key signing key will be used to sign the zone signing key, and the zone signing key will be used to sign the records inside the zone. Its not entirely clear to me which private keys need to be online at which step in the process, but for my immediate purposes I don't mind too much - I'm keeping everything online the whole time for now.

bind comes with a utility called dnssec-keygen to generate the keys, and it is used much like any other command-line key generator such as gpg oropenssl:

$ /usr/sbin/dnssec-keygen -a RSASHA1 -b 768 -n ZONE hawaga.org.uk
Generating key pair...................++++++++ ..++++++++
Khawaga.org.uk.+005+48075
$ /usr/sbin/dnssec-keygen  -f KSK -a RSASHA1 -b 2048 -n ZONE hawaga.org.uk
...
$ ls Khawaga*
Khawaga.org.uk.+005+48075.key      Khawaga.org.uk.+005+48196.key
Khawaga.org.uk.+005+48075.private  Khawaga.org.uk.+005+48196.private

Now, I'm going to sign the zone:

$ /usr/sbin/dnssec-signzone -S -l dlv.isc.org -o hawaga.org.uk db.hawaga 
Fetching KSK 48196/RSASHA1 from key repository.
Verifying the zone using the following algorithms: RSASHA1.
Zone signing complete:
Algorithm: RSASHA1: KSKs: 1 active, 0 stand-by, 0 revoked
                    ZSKs: 1 active, 0 stand-by, 0 revoked
db.hawaga.signed

and I get these output files:

-rw-r--r-- 1 benc benc 35505 2011-01-04 21:47 db.hawaga.signed
-rw-r--r-- 1 benc benc   195 2011-01-04 21:47 dlvset-hawaga.org.uk.
-rw-r--r-- 1 benc benc   171 2011-01-04 21:47 dsset-hawaga.org.uk.

The signed zone lives in db.hawaga.signed. It looks like my input zone but with some new records added. For every input records, there is a new RRSIG record; there are DNSKEY records which contain the public keys generated above; and there are NSEC records which are used to securely deny the existence of entries.

In my bind directory, I replace my original db.hawaga zone file with the new signed version and restart bind.

Now bind will serving the appropriate DNSSEC records. I can see them by querying using the +dnssec flag to dig:

$ dig +dnssec www.hawaga.org.uk
[...]
;; ANSWER SECTION:
www.hawaga.org.uk.      3600    IN      CNAME   paella.hawaga.org.uk.
www.hawaga.org.uk.      3600    IN      RRSIG   CNAME 5 4 3600 20110208095742 20110109095742 48075 hawaga.org.uk. lNdWf71In/J8F7evBxxi1yTw7Fx4WRkcHK9vOilEyLmDq31uwYlTtf+d bAZ4WHSeF9aHGRlvy1c6bAWEgWs64K8RNr/7pkZW+y0kmP8qL+Eu0nOH DvvQQ32eShnoEYGc
paella.hawaga.org.uk.   3600    IN      A       174.143.245.7
paella.hawaga.org.uk.   3600    IN      RRSIG   A 5 4 3600 20110208095742 20110109095742 48075 hawaga.org.uk. EeUBihcnpDJad9JJnwLcR0nm9ef1E58fxdwio/SK1iSorkFZLZjxvk7r GYkCR0aCwAw1F2/lJ3kb6pTk+10H00lyowQc8crtBPIDkwiqDE0pTtEE JPCm51XuAc3bz/lX
[...]

So now my zone is signed.

But! (and this is a big but) there is no trust path for any resolver to validate that - there is no reason for any resolver to believe that it really was *me* that generated the keys rather than some random attacker.

In X.509, certificate authorities tell you how to trust someone by signing certificates. In plain DNSSEC, the parent zone acts as a CA by providing a signed DS (delegation signer) record in addition to an NS record. In DLV, ISC's DLV service fills this role by providing the signed DS record.

I'm going to use ISC's DLV zone (which appears to be the only serious one). There's a web interface to sign up, at https://dlv.isc.org/. When you sign up there, the interface authenticates your ownership of the zone by giving you a cookie TXT record to put in your zone. Once that authentication has happened, you can upload a DNSKEY, DS or DLV record.

Then ISC will publish a DLV record set for you (presumably it constructs one for you if you give a DNSKEY or DS record?). For example, mine looks like this:

hawaga.org.uk.dlv.isc.org. 877  IN      DLV     48196 5 2 FD144A3B36AC7E939DD7A04D5C6F967A07D01FB0029CA8B440F6CC2D 6F2415B8
hawaga.org.uk.dlv.isc.org. 877  IN      DLV     48196 5 1 0DE399B36303676F1243A923B6FC3893AE248F90

It takes a few hours for that DLV record to appear, but when it does, that is enough for a DLV-enabled resolver to be able to authenticate.

DNS-OARC has some open DNSSEC-enabled resolvers that can be used for testing this. I can check my DNSSEC using one of them:

$ dig +dnssec @2001:4f8:3:2bc:1::64:20 -t a www.hawaga.org.uk
[...]
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 4, AUTHORITY: 3, ADDITIONAL: 5
[...]

and look out for the ad bit to appear in the flags field. It does, which means that resolver trusts my results.

OK. That's the setup pretty much done. Two last notes:

Firstly, when a bind resolver is authoritative for a zone, then it won't check signatures and give an ad flag on results for that zone. That left me puzzling for a while, wondering why it wasn't checking my results, and is why I have to use one of DNS-OARC's resolvers above.

Secondly, this zone is only signed for 30 days. After that period (specifically, 30 days after I ran the dnssec-signzone command), the signatures will become invalid. This is quite a change from my previous non-DNSSEC zone update habits - I changed my zone a few times per year and only when I needed a change, but now I'm going to need to update it at least every month, and inaction will cause it to break. In a subsequent post, I'm going to write about some of the automation I did to do this.

dnssec: Configuring my resolver

DNSSEC, a mechanism for securing DNS, has around for a long time but only in the last year or so has it seen serious deployment. The root zone was signed about 6 months ago, which provides a security root from which all other DNSSEC can flow. Last time I looked at DNSSEC that was far in the untimetabled future, so I didn't put much effort then to get it actually working.

In this post, I'm going to write about configuring bind to check DNSSEC when I make DNS queries. In later posts, I'll write about the other side of things: securing my own zones with DNS.

I mentioned the root zone being signed above. That's one way of checking DNSSEC signatures (and in the long term, the main way). Another way is DLV (DNSSEC Lookaside Validation) which acts as a certificate authority, providing a way for a DNS zone to be signed without having to have a path all the way from the root. A third way is by listing the keys for known domains, which allows everything under those domains to be validated without needing a signature from a higher level in DNS. (This third way is how IANA's Interim Trust Anchor Repository worked, before being superseded by the signing of the root zone).

I want to configure both validation from the root, and DLV.

I very roughly followed along with this page, though different versions of bind, and differences in what I want to do, lead to differences.

First I need the root key-signing (public) key. This is the single well-known value that must be securely obtained. So I use insecure DNS to obtain it:

$ dig . dnskey | grep "257 " > root.dnskey
$ cat root.dnskey
.                       172363  IN      DNSKEY  257 3 8
AwEAAagAIKlVZrpC6Ia7gEzahOR+9W29euxhJhVVLOyQbSEW0O8gcCjF
FVQUTf6v58fLjwBd0YI0EzrAcQqBGCzh/RStIoO8g0NfnfL2MTJRkxoX
bfDaUeVPQuYEhg37NZWAJQ9VnMVDxP/VHL496M/QZxkjf5/Efucp2gaD
X6RS6CXpoY68LsvPVjR0ZSwzz1apAzvN9dlzEheX7ICJBBtuA6G3LQpz
W5hOA2hzCTMjJPJ8LbqF6dsV6DoBQzgul0sGIcGOYl7OyQdXfZ57relS
Qageu+ipAdTTJ25AsRTAoub8ONGcLmqrAmRLKBP1dfwhYB4N7knNnulq
QxA+Uk1ihz0=

Now at this point I should carefully check that this is right against various out-of-band sources. I didn't bother.

I also want a trusted key for DLV. That is available here.

Next I need to tell bind about these keys. I'm going to use bind's managed-key mechanism for this.
I put the key material into a file called benc-managed-keys, like this:

managed-keys {
   dlv.isc.org. initial-key 257 3 5 "BEAAAAPHMu/5onzrEE7z1egmhg/WPO0+juoZrW3euWEn4MxDCE1+lLy2 brhQv5rN32RKtMzX6Mj70jdzeND4XknW58dnJNPCxn8+jAGl2FZLK8t+ 1uq4W+nnA3qO2+DL+k6BD4mewMLbIYFwe0PG73Te9fZ2kJb56dhgMde5 ymX4BI/oQ+cAK50/xvJv00Frf8kw6ucMTwFlgPe+jnGxPPEmHAte/URk Y62ZfkLoBAADLHQ9IrS2tryAe7mbBZVcOwIeU/Rw/mRx/vwwMCTgNboM QKtUdvNXDrYJDSHZws3xiRXF1Rf+al9UmZfSav/4NWLKjHzpT59k/VSt TDN0YUuWrBNh";
   . initial-key 257 3 8 "AwEAAagAIKlVZrpC6Ia7gEzahOR+9W29euxhJhVVLOyQbSEW0O8gcCjF FVQUTf6v58fLjwBd0YI0EzrAcQqBGCzh/RStIoO8g0NfnfL2MTJRkxoX bfDaUeVPQuYEhg37NZWAJQ9VnMVDxP/VHL496M/QZxkjf5/Efucp2gaD X6RS6CXpoY68LsvPVjR0ZSwzz1apAzvN9dlzEheX7ICJBBtuA6G3LQpz W5hOA2hzCTMjJPJ8LbqF6dsV6DoBQzgul0sGIcGOYl7OyQdXfZ57relS Qageu+ipAdTTJ25AsRTAoub8ONGcLmqrAmRLKBP1dfwhYB4N7knNnulq QxA+Uk1ihz0=";

};

Note that the format is different from the key files obtained - the IN DNSKEY keywords go away and initial-key goes in instead, and the key material is now quoted with " and with a ; at the end of the line.

Then I told bind to import this into its configuration:

$ grep benc-managed-keys *
named.conf:include "/etc/bind/benc-managed-keys";

I need to tell bind to start using dnssec by adding these to named.conf.options:

dnssec-enable yes;
dnssec-validation yes;
dnssec-lookaside auto;

Now restart bind and hope that it all works.

How can I test my setup? I can use dig with the +dnssec parameter. This adds a flag to the query saying that DNSSEC is desired. For example:

$ dig @192.168.1.254 +dnssec hawaga.org.uk

dig will give a flag line in its output, like this:

;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

If DNSSEC was able to check the results (which means that the result was signed, and that the resolver that you just queried was able to validate the results, then there will be another flag in there: ad. If not, then something didn't work correctly.

You can get more DNSSEC logging by adding this to named.conf:

logging {
           channel dnssec_log {             // a DNSSEC log channel
                     file "/var/log/bind/dnssec.log" size 200m;
                   print-time yes;        // timestamp the entries
                     print-category yes;    // add category name to entries
                   print-severity yes;    // add severity level to entries
                     severity debug 3;      // print debug message <= 3
   
             };
   
       category dnssec  { dnssec_log;  };
     };

So that's DNSSEC configured for resolving. Next, I want to sign my own zones so that others can verify them with DNSSEC on their own resolvers - I'll write about that in another post.