A string of DNS protocol bugs.

I went to turn on DNSSEC for cqx.ltd.uk today - the server that signed it broken right before my Christmas busy period so I disabled DNSSEC on that zone until I got round to fixing it.

I've encountered three different apparent protocol implementation bugs in the space of a few hours:

• Andrews and Arnold's web based control panel accepts DS records as generated by BIND's dnssec-keygen tool but then throws a complicated looking error when talking to Nominet, the UK domain registry, to put those records where they need to be. As far as I can tell, this is because the BIND output has whitespace in the middle of a hex string, something RFC 4034 s5.3 seems to think is acceptable. Why is installing crypto keys always so hard?
• For a while, Hetzner's recursive resolvers were unable to verify (and therefore refused to answer) results for my zone. I have a suspicion (but I don't have much to go on other than a hunch) that this was something to do with DS records and the actual zone having some kind of mismatch - although Google Public DNS at 8.8.8.8, and Verisign's DNSSEC checker both worked ok.
• I discovered an implementation quirk in the Haskell dns library, which I use inside a debugging tool I'm slowly building. This is to do with the mechanism which DNS uses to compress replies: where a domain name would be repeated in a response, it can be replaced by a pointer to another occurence of that name in the reply. It looks like in this case that the dns library will only accept those pointers if they point to regions of the reply that have specifically already been parsed by the domain name parsing code, rather than pointers to arbitrary bytes in the reply. This is frustratingly familiar to another bug I encountered (at Campus London) where their (not-so) transparent firewall was reordering DNS replies; giving a bug that only manifested when I was sitting in their cafe. (github issue #103)

/etc/hosts - the gift that keeps on giving

/etc/hosts

The gift that keeps on giving, when you're paid on an hourly rate to do tech support. It is like the regexps of DNS: "I know, I'll modify /etc/hosts" ... now you have two problems.

• You modify /etc/hosts because you want to override the global DNS view of name N.
• Things works!
• Because things work, you don't undo your change in /etc/hosts. After all, things work.
• A year passes.
• You forget that you made the change.
• Suddenly on your machine only, but no one else's, you can't access the website at N any more.
• Hours of debugging!

Why modify /etc/hosts? Testing a new version of a site under real name (only on your own machines though), or because DNS is broken and you don't know why, or because you didn't plan your DNS change so now the caches are not updating fast enough and you are impatient.

/etc/hosts doesn't have an "eventual consistency" mehanism in place - DNS caches eventually expire and at least give vaguely consistent behaviour across the whole internet over a long enough period of time. /etc/hosts will never converge.

ping reverse dns

Slightly unexpected hostname lookup on a CNAME.

maven.op is a CNAME to lulu; the reverse DNS points only to lulu.

Someone has already figured out the lulu hostname by the first line of output because it's shown there. But the ping lines for a few seconds show the name I gave and then turn to the "real" hostname (perhaps when a reverse DNS happens? rather than using the name that we looked up forward-wise to begin with?)

No big deal, but slightly unexpected.

benc@utsire:~$ ping maven.ops.xeus.co.uk
PING lulu.xeus.co.uk (46.4.100.47) 56(84) bytes of data.
64 bytes from maven.ops.xeus.co.uk (46.4.100.47): icmp_req=1 ttl=51 time=466 ms
64 bytes from maven.ops.xeus.co.uk (46.4.100.47): icmp_req=2 ttl=51 time=51.1 ms
64 bytes from maven.ops.xeus.co.uk (46.4.100.47): icmp_req=3 ttl=51 time=51.9 ms
64 bytes from lulu.xeus.co.uk (46.4.100.47): icmp_req=4 ttl=51 time=60.1 ms
64 bytes from lulu.xeus.co.uk (46.4.100.47): icmp_req=5 ttl=51 time=96.5 ms
64 bytes from lulu.xeus.co.uk (46.4.100.47): icmp_req=6 ttl=51 time=50.9 ms
64 bytes from lulu.xeus.co.uk (46.4.100.47): icmp_req=7 ttl=51 time=49.5 ms
64 bytes from lulu.xeus.co.uk (46.4.100.47): icmp_req=8 ttl=51 time=50.6 ms

benc@utsire:~$ ping -V
ping utility, iputils-sss20101006

Nominet .uk proposals again...

Nominet are proposing a second consulation on their wacky plan to open up second level domains in .uk - so I could have hawaga.org.uk or cqx.uk (assuming someone didn't get there first).

They talk now about owners of existing *.*.uk domains being given first dibs.

One of my concerns is that it fragments the UK namespace - there is already a "default zone" of co.uk and so that makes me wonder if (in the long term) co.uk should be abandoned entirely, and in the mean time, the co.uk and second level uk should be the same: one registration fee gives the same name registered in both zones. If you have the .co.uk, you get the .uk. If you don't have the .co.uk, you don't get the .uk.

That at least gives a bit of an easier user explanation "a .uk is the same as .co.uk used to be"; though its still loads of technical work on the site side for people who want such domains.

And it still destroys the history of having decent information about what kind of entity you are interacting with.

direct.uk consultation notes

Nominet, the authority for the .uk top level domain has a consultation on opening up registration of second level domains, such as example.uk. (At present, people generally can register third level domains such as example.co.uk, example.org.uk, example.ac.uk or example.gov.uk)
Tomorrow I'm going to a round table session (will King Arthur be there?), and so in preparation I made some notes on my response to their consulation document.My notes are here.

autogenerating reverse DNS for ipv6

I was getting annoyed by manually configuring an IPv6 reverse domain.

For reverse DNS, you need to break the IP address up into pieces (bytes for IPv4, nibbles for IPv6), reverse them, put dots between pieces, to get a domain name. Then at that domain name, you put a reference to the hostname for that IP.

So an IP address like 2001:8b0:7c:1:216:76ff:fe16:755a turns into a domain name a.5.5.7.6.1.e.f.f.f.6.7.6.1.2.0.1.0.0.0.c.7.0.0.0.b.8.0.1.0.0.2.ip6.arpa., and there you can find a PTR record pointing to the hostname dildano.hawaga.org.uk

Forming those long domain names was/is quite awkward, and its a task well suited to automation. All of the hosts already have forward DNS entries, so there's not even much additional information needed to generate the reverse zone.

I wrote a tool (in an unholy alliance of Haskell and dig) which queries a bunch of forward zones and outputs the appropriate reverse DNS records ready for pasting into a zone file.

You specify zones (and appropriate servers) that will be asked for AAAA records; then all of the AAAA records which refer to IPv6 addresses on the specified network will be converted into PTR records and sent to stdout, ready to paste into a zone file.

$ dnsrz hawaga.org.uk@dildano.hawaga.org.uk clifford.ac@malander.clifford.ac charlottevrinten.org@dildano.hawaga.org.uk mrsclifford.eu@malander.clifford.ac --prefix=200108b0007c0001
 
3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 PTR clifford.ac.
3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 PTR malander.clifford.ac.
3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 PTR malander.mrsclifford.eu.
4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 PTR fecolith.clifford.ac.
4.1.2.0.0.f.e.f.f.f.3.9.d.0.2.0 PTR pomade.clifford.ac.
a.5.5.7.6.1.e.f.f.f.6.7.6.1.2.0 PTR dildano.hawaga.org.uk.
c.0.2.a.4.c.e.f.f.f.3.6.b.1.2.0 PTR newsnowdrop.mrsclifford.eu.
0.a.0.c.b.a.e.f.f.f.3.6.1.2.2.0 PTR tenesmus.clifford.ac.
7.2.f.0.1.9.e.f.f.f.b.4.5.2.2.0 PTR coprolith.clifford.ac.
c.2.5.d.b.f.e.f.f.f.b.e.7.2.a.b PTR pygar.hawaga.org.uk.
c.2.5.d.b.f.e.f.f.f.b.e.7.2.a.b PTR pygar-6.hawaga.org.uk.
b.6.5.8.2.f.e.f.f.f.8.c.c.b.a.c PTR laptop.hawaga.org.uk.

I wanted to use the Haskell dns package which I've used a bit before; but it didn't have enough features: no zone transfer capability, for a start... so I invoke dig and parse that out.

The commandline syntax is: <zonename>@<DNS server> where zonename is a forward zone, and the specified server will answer AXFRs for that zone. Thats quite icky but it gets around needing a full Haskell DNS implementation.

The code is on github under benclifford/dnsrz.

(later: as fits my tradition of writing a tool and then finding someone has done something similar first, bind comes with a tool arpaname which will convert an IP address into a reverse name, though it doesn't do all the other stuff above, but does work for ipv4 too: http://ftp.isc.org/isc/bind9/cur/9.9/doc/arm/man.arpaname.html

dnstracer

I started getting into writing a program for checking all paths of a DNS query. Then I discovered that there are two (1, 2) tools, both called dnstracer around for doing that... no new ideas under the sun. I guess I'll have to make mine cooler.

dmarc

A bunch of email providers announced DMARC which builds on top of SPF and DKIM to allow domains to specify more policy when SPF and/or DKIM fail.

I already have SPF and DKIM set up on my personal domain, hawaga.org.uk, which has been round for over a decade. I run mail servers for various other domains, but those are much younger and much less widely used.

Its been hard to quantify how much this has helped/not helped. I don't get complaints about spam originating from my address. I used to get a lost of postmaster backscatter but not any more - not sure why, though I can invent various possible reasons.

One of the interesting things with DMARC is that it claims to provide feedback about what filtering is happening, from receiving/filtering parties - that makes it especially interesting, I think.

So, given that I already have DKIM and SPF, what extra do I need to do to get something useful from DMARC?

I need to publish a policy in DNS, under my sending domain. (this is also how SPF and DKIM do things)

So I've put in this policy on the 4th of Feb:
_dmarc.hawaga.org.uk. 3583 IN TXT "v=DMARC1\;p=none\;rua=mailto:benc@hawaga.org.uk\;ruf=mailto:benc@hawaga.org.uk\;ri=3600"
That says to not enforce any policy, but to email benc@hawaga.org.uk with reports every 3600 seconds (= 1 hour).

I set this up at about 5pm on a Saturday and about 11am on Sunday morning my first report arrived, with a timestamp range of a day, which must extend back before I turned this on...

In there, three messages from my main outbound mail server, and no others.

Lets see what else I get...

A couple of weeks later...

I got daily reports most days from Google (I think maybe the day I didn't get a report was because I hadn't sent any mail into google all(?)).

A few days after the above I added in two other domains: my company domain with only occasionally sends mail, and my girlfriend's vanity domain. Neither of those have SPF or DKIM on them, even though they come from the same mail servers as hawaga.org.uk.

There was a noticeable lack of reports from anyone other than Google. I asked around (on Google+) to see if anyone had reports from elsewhere (eg AOL or Yahoo, because those were also listed) but no one said yes.

So what about the reports?

Well, there were surprisingly more mail servers than I expected: along with my own two outbound servers, there were about 10 other servers, being the outbound mail servers of a handful of research institutes that I work with. Those reports were tagged by google as being via a mailing list. Its not clear to me what defines a message as being via a mailing list, but I guess it would mean that they'll put less weight on my SPF records? It also highlights how a naive interaction between mailing lists and SPF can result in your message being treated as spam.

I also got some DKIM fails reported from my own legitimate mail server. The best I have been able to diagnose there is that I had sendmail set to deliver mail without a DKIM signature if the dkim milter timed out; but if that's going to contribute negatively to spam treatment, then I think a better configuration is to have the milter set to retry later, resulting in more delayed mail, but more DKIM-signed mail.

The extra domains I added had no DKIM on them, but those weren't treated as DKIM-fails. Instead they were reported as DKIM 'none'. I'm not sure what causes none rather than fail, but my guess is its something to do with the fact that hawaga.org.uk has DKIM records in its DNS, and thats being treated as an indication that there should be DKIM signatures on messages. I think that's extra meaning that I hadn't understood DKIM DNS records to mean.

I have a similar confusion with the interaction between SPF and DMARC: SPF has multiple output states, not just pass or fail, and its not clear to me how those are treated by DMARC.

Processingwise: the reports come as zipped XML documents. It was relatively straightforward to munge these like any other XML (though I made it harder for myself by learning a new Haskell XML library rather than using ones I already knew).

Its unclear to me how I know that a report really is from a particular sender, and what the threat model is for people injecting false DKIM reports - perhaps injecting them to suggest that people's use of DKIM and SPF is causing their mail to be dropped, and thus encouraging them to turn off SPF and DKIM?

So for now, I'll keep this switched on, in monitoring-only mode. I don't feel I understand it well enough to turn it on in enforcement mode (especially as I'm not the only user sending mail under hawaga.org.uk. I think its very interesting and probably useful to be able to specify policy this way; but the policy language at the moment feels either vaguely defined, or at the least not concisely described, in a way that makes me comfortable.

DKIM - domainkeys identified mail

Looks like I never wrote a blog posting on setting up DKIM. I just realised one of my servers wasn't set up after a re-install, so I'm having to remember how to do it again.

I'm using sendmail. (yes, shut up) and DKIM hooks in using its milter (mail filter) mechanism.

# apt-get install dkim-filter

Now wire it into sendmail.mc:
INPUT_MAIL_FILTER(`dkim', `S=/var/run/dkim-filter/dkim-filter.sock')

Now when mail comes in, you should see it gets headers like this added by your mail server (dildano.hawaga.org.uk in this case) when DKIM verification happens (eg in mail from gmail).:

Authentication-Results: dildano.hawaga.org.uk; dkim=pass (1024-bit key)
 header.i=@hawaga.org.uk; dkim-adsp=none

The other half of the equation is DKIM signing my outbound mail, so that other people who do checks like this can verify/not-verify my email.

DKIM needs a public/private keypaid

# dkim-genkey -b 1024 -d hawaga.org.uk -s hampshire

-s specifies a selector name. This is a fairly arbitrary identifier used to identify this keypair, because a domain can have multiple keypairs (for example, one per mail server). In the hawaga.org.uk domain, I seem to use names of English counties.

# ls
hampshire.private  hampshire.txt
# cat hampshire.txt
hampshire._domainkey IN TXT "v=DKIM1; g=*; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDUP+5f0nEWyYICxr8rLN8xannlteBg4WF2Fat/MS8CiAa1lE2wgvhKYJJD/ydJ//5B9fBZAwSXTAq2ZCQYIfRf985Yip0BK80ECTlOunaSnMY/4/RzmkXGpndJaHIFqmSWDhML1yBP6W6owJDXIPDCAbV80kd5Z5aAkv8518lk+wIDAQAB" ; ----- DKIM hampshire for hawaga.org.uk

That .txt file is a DNS record to install under hawaga.org.uk. When you've installed it, you can check with:
dig -t txt hampshire._domainkey.hawaga.org.uk @localhost

That's the public key installed. Now the private key.

In /etc/dkim-filter.conf:

Domain hawaga.org.uk
KeyFile /etc/mail/hampshire.private
Selector hampshire

# /etc/init.d/dkim-filter restart

Now send out some mail to some other account. It should have a DKIM signature header added like this:

DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=hawaga.org.uk;
 s=hampshire; t=1328349841;
 bh=hGo8Oadbgx3cVNwLr3hGDRfMX5LwWwXuz2PzqEowx0I=;
 h=Date:From:To:Subject:Message-ID:MIME-Version:Content-Type;
 b=oBeSDSzxz7/awSnxuos6jyJuBoYH2MbiB3HDpbZfLQnTTdEJdx2WD0ubSVAaKAJmV
  ma5xuSaNGeS7X3Xg49obL6nWA89tiOeVAq9FO+7NP+v2DmUPFxEYkLeQJUANYKzAw/
  r8ag9XnbRkxvY+J/rrmeaAjJdnfgUQlKSHlV5CWE=

... and if that other account happens to do DKIM verification, you should see its version of:

Authentication-Results: paella.hawaga.org.uk; dkim=pass (1024-bit key) header.i=@hawaga.org.uk

DHmPC

I have a server running inside EC2. It gets its network details using dhcp.

ubuntu@s0:~$ hostname
s0.barwen.ch
ubuntu@s0:~$ hostname --fqdn
hostname: Name or service not known

grr.

This happens pretty much every time the VM reboots. Its something to do with getting a new private IP address each time it reboots.

Although this manages to work:

hostname --all-fqdns
s0.barwen.ch polm.pl stacheldraht.it s0.barwen.ch

The autoconfigured resolv.conf looks like this:

nameserver 172.16.0.23
domain eu-west-1.compute.internal
search eu-west-1.compute.internal

and if I comment out or remove both the domain and search lines, then everything works...

Those lines are wrong anyway - this machine is in my barwen.ch domain. It just happens to be hosted in amazon's network...

batch checking DNS delegation

I'm working with someone who has ten or so domains. All the domains are registered in different places, and with slightly different DNS settings. As part of tidying that up to get everything consistent, I wrote the following bash+dig script to display the delegation of each zone from its parent.

That is importantly not the same as what the name servers for the zone return as NS records. The "authoritative" source of NS (nameserver) records for a zone is that zone itself. Using dig to query the NS records seems to be returning those, unsurprisingly.

However, in order to query a zone, there is a second place where name servers must be configured, separately from in the zone itself. That is in the parent zone. If those are wrong, then you can get awkward to diagnose problems: you can see from dig that the nameservers are right, yet lookups go to the wrong place.

Hence my script.

You can see even on my own domains there's a slight misconfiguration: barwen.ch claims to have s0.barwen.ch as a server, but the .ch registry isn't delegating to it. That won't cause bad DNS lookups but will cause s0.barwen.ch to not be used as a nameserver sometimes. Worse is when the delegation points to an old server that then returns a new servers DNS, giving the illusion that all is well, until you turn off the old server (which is the problem I have on other zones)

$ ./list-domains-NS.sh 
ZONE hawaga.org.uk
NAMESERVERS ACCORDING TO GOOGLE DNS
dildano.hawaga.org.uk.
paella.hawaga.org.uk.
NAMESERVERS ACCORDING TO org.uk
hawaga.org.uk.  172800 IN NS paella.hawaga.org.uk.
hawaga.org.uk.  172800 IN NS dildano.hawaga.org.uk.

ZONE barwen.ch
NAMESERVERS ACCORDING TO GOOGLE DNS
paella.hawaga.org.uk.
s0.barwen.ch.
dildano.hawaga.org.uk.
NAMESERVERS ACCORDING TO ch
barwen.ch.  3600 IN NS dildano.hawaga.org.uk.
barwen.ch.  3600 IN NS paella.hawaga.org.uk.

So here's the script:

#!/bin/bash
cat domainlist.txt | while read d ; do
  echo "ZONE $d"
  echo "NAMESERVERS ACCORDING TO GOOGLE DNS"
  dig @8.8.8.8 -t ns $d +short

  PARENT=$(echo $d | sed 's/^[^.]*\.//')
  echo "NAMESERVERS ACCORDING TO $PARENT"

  PARENTNS=$(dig +short -t NS ${PARENT}. | head -n 1)

  dig @$PARENTNS -t NS +noall +authority +norecurse $d

  echo
done

sshfp dns

I've set up DNSSEC, so I'm on the path to trusting DNS more. I can put SSH key fingerprints for my hosts into DNS, and SSH clients can check those.

Even with insecure DNS, this is probably better than what you do now, which is to just to choose 'yes' to the following prompt without actually checking: (seriously, do you ever bother?)

The authenticity of host 's0.barwen.ch (192.168.55.55)' can't be established.
RSA key fingerprint is 9e:81:ab:cb:2a:ad:26:2f:10:ed:dd:5c:55:dd:ea:58.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? 

SSH can check s0's DNS record to see if a fingerprint is stored there, and tell you if it matches. So lets set that up.

I need to add an SSHFP record to the DNS for s0.barwen.ch.

On host fubar (without needing to be root):

$ ssh-keygen -r s0.barwen.ch
s0.barwen.ch IN SSHFP 1 1 560f08c1687a60e62a65ef427e63698ae1797d6f
s0.barwen.ch IN SSHFP 2 1 4ef38fd457d0afec50ca21eacb771f724e6d7236

So those are the records to add to barwen.ch's DNS.
(btw, vim on my machine doesn't like SSHFP records and highlights everythign red - eww)

Now wait for DNS to settle, and when I connect for the first time, I get a different message (my emphasis).

The authenticity of host 's0.barwen.ch (192.168.55.55)' can't be established.
RSA key fingerprint is 9e:81:ab:cb:2a:ad:26:2f:10:ed:dd:5c:55:dd:ea:58.
Matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? 

Cool.

You might need to set the client option VerifyHostKeyDNS ask in your ~/.ssh/config - if you really trust DNS, you can set it to yes instead, and it won't even ask you when there's an SSHFP record present.

You can try this yourself, even without a user account, because host key verification happens before user authentication: ssh -o 'VerifyHostKeyDNS ask' s0.barwen.ch

dnssec: automated re-signing of hawaga.org.uk

In a previous post on DNSSEC-signing my zone hawaga.org.uk, I mentioned that signatures will expire after 30 days, and so I (or rather one of my computers) will need to re-sign the zone at least every month.

Basically I need to run the dnssec-signzone command again, but there is some dancing around that needs to happen.

The most awkward was that I need to increment the zone serial number in the SOA record of my zone. Previously I've maintained this by hand, keeping it in format YYYYMMDDNN (year, month, day, sequence-number-on-that-day). That format is quite appealing because even if I forget what number I got up to, I can wait a day and know that I have a number in sequence.

dnssec-signzone offers a couple of options for doing things to serial numbers, but neither was what I wanted: one will increment the input SOA by one, but I want to maintain a pristine source zone file; another will set the SOA to the number of seconds since the unix epoch. This changes the format away from what I want.

So I wrote a quick utility, soatick, to generate zone serial numbers based on the current time and a state file, so that each invocation will generate a new serial number matching the format that I want:

$ ./soatick
2011010901
$ ./soatick
2011010902
$ ./soatick
2011010903

Now I'll use the m4 macro processor to put this in place before signing the zone:

export NEWSERIAL=$(/home/benc/src/soatick/soatick )
m4 -D___SERIAL___=$NEWSERIAL < db.hawaga.template > db.hawaga.generated
/usr/sbin/dnssec-signzone -S -t -a -l dlv.isc.org -f db.hawaga.signed -o hawaga.org.uk db.hawaga.generated

I put the above in a script called from cron, and set it to run every week.

Now a weakness here is that I have to keep my signing key unpassworded and on a system connected to the internet. The zone-signing and key-signing key separation should help here, by allowing me to keep a more important key offline and a less important key online, but I haven't investigated it in any greater depth - perhaps I should...

dnssec: signing hawaga.org.uk

In this post I will sign my main DNS zone hawaga.org.uk using DNSSEC so that other resolvers can verify the contents when they make a DNS lookup.

The parent domain org.uk does not do DNSSEC at the moment so I am going to use DLV (DNSSEC Lookaside Validation). DLV will let me publish my zone public keys in DLVs registry instead of in the parent org.uk zone, and my zones will be validated by anyone who has configured DLV in their resolver, rather than by all DNSSEC users (pretty much the first is a strict subset of the second). This is something of a hack - hopefully in a few months org.uk will support DNSSEC and I'll be able to directly publish my keys there instead.

Most of what follows works for both normal DNSSEC and DNSSEC-with-DLV.

I need to generate two public/private keypairs: a zone signing key, and a key signing key. The key signing key will be used to sign the zone signing key, and the zone signing key will be used to sign the records inside the zone. Its not entirely clear to me which private keys need to be online at which step in the process, but for my immediate purposes I don't mind too much - I'm keeping everything online the whole time for now.

bind comes with a utility called dnssec-keygen to generate the keys, and it is used much like any other command-line key generator such as gpg oropenssl:

$ /usr/sbin/dnssec-keygen -a RSASHA1 -b 768 -n ZONE hawaga.org.uk
Generating key pair...................++++++++ ..++++++++
Khawaga.org.uk.+005+48075
$ /usr/sbin/dnssec-keygen  -f KSK -a RSASHA1 -b 2048 -n ZONE hawaga.org.uk
...
$ ls Khawaga*
Khawaga.org.uk.+005+48075.key      Khawaga.org.uk.+005+48196.key
Khawaga.org.uk.+005+48075.private  Khawaga.org.uk.+005+48196.private

Now, I'm going to sign the zone:

$ /usr/sbin/dnssec-signzone -S -l dlv.isc.org -o hawaga.org.uk db.hawaga 
Fetching KSK 48196/RSASHA1 from key repository.
Verifying the zone using the following algorithms: RSASHA1.
Zone signing complete:
Algorithm: RSASHA1: KSKs: 1 active, 0 stand-by, 0 revoked
                    ZSKs: 1 active, 0 stand-by, 0 revoked
db.hawaga.signed

and I get these output files:

-rw-r--r-- 1 benc benc 35505 2011-01-04 21:47 db.hawaga.signed
-rw-r--r-- 1 benc benc   195 2011-01-04 21:47 dlvset-hawaga.org.uk.
-rw-r--r-- 1 benc benc   171 2011-01-04 21:47 dsset-hawaga.org.uk.

The signed zone lives in db.hawaga.signed. It looks like my input zone but with some new records added. For every input records, there is a new RRSIG record; there are DNSKEY records which contain the public keys generated above; and there are NSEC records which are used to securely deny the existence of entries.

In my bind directory, I replace my original db.hawaga zone file with the new signed version and restart bind.

Now bind will serving the appropriate DNSSEC records. I can see them by querying using the +dnssec flag to dig:

$ dig +dnssec www.hawaga.org.uk
[...]
;; ANSWER SECTION:
www.hawaga.org.uk.      3600    IN      CNAME   paella.hawaga.org.uk.
www.hawaga.org.uk.      3600    IN      RRSIG   CNAME 5 4 3600 20110208095742 20110109095742 48075 hawaga.org.uk. lNdWf71In/J8F7evBxxi1yTw7Fx4WRkcHK9vOilEyLmDq31uwYlTtf+d bAZ4WHSeF9aHGRlvy1c6bAWEgWs64K8RNr/7pkZW+y0kmP8qL+Eu0nOH DvvQQ32eShnoEYGc
paella.hawaga.org.uk.   3600    IN      A       174.143.245.7
paella.hawaga.org.uk.   3600    IN      RRSIG   A 5 4 3600 20110208095742 20110109095742 48075 hawaga.org.uk. EeUBihcnpDJad9JJnwLcR0nm9ef1E58fxdwio/SK1iSorkFZLZjxvk7r GYkCR0aCwAw1F2/lJ3kb6pTk+10H00lyowQc8crtBPIDkwiqDE0pTtEE JPCm51XuAc3bz/lX
[...]

So now my zone is signed.

But! (and this is a big but) there is no trust path for any resolver to validate that - there is no reason for any resolver to believe that it really was *me* that generated the keys rather than some random attacker.

In X.509, certificate authorities tell you how to trust someone by signing certificates. In plain DNSSEC, the parent zone acts as a CA by providing a signed DS (delegation signer) record in addition to an NS record. In DLV, ISC's DLV service fills this role by providing the signed DS record.

I'm going to use ISC's DLV zone (which appears to be the only serious one). There's a web interface to sign up, at https://dlv.isc.org/. When you sign up there, the interface authenticates your ownership of the zone by giving you a cookie TXT record to put in your zone. Once that authentication has happened, you can upload a DNSKEY, DS or DLV record.

Then ISC will publish a DLV record set for you (presumably it constructs one for you if you give a DNSKEY or DS record?). For example, mine looks like this:

hawaga.org.uk.dlv.isc.org. 877  IN      DLV     48196 5 2 FD144A3B36AC7E939DD7A04D5C6F967A07D01FB0029CA8B440F6CC2D 6F2415B8
hawaga.org.uk.dlv.isc.org. 877  IN      DLV     48196 5 1 0DE399B36303676F1243A923B6FC3893AE248F90

It takes a few hours for that DLV record to appear, but when it does, that is enough for a DLV-enabled resolver to be able to authenticate.

DNS-OARC has some open DNSSEC-enabled resolvers that can be used for testing this. I can check my DNSSEC using one of them:

$ dig +dnssec @2001:4f8:3:2bc:1::64:20 -t a www.hawaga.org.uk
[...]
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 4, AUTHORITY: 3, ADDITIONAL: 5
[...]

and look out for the ad bit to appear in the flags field. It does, which means that resolver trusts my results.

OK. That's the setup pretty much done. Two last notes:

Firstly, when a bind resolver is authoritative for a zone, then it won't check signatures and give an ad flag on results for that zone. That left me puzzling for a while, wondering why it wasn't checking my results, and is why I have to use one of DNS-OARC's resolvers above.

Secondly, this zone is only signed for 30 days. After that period (specifically, 30 days after I ran the dnssec-signzone command), the signatures will become invalid. This is quite a change from my previous non-DNSSEC zone update habits - I changed my zone a few times per year and only when I needed a change, but now I'm going to need to update it at least every month, and inaction will cause it to break. In a subsequent post, I'm going to write about some of the automation I did to do this.

dnssec: Configuring my resolver

DNSSEC, a mechanism for securing DNS, has around for a long time but only in the last year or so has it seen serious deployment. The root zone was signed about 6 months ago, which provides a security root from which all other DNSSEC can flow. Last time I looked at DNSSEC that was far in the untimetabled future, so I didn't put much effort then to get it actually working.

In this post, I'm going to write about configuring bind to check DNSSEC when I make DNS queries. In later posts, I'll write about the other side of things: securing my own zones with DNS.

I mentioned the root zone being signed above. That's one way of checking DNSSEC signatures (and in the long term, the main way). Another way is DLV (DNSSEC Lookaside Validation) which acts as a certificate authority, providing a way for a DNS zone to be signed without having to have a path all the way from the root. A third way is by listing the keys for known domains, which allows everything under those domains to be validated without needing a signature from a higher level in DNS. (This third way is how IANA's Interim Trust Anchor Repository worked, before being superseded by the signing of the root zone).

I want to configure both validation from the root, and DLV.

I very roughly followed along with this page, though different versions of bind, and differences in what I want to do, lead to differences.

First I need the root key-signing (public) key. This is the single well-known value that must be securely obtained. So I use insecure DNS to obtain it:

$ dig . dnskey | grep "257 " > root.dnskey
$ cat root.dnskey
.                       172363  IN      DNSKEY  257 3 8
AwEAAagAIKlVZrpC6Ia7gEzahOR+9W29euxhJhVVLOyQbSEW0O8gcCjF
FVQUTf6v58fLjwBd0YI0EzrAcQqBGCzh/RStIoO8g0NfnfL2MTJRkxoX
bfDaUeVPQuYEhg37NZWAJQ9VnMVDxP/VHL496M/QZxkjf5/Efucp2gaD
X6RS6CXpoY68LsvPVjR0ZSwzz1apAzvN9dlzEheX7ICJBBtuA6G3LQpz
W5hOA2hzCTMjJPJ8LbqF6dsV6DoBQzgul0sGIcGOYl7OyQdXfZ57relS
Qageu+ipAdTTJ25AsRTAoub8ONGcLmqrAmRLKBP1dfwhYB4N7knNnulq
QxA+Uk1ihz0=

Now at this point I should carefully check that this is right against various out-of-band sources. I didn't bother.

I also want a trusted key for DLV. That is available here.

Next I need to tell bind about these keys. I'm going to use bind's managed-key mechanism for this.
I put the key material into a file called benc-managed-keys, like this:

managed-keys {
   dlv.isc.org. initial-key 257 3 5 "BEAAAAPHMu/5onzrEE7z1egmhg/WPO0+juoZrW3euWEn4MxDCE1+lLy2 brhQv5rN32RKtMzX6Mj70jdzeND4XknW58dnJNPCxn8+jAGl2FZLK8t+ 1uq4W+nnA3qO2+DL+k6BD4mewMLbIYFwe0PG73Te9fZ2kJb56dhgMde5 ymX4BI/oQ+cAK50/xvJv00Frf8kw6ucMTwFlgPe+jnGxPPEmHAte/URk Y62ZfkLoBAADLHQ9IrS2tryAe7mbBZVcOwIeU/Rw/mRx/vwwMCTgNboM QKtUdvNXDrYJDSHZws3xiRXF1Rf+al9UmZfSav/4NWLKjHzpT59k/VSt TDN0YUuWrBNh";
   . initial-key 257 3 8 "AwEAAagAIKlVZrpC6Ia7gEzahOR+9W29euxhJhVVLOyQbSEW0O8gcCjF FVQUTf6v58fLjwBd0YI0EzrAcQqBGCzh/RStIoO8g0NfnfL2MTJRkxoX bfDaUeVPQuYEhg37NZWAJQ9VnMVDxP/VHL496M/QZxkjf5/Efucp2gaD X6RS6CXpoY68LsvPVjR0ZSwzz1apAzvN9dlzEheX7ICJBBtuA6G3LQpz W5hOA2hzCTMjJPJ8LbqF6dsV6DoBQzgul0sGIcGOYl7OyQdXfZ57relS Qageu+ipAdTTJ25AsRTAoub8ONGcLmqrAmRLKBP1dfwhYB4N7knNnulq QxA+Uk1ihz0=";

};

Note that the format is different from the key files obtained - the IN DNSKEY keywords go away and initial-key goes in instead, and the key material is now quoted with " and with a ; at the end of the line.

Then I told bind to import this into its configuration:

$ grep benc-managed-keys *
named.conf:include "/etc/bind/benc-managed-keys";

I need to tell bind to start using dnssec by adding these to named.conf.options:

dnssec-enable yes;
dnssec-validation yes;
dnssec-lookaside auto;

Now restart bind and hope that it all works.

How can I test my setup? I can use dig with the +dnssec parameter. This adds a flag to the query saying that DNSSEC is desired. For example:

$ dig @192.168.1.254 +dnssec hawaga.org.uk

dig will give a flag line in its output, like this:

;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

If DNSSEC was able to check the results (which means that the result was signed, and that the resolver that you just queried was able to validate the results, then there will be another flag in there: ad. If not, then something didn't work correctly.

You can get more DNSSEC logging by adding this to named.conf:

logging {
           channel dnssec_log {             // a DNSSEC log channel
                     file "/var/log/bind/dnssec.log" size 200m;
                   print-time yes;        // timestamp the entries
                     print-category yes;    // add category name to entries
                   print-severity yes;    // add severity level to entries
                     severity debug 3;      // print debug message <= 3
   
             };
   
       category dnssec  { dnssec_log;  };
     };

So that's DNSSEC configured for resolving. Next, I want to sign my own zones so that others can verify them with DNSSEC on their own resolvers - I'll write about that in another post.