dnssec: signing hawaga.org.uk

In this post I will sign my main DNS zone hawaga.org.uk using DNSSEC so that other resolvers can verify the contents when they make a DNS lookup.

The parent domain org.uk does not do DNSSEC at the moment so I am going to use DLV (DNSSEC Lookaside Validation). DLV will let me publish my zone public keys in DLVs registry instead of in the parent org.uk zone, and my zones will be validated by anyone who has configured DLV in their resolver, rather than by all DNSSEC users (pretty much the first is a strict subset of the second). This is something of a hack - hopefully in a few months org.uk will support DNSSEC and I'll be able to directly publish my keys there instead.

Most of what follows works for both normal DNSSEC and DNSSEC-with-DLV.

I need to generate two public/private keypairs: a zone signing key, and a key signing key. The key signing key will be used to sign the zone signing key, and the zone signing key will be used to sign the records inside the zone. Its not entirely clear to me which private keys need to be online at which step in the process, but for my immediate purposes I don't mind too much - I'm keeping everything online the whole time for now.

bind comes with a utility called dnssec-keygen to generate the keys, and it is used much like any other command-line key generator such as gpg oropenssl:

$ /usr/sbin/dnssec-keygen -a RSASHA1 -b 768 -n ZONE hawaga.org.uk
Generating key pair...................++++++++ ..++++++++
Khawaga.org.uk.+005+48075
$ /usr/sbin/dnssec-keygen  -f KSK -a RSASHA1 -b 2048 -n ZONE hawaga.org.uk
...
$ ls Khawaga*
Khawaga.org.uk.+005+48075.key      Khawaga.org.uk.+005+48196.key
Khawaga.org.uk.+005+48075.private  Khawaga.org.uk.+005+48196.private

Now, I'm going to sign the zone:

$ /usr/sbin/dnssec-signzone -S -l dlv.isc.org -o hawaga.org.uk db.hawaga 
Fetching KSK 48196/RSASHA1 from key repository.
Verifying the zone using the following algorithms: RSASHA1.
Zone signing complete:
Algorithm: RSASHA1: KSKs: 1 active, 0 stand-by, 0 revoked
                    ZSKs: 1 active, 0 stand-by, 0 revoked
db.hawaga.signed

and I get these output files:

-rw-r--r-- 1 benc benc 35505 2011-01-04 21:47 db.hawaga.signed
-rw-r--r-- 1 benc benc   195 2011-01-04 21:47 dlvset-hawaga.org.uk.
-rw-r--r-- 1 benc benc   171 2011-01-04 21:47 dsset-hawaga.org.uk.

The signed zone lives in db.hawaga.signed. It looks like my input zone but with some new records added. For every input records, there is a new RRSIG record; there are DNSKEY records which contain the public keys generated above; and there are NSEC records which are used to securely deny the existence of entries.

In my bind directory, I replace my original db.hawaga zone file with the new signed version and restart bind.

Now bind will serving the appropriate DNSSEC records. I can see them by querying using the +dnssec flag to dig:

$ dig +dnssec www.hawaga.org.uk
[...]
;; ANSWER SECTION:
www.hawaga.org.uk.      3600    IN      CNAME   paella.hawaga.org.uk.
www.hawaga.org.uk.      3600    IN      RRSIG   CNAME 5 4 3600 20110208095742 20110109095742 48075 hawaga.org.uk. lNdWf71In/J8F7evBxxi1yTw7Fx4WRkcHK9vOilEyLmDq31uwYlTtf+d bAZ4WHSeF9aHGRlvy1c6bAWEgWs64K8RNr/7pkZW+y0kmP8qL+Eu0nOH DvvQQ32eShnoEYGc
paella.hawaga.org.uk.   3600    IN      A       174.143.245.7
paella.hawaga.org.uk.   3600    IN      RRSIG   A 5 4 3600 20110208095742 20110109095742 48075 hawaga.org.uk. EeUBihcnpDJad9JJnwLcR0nm9ef1E58fxdwio/SK1iSorkFZLZjxvk7r GYkCR0aCwAw1F2/lJ3kb6pTk+10H00lyowQc8crtBPIDkwiqDE0pTtEE JPCm51XuAc3bz/lX
[...]

So now my zone is signed.

But! (and this is a big but) there is no trust path for any resolver to validate that - there is no reason for any resolver to believe that it really was *me* that generated the keys rather than some random attacker.

In X.509, certificate authorities tell you how to trust someone by signing certificates. In plain DNSSEC, the parent zone acts as a CA by providing a signed DS (delegation signer) record in addition to an NS record. In DLV, ISC's DLV service fills this role by providing the signed DS record.

I'm going to use ISC's DLV zone (which appears to be the only serious one). There's a web interface to sign up, at https://dlv.isc.org/. When you sign up there, the interface authenticates your ownership of the zone by giving you a cookie TXT record to put in your zone. Once that authentication has happened, you can upload a DNSKEY, DS or DLV record.

Then ISC will publish a DLV record set for you (presumably it constructs one for you if you give a DNSKEY or DS record?). For example, mine looks like this:

hawaga.org.uk.dlv.isc.org. 877  IN      DLV     48196 5 2 FD144A3B36AC7E939DD7A04D5C6F967A07D01FB0029CA8B440F6CC2D 6F2415B8
hawaga.org.uk.dlv.isc.org. 877  IN      DLV     48196 5 1 0DE399B36303676F1243A923B6FC3893AE248F90

It takes a few hours for that DLV record to appear, but when it does, that is enough for a DLV-enabled resolver to be able to authenticate.

DNS-OARC has some open DNSSEC-enabled resolvers that can be used for testing this. I can check my DNSSEC using one of them:

$ dig +dnssec @2001:4f8:3:2bc:1::64:20 -t a www.hawaga.org.uk
[...]
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 4, AUTHORITY: 3, ADDITIONAL: 5
[...]

and look out for the ad bit to appear in the flags field. It does, which means that resolver trusts my results.

OK. That's the setup pretty much done. Two last notes:

Firstly, when a bind resolver is authoritative for a zone, then it won't check signatures and give an ad flag on results for that zone. That left me puzzling for a while, wondering why it wasn't checking my results, and is why I have to use one of DNS-OARC's resolvers above.

Secondly, this zone is only signed for 30 days. After that period (specifically, 30 days after I ran the dnssec-signzone command), the signatures will become invalid. This is quite a change from my previous non-DNSSEC zone update habits - I changed my zone a few times per year and only when I needed a change, but now I'm going to need to update it at least every month, and inaction will cause it to break. In a subsequent post, I'm going to write about some of the automation I did to do this.

dnssec: Configuring my resolver

DNSSEC, a mechanism for securing DNS, has around for a long time but only in the last year or so has it seen serious deployment. The root zone was signed about 6 months ago, which provides a security root from which all other DNSSEC can flow. Last time I looked at DNSSEC that was far in the untimetabled future, so I didn't put much effort then to get it actually working.

In this post, I'm going to write about configuring bind to check DNSSEC when I make DNS queries. In later posts, I'll write about the other side of things: securing my own zones with DNS.

I mentioned the root zone being signed above. That's one way of checking DNSSEC signatures (and in the long term, the main way). Another way is DLV (DNSSEC Lookaside Validation) which acts as a certificate authority, providing a way for a DNS zone to be signed without having to have a path all the way from the root. A third way is by listing the keys for known domains, which allows everything under those domains to be validated without needing a signature from a higher level in DNS. (This third way is how IANA's Interim Trust Anchor Repository worked, before being superseded by the signing of the root zone).

I want to configure both validation from the root, and DLV.

I very roughly followed along with this page, though different versions of bind, and differences in what I want to do, lead to differences.

First I need the root key-signing (public) key. This is the single well-known value that must be securely obtained. So I use insecure DNS to obtain it:

$ dig . dnskey | grep "257 " > root.dnskey
$ cat root.dnskey
.                       172363  IN      DNSKEY  257 3 8
AwEAAagAIKlVZrpC6Ia7gEzahOR+9W29euxhJhVVLOyQbSEW0O8gcCjF
FVQUTf6v58fLjwBd0YI0EzrAcQqBGCzh/RStIoO8g0NfnfL2MTJRkxoX
bfDaUeVPQuYEhg37NZWAJQ9VnMVDxP/VHL496M/QZxkjf5/Efucp2gaD
X6RS6CXpoY68LsvPVjR0ZSwzz1apAzvN9dlzEheX7ICJBBtuA6G3LQpz
W5hOA2hzCTMjJPJ8LbqF6dsV6DoBQzgul0sGIcGOYl7OyQdXfZ57relS
Qageu+ipAdTTJ25AsRTAoub8ONGcLmqrAmRLKBP1dfwhYB4N7knNnulq
QxA+Uk1ihz0=

Now at this point I should carefully check that this is right against various out-of-band sources. I didn't bother.

I also want a trusted key for DLV. That is available here.

Next I need to tell bind about these keys. I'm going to use bind's managed-key mechanism for this.
I put the key material into a file called benc-managed-keys, like this:

managed-keys {
   dlv.isc.org. initial-key 257 3 5 "BEAAAAPHMu/5onzrEE7z1egmhg/WPO0+juoZrW3euWEn4MxDCE1+lLy2 brhQv5rN32RKtMzX6Mj70jdzeND4XknW58dnJNPCxn8+jAGl2FZLK8t+ 1uq4W+nnA3qO2+DL+k6BD4mewMLbIYFwe0PG73Te9fZ2kJb56dhgMde5 ymX4BI/oQ+cAK50/xvJv00Frf8kw6ucMTwFlgPe+jnGxPPEmHAte/URk Y62ZfkLoBAADLHQ9IrS2tryAe7mbBZVcOwIeU/Rw/mRx/vwwMCTgNboM QKtUdvNXDrYJDSHZws3xiRXF1Rf+al9UmZfSav/4NWLKjHzpT59k/VSt TDN0YUuWrBNh";
   . initial-key 257 3 8 "AwEAAagAIKlVZrpC6Ia7gEzahOR+9W29euxhJhVVLOyQbSEW0O8gcCjF FVQUTf6v58fLjwBd0YI0EzrAcQqBGCzh/RStIoO8g0NfnfL2MTJRkxoX bfDaUeVPQuYEhg37NZWAJQ9VnMVDxP/VHL496M/QZxkjf5/Efucp2gaD X6RS6CXpoY68LsvPVjR0ZSwzz1apAzvN9dlzEheX7ICJBBtuA6G3LQpz W5hOA2hzCTMjJPJ8LbqF6dsV6DoBQzgul0sGIcGOYl7OyQdXfZ57relS Qageu+ipAdTTJ25AsRTAoub8ONGcLmqrAmRLKBP1dfwhYB4N7knNnulq QxA+Uk1ihz0=";

};

Note that the format is different from the key files obtained - the IN DNSKEY keywords go away and initial-key goes in instead, and the key material is now quoted with " and with a ; at the end of the line.

Then I told bind to import this into its configuration:

$ grep benc-managed-keys *
named.conf:include "/etc/bind/benc-managed-keys";

I need to tell bind to start using dnssec by adding these to named.conf.options:

dnssec-enable yes;
dnssec-validation yes;
dnssec-lookaside auto;

Now restart bind and hope that it all works.

How can I test my setup? I can use dig with the +dnssec parameter. This adds a flag to the query saying that DNSSEC is desired. For example:

$ dig @192.168.1.254 +dnssec hawaga.org.uk

dig will give a flag line in its output, like this:

;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

If DNSSEC was able to check the results (which means that the result was signed, and that the resolver that you just queried was able to validate the results, then there will be another flag in there: ad. If not, then something didn't work correctly.

You can get more DNSSEC logging by adding this to named.conf:

logging {
           channel dnssec_log {             // a DNSSEC log channel
                     file "/var/log/bind/dnssec.log" size 200m;
                   print-time yes;        // timestamp the entries
                     print-category yes;    // add category name to entries
                   print-severity yes;    // add severity level to entries
                     severity debug 3;      // print debug message <= 3
   
             };
   
       category dnssec  { dnssec_log;  };
     };

So that's DNSSEC configured for resolving. Next, I want to sign my own zones so that others can verify them with DNSSEC on their own resolvers - I'll write about that in another post.

IPv6 in Amazon EC2

Amazon declares that IPv6 is unsupported on EC2 (the Elastic Compute Cloud), but I wanted it anyway.

I tried two ways, one which worked well, and one which did not.
The first way I tried, which did not work well, was using 6to4; the way which did work was a tunnel from Hurricane Electric.

I did everything below on one of the free Linux micro-instances supplied by Amazon, with an elastic IP address attached so that I'd have a permanent address.

The first approach I tried was using 6to4. This is a protocol which automatically gives a large range of IPv6 addresses to anyone with a single static IPv4, through a decentralised network of protocol gateways.

In another blog post, I described how to get 6to4 running on Linux in 5 command lines. I ran those commands on my EC2 instance and end up with my own IPv6 address configured.

But having an IPv6 address configured is not the same as having an IPv6 address reachable from the internet. There were severe reachability problems. The problems boil down to two causes:

• The decentralised management of 6to4 leads to lots of gateways being broken in a way that makes them act of blackholes; and the protocol makes it hard to discover which those gateways are to fix them. This is a problem for any host, not just for EC2 instances.
• Even with the EC2 firewall turned off as much as possible (i.e. no firewall rules at all), EC2 doesn't cleanly deliver IP traffic to instances. For most traffic, this is not a problem; but it interacts with 6to4 in a terrible way:
  your instance can only receive traffic from any particular 6to4 gateway if it has recently sent at least one packet to that gateway; there is a global network of 6to4 gateways, any of which can send traffic to you; when you send 6to4 traffic, you send it only to your closest 6to4 gateway. As a result, basically no 6to4 gateways can ever send traffic to you.

That second bullet point was especially hard to debug before I figured out what the EC2 firewall was doing - because certain pings and probes to test reachability caused the firewall to open up, at which point IPv6 traffic would flow between some sets of hosts until a few minutes later when it would mysteriously stop working again.

6to4 is basically useless on ec2.

---

The second method I tried, with much greater success [to the extent that, a year later, I regard this as production quality] is a manually configured tunnel via Hurricane Electric. HE have been around a long time; have a good reputation; and I used them before years ago.

The configuration at their end is a set of fairly straightforward web forms. I was allocated a /64 prefix, but they have options for more.

That web form also gives example configuration instructions for a variety of platforms. The Linux-net-tools instructions are the ones I used. I pasted the 4 given commands literally into a root prompt on my EC2 machine, and those configured the interface correctly (at least until reboot).

I get similar "stateful firewall" behaviour from EC2 as I mentioned above, but the difference here is that that the connection is only between me and the HE tunnel endpoint, rather than to arbitrary 6to4 gateways around the network. As long as *anything* goes over the tunnel every few minutes, then connectivity with the entire IPv6 world stays up. Compared to 6to4, when that tunnel is up, the connectivity from machines D and P seem *much* better. I can ping both ways without any mysterious losses. I need a ping to the tunnel endpoint (or anywhere on the IPv6 internet, really) every minute or so. That's no big deal - I have MRTG set up to measure some ipv6 latencies anyway, and that is generating this traffic as a side-effect.

So HE is a little bit (a few web forms) more effort to set up. But the connectivity is much much better. I recommend HE over 6to4 for this.

Other links: aco wrote about getting IPv6 on EC2 using sixxs, and if you're interested in getting a shell account on this machine (barwen.ch) to try for yourself: www.barwen.ch
. I found this this online ping tool useful during testing.

Modified: 2011-04-19 Rephrasing a bit based on ongoing experience, and some more hyperlinks
Modified: 2012-05-05 Rephrasing some more.

6 to 4 in 5

I have native IPv6 on my LAN for one server. I have another server which lives on LAN with only native IPv4. Can I get IPv6 on that? Yes!

My present configuration uses 6to4. This is a very straightforward mechanism to configure on linux, without needing any setup outside of my own machine. On the downside, it gives very little control of routing which can lead to strange connectivity problems that are hard to debug.

I followed the
Linux IPv6 HOWTO howto pretty faithfully.

I needed these 5 steps:

$ ipv4="174.143.245.7"; printf "2002:%02x%02x:%02x%02x::1" `echo $ipv4 | tr "." " "`

This tells me that my prefix is: 2002:ae8f:f507::/48

I get a 16-bit (4 hex-digit) subnet number to pick my own networks (I'll pick 2 in this case, fairly arbitrarily), and then the remaining 64 bits go towards identifying the host on the network (which I'll pick 0123:4567:89ab:cdef. Another good choice would be ::1)

So I end up choosing the IP address: 2002:ae8f:f507:2:0123:4567:89ab:cdef

Next:

modprobe sit  (it was compiled but not loaded)
ifconfig sit0 add 2002:ae8f:f507:2:0123:4567:89ab:cdef/16 up
/sbin/route -A inet6 add 2000::/3 gw ::192.88.99.1 dev sit0

and now I can ping my own machine:

benc@paella:~$ ping6 dildano.hawaga.org.uk
PING dildano.hawaga.org.uk(dildano.hawaga.org.uk) 56 data bytes
64 bytes from dildano.hawaga.org.uk: icmp_seq=1 ttl=62 time=136 ms
64 bytes from dildano.hawaga.org.uk: icmp_seq=2 ttl=62 time=132 ms
64 bytes from dildano.hawaga.org.uk: icmp_seq=3 ttl=62 time=132 ms

--- dildano.hawaga.org.uk ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 132.002/133.335/136.002/1.931 ms

easy peasy. shame it's quite unreliable.

building an analytic engine: plan 28

A while ago, I pledged some money to plan 28 through pledgebank.

In short:

Plan 28 is asking 10,000 people to pledge money towards building the Analytical Engine

It sounds like the project is going to go ahead even though they haven't reached the 10000 people they wanted. But maybe you want to make a pledge anyway?

3d cat

here is a picture of me with a stuffed cat at the oxford natural history museum. I happened to have two frames with a slight difference in position, so I combined them in an animated gif in a 3d display technique that I've seen before. It has the advantage that you don't need special viewing apparatus, but the disadvantage that you get seasick watching it.

pool.ntp.org

I've used pool.ntp.org to give me reasonable NTP servers before. On Christmas day I investigated how I could add my server. It turns out there is a self-service user interface to do so, with a scoring system that dynamically decides if your server is good enough to be published.

So I added by NTP server ntp.hawaga.org.uk (that also does a bunch of other stuff), waited a while for the score to rise, and then sat back and watched the port 123 packets flow.

There is lots of fun stuff about NTP breaking in ways which result in server floods - summarised in Wikipedia's NTP_server_misuse_and_abuse

One accusation that I've seen is that 50% of traffic is well behaved clients, and that the other 50% is a small number of misbehaving hosts which poll and poll and poll and poll. I've seen that behaviour in some of my tcpdump runs, though that is not too bad today:

Over a 15 minute period, I got 282 packets, less that one per second, from 53 different IP addresses, with a very unbalanced distribution: 24 hosts sending one packet, 50 sending less than one per minute (15 packets total), 3 sending more than one per minute, and one host sent 65 packets.

Update (Jan 6th 2011): You can watch the score history for my server in my pool.ntp.org profile, although at time of writing its kinda dull, being a flat top score for the past few days.

Update (Jan 7th 2011): I grabbed packets for about the last 24, and see 2736 distinct IP addresses, 32778 packets total (8 packets avg), noisiest host was 85.248.56.73 with 4628 packets. Anyway that's the most number of users I've ever provided service to, I think!

Update (Jan 17th 2011): I now have MRTG monitoring this server's estimated offset from correct time (similar but different to the profile graph above):