14 January, 2011

IPv6 in Amazon EC2

Amazon declares that IPv6 is unsupported on EC2 (the Elastic Compute Cloud), but I wanted it anyway.

I tried two ways, one which worked well, and one which did not.
The first way I tried, which did not work well, was using 6to4; the way which did work was a tunnel from Hurricane Electric.

I did everything below on one of the free Linux micro-instances supplied by Amazon, with an elastic IP address attached so that I'd have a permanent address.

The first approach I tried was using 6to4. This is a protocol which automatically gives a large range of IPv6 addresses to anyone with a single static IPv4, through a decentralised network of protocol gateways.

In another blog post, I described how to get 6to4 running on Linux in 5 command lines. I ran those commands on my EC2 instance and end up with my own IPv6 address configured.

But having an IPv6 address configured is not the same as having an IPv6 address reachable from the internet. There were severe reachability problems. The problems boil down to two causes:

  • The decentralised management of 6to4 leads to lots of gateways being broken in a way that makes them act of blackholes; and the protocol makes it hard to discover which those gateways are to fix them. This is a problem for any host, not just for EC2 instances.
  • Even with the EC2 firewall turned off as much as possible (i.e. no firewall rules at all), EC2 doesn't cleanly deliver IP traffic to instances. For most traffic, this is not a problem; but it interacts with 6to4 in a terrible way:
    your instance can only receive traffic from any particular 6to4 gateway if it has recently sent at least one packet to that gateway; there is a global network of 6to4 gateways, any of which can send traffic to you; when you send 6to4 traffic, you send it only to your closest 6to4 gateway. As a result, basically no 6to4 gateways can ever send traffic to you.

That second bullet point was especially hard to debug before I figured out what the EC2 firewall was doing - because certain pings and probes to test reachability caused the firewall to open up, at which point IPv6 traffic would flow between some sets of hosts until a few minutes later when it would mysteriously stop working again.

6to4 is basically useless on ec2.


The second method I tried, with much greater success [to the extent that, a year later, I regard this as production quality] is a manually configured tunnel via Hurricane Electric. HE have been around a long time; have a good reputation; and I used them before years ago.

The configuration at their end is a set of fairly straightforward web forms. I was allocated a /64 prefix, but they have options for more.

That web form also gives example configuration instructions for a variety of platforms. The Linux-net-tools instructions are the ones I used. I pasted the 4 given commands literally into a root prompt on my EC2 machine, and those configured the interface correctly (at least until reboot).


I get similar "stateful firewall" behaviour from EC2 as I mentioned above, but the difference here is that that the connection is only between me and the HE tunnel endpoint, rather than to arbitrary 6to4 gateways around the network. As long as *anything* goes over the tunnel every few minutes, then connectivity with the entire IPv6 world stays up. Compared to 6to4, when that tunnel is up, the connectivity from machines D and P seem *much* better. I can ping both ways without any mysterious losses. I need a ping to the tunnel endpoint (or anywhere on the IPv6 internet, really) every minute or so. That's no big deal - I have MRTG set up to measure some ipv6 latencies anyway, and that is generating this traffic as a side-effect.

So HE is a little bit (a few web forms) more effort to set up. But the connectivity is much much better. I recommend HE over 6to4 for this.

Other links: aco wrote about getting IPv6 on EC2 using sixxs, and if you're interested in getting a shell account on this machine (barwen.ch) to try for yourself: www.barwen.ch
. I found this this online ping tool useful during testing.


Modified: 2011-04-19 Rephrasing a bit based on ongoing experience, and some more hyperlinks
Modified: 2012-05-05 Rephrasing some more.


Flattr this

8 comments:

  1. Another option is Miredo which uses a single UDP port. Even without security group, client will puncture a hole on NAT by itself. It seems work for me.

    I guess the reason why 6to4 sometime doesn't work is probably due to different return address from multiple address host which may use a native 6to4 return path.

    ReplyDelete
  2. Miredo should work fine for outbound/client-like access (although I haven't tried it in EC2, I use it elsewhere, and it sounds like you have). But it gives you a different ipv6 address for every miredo initialisation (because the UDP port used for the connection is encoded into the generated ipv6 address). (unless maybe the miredo client has an option to force the same port to be used every time, but I didn't see one).
    That frequently changing IP address makes not as suitable for a server environment.

    I agree with your diagnosis of 6to4.

    ReplyDelete
  3. oh I see miredo has a 'BindPort' config option which might allow the UDP port to be fixed (the OS X repackaging of Miredo that I use doesn't have that option exposed in its config GUI.

    ReplyDelete
  4. Also, you get no reverse DNS with miredo. I really should get round to finishing off my draft posting comparing different tunnelling methods.

    ReplyDelete
  5. Teredo address always has 12 random bits even when UDP is fixed.

    ReplyDelete
  6. Hi thanks for linking to my article, I prefer SiXS :) only because I used them for a long time on a number of hosts, and have a POP realy nearby (3 msec RTT). Recently I've switched to HE for my home setup (i have an Airport Extreme Base Station which works with HE tunnels), so'l I'll check out how well they do.

    ReplyDelete
  7. Anonymous10/4/13 21:32

    Hi, thanks Ben for this really informative article, its a deep analysis and experience report not found somewhere else!!!

    Are the drawbacks of 6to4 for EC2 still valid, even after 2 years?
    I consider using 6to4 for a bunch of embedded-controllers, because there is no relation to any tunnel broker and it is much easier to install, it looks like it is part of the standard.

    Thanks
    Achim

    ReplyDelete
  8. Achim, yes I believe those problems still happen (but I have not tested recently). In general I would recommend against using 6to4 anywhere, really. It is easy to configure on each host, but then often doesn't work very well in practice and the problems are extremely hard to track down, compared to eg using a tunnel broker.

    ReplyDelete