bugs that never go away

I had a problem trying to get mrtg to read ssCpuRawUser.0 and related snmp variables. it was giving some error, even though it can read various other snmp variables fine. i googled. I got a few pages of people going all the way back to 2001 with the same error. no solutions. there's even a bug where the mrtg author WORKSFORMEs it. fuck this. I think there's an XKCD about this. So, I can read the variable with the net-snmp commandline tool and feed it in that way. yay for being able to write my own plugins.

#!/bin/bash
# $1 = name1
# $2 = name
# $2 = host
snmpget -m + -c public -v 1 $3 $1 | sed 's/^.*Counter32: //'
snmpget -m + -c public -v 1 $3 $2 | sed 's/^.*Counter32: //'

dmarc

A bunch of email providers announced DMARC which builds on top of SPF and DKIM to allow domains to specify more policy when SPF and/or DKIM fail.

I already have SPF and DKIM set up on my personal domain, hawaga.org.uk, which has been round for over a decade. I run mail servers for various other domains, but those are much younger and much less widely used.

Its been hard to quantify how much this has helped/not helped. I don't get complaints about spam originating from my address. I used to get a lost of postmaster backscatter but not any more - not sure why, though I can invent various possible reasons.

One of the interesting things with DMARC is that it claims to provide feedback about what filtering is happening, from receiving/filtering parties - that makes it especially interesting, I think.

So, given that I already have DKIM and SPF, what extra do I need to do to get something useful from DMARC?

I need to publish a policy in DNS, under my sending domain. (this is also how SPF and DKIM do things)

So I've put in this policy on the 4th of Feb:
_dmarc.hawaga.org.uk. 3583 IN TXT "v=DMARC1\;p=none\;rua=mailto:benc@hawaga.org.uk\;ruf=mailto:benc@hawaga.org.uk\;ri=3600"
That says to not enforce any policy, but to email benc@hawaga.org.uk with reports every 3600 seconds (= 1 hour).

I set this up at about 5pm on a Saturday and about 11am on Sunday morning my first report arrived, with a timestamp range of a day, which must extend back before I turned this on...

In there, three messages from my main outbound mail server, and no others.

Lets see what else I get...

A couple of weeks later...

I got daily reports most days from Google (I think maybe the day I didn't get a report was because I hadn't sent any mail into google all(?)).

A few days after the above I added in two other domains: my company domain with only occasionally sends mail, and my girlfriend's vanity domain. Neither of those have SPF or DKIM on them, even though they come from the same mail servers as hawaga.org.uk.

There was a noticeable lack of reports from anyone other than Google. I asked around (on Google+) to see if anyone had reports from elsewhere (eg AOL or Yahoo, because those were also listed) but no one said yes.

So what about the reports?

Well, there were surprisingly more mail servers than I expected: along with my own two outbound servers, there were about 10 other servers, being the outbound mail servers of a handful of research institutes that I work with. Those reports were tagged by google as being via a mailing list. Its not clear to me what defines a message as being via a mailing list, but I guess it would mean that they'll put less weight on my SPF records? It also highlights how a naive interaction between mailing lists and SPF can result in your message being treated as spam.

I also got some DKIM fails reported from my own legitimate mail server. The best I have been able to diagnose there is that I had sendmail set to deliver mail without a DKIM signature if the dkim milter timed out; but if that's going to contribute negatively to spam treatment, then I think a better configuration is to have the milter set to retry later, resulting in more delayed mail, but more DKIM-signed mail.

The extra domains I added had no DKIM on them, but those weren't treated as DKIM-fails. Instead they were reported as DKIM 'none'. I'm not sure what causes none rather than fail, but my guess is its something to do with the fact that hawaga.org.uk has DKIM records in its DNS, and thats being treated as an indication that there should be DKIM signatures on messages. I think that's extra meaning that I hadn't understood DKIM DNS records to mean.

I have a similar confusion with the interaction between SPF and DMARC: SPF has multiple output states, not just pass or fail, and its not clear to me how those are treated by DMARC.

Processingwise: the reports come as zipped XML documents. It was relatively straightforward to munge these like any other XML (though I made it harder for myself by learning a new Haskell XML library rather than using ones I already knew).

Its unclear to me how I know that a report really is from a particular sender, and what the threat model is for people injecting false DKIM reports - perhaps injecting them to suggest that people's use of DKIM and SPF is causing their mail to be dropped, and thus encouraging them to turn off SPF and DKIM?

So for now, I'll keep this switched on, in monitoring-only mode. I don't feel I understand it well enough to turn it on in enforcement mode (especially as I'm not the only user sending mail under hawaga.org.uk. I think its very interesting and probably useful to be able to specify policy this way; but the policy language at the moment feels either vaguely defined, or at the least not concisely described, in a way that makes me comfortable.

DKIM - domainkeys identified mail

Looks like I never wrote a blog posting on setting up DKIM. I just realised one of my servers wasn't set up after a re-install, so I'm having to remember how to do it again.

I'm using sendmail. (yes, shut up) and DKIM hooks in using its milter (mail filter) mechanism.

# apt-get install dkim-filter

Now wire it into sendmail.mc:
INPUT_MAIL_FILTER(`dkim', `S=/var/run/dkim-filter/dkim-filter.sock')

Now when mail comes in, you should see it gets headers like this added by your mail server (dildano.hawaga.org.uk in this case) when DKIM verification happens (eg in mail from gmail).:

Authentication-Results: dildano.hawaga.org.uk; dkim=pass (1024-bit key)
 header.i=@hawaga.org.uk; dkim-adsp=none

The other half of the equation is DKIM signing my outbound mail, so that other people who do checks like this can verify/not-verify my email.

DKIM needs a public/private keypaid

# dkim-genkey -b 1024 -d hawaga.org.uk -s hampshire

-s specifies a selector name. This is a fairly arbitrary identifier used to identify this keypair, because a domain can have multiple keypairs (for example, one per mail server). In the hawaga.org.uk domain, I seem to use names of English counties.

# ls
hampshire.private  hampshire.txt
# cat hampshire.txt
hampshire._domainkey IN TXT "v=DKIM1; g=*; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDUP+5f0nEWyYICxr8rLN8xannlteBg4WF2Fat/MS8CiAa1lE2wgvhKYJJD/ydJ//5B9fBZAwSXTAq2ZCQYIfRf985Yip0BK80ECTlOunaSnMY/4/RzmkXGpndJaHIFqmSWDhML1yBP6W6owJDXIPDCAbV80kd5Z5aAkv8518lk+wIDAQAB" ; ----- DKIM hampshire for hawaga.org.uk

That .txt file is a DNS record to install under hawaga.org.uk. When you've installed it, you can check with:
dig -t txt hampshire._domainkey.hawaga.org.uk @localhost

That's the public key installed. Now the private key.

In /etc/dkim-filter.conf:

Domain hawaga.org.uk
KeyFile /etc/mail/hampshire.private
Selector hampshire

# /etc/init.d/dkim-filter restart

Now send out some mail to some other account. It should have a DKIM signature header added like this:

DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=hawaga.org.uk;
 s=hampshire; t=1328349841;
 bh=hGo8Oadbgx3cVNwLr3hGDRfMX5LwWwXuz2PzqEowx0I=;
 h=Date:From:To:Subject:Message-ID:MIME-Version:Content-Type;
 b=oBeSDSzxz7/awSnxuos6jyJuBoYH2MbiB3HDpbZfLQnTTdEJdx2WD0ubSVAaKAJmV
  ma5xuSaNGeS7X3Xg49obL6nWA89tiOeVAq9FO+7NP+v2DmUPFxEYkLeQJUANYKzAw/
  r8ag9XnbRkxvY+J/rrmeaAjJdnfgUQlKSHlV5CWE=

... and if that other account happens to do DKIM verification, you should see its version of:

Authentication-Results: paella.hawaga.org.uk; dkim=pass (1024-bit key) header.i=@hawaga.org.uk

yubikey for encryption, not verification, passwords

I previously mentioned that my yubikey has a mode where it can enter a 64 character fixed string. I've been regarding that as useful in systems that are too closed to support HOTP. But I just realised that they also have a more "sensible" use on systems that due to more fundamental technical reasons cannot have a changing password - where that password is used to actually encrypt data, rather than being verified against an expected password - for example, GPG or encrypted home directories.

one line to make your site look nice on iPhone

Well, I came across a magic line of HTML for making a website look basically readable on an iPhone. Not magic in the sense that I don't understand what it does. But magic in the sense that its a single line that is the first big step to making a site look OK.

The line is (to go in your <head> section:
<meta name="viewport" content="width = device-width" />

What it does is make the iPhone web browser render the HTML at a sensible readable font size, with word wrapping at the end of the screen. (the default, otherwise, is to try to fit a regular screen worth of pixels across, then zoom out to make it fit on the small iPhone screen - that means the user has to zoom and pan to do anything).

Now my pages still look like crappy hand written HTML, but at least they're readable on an iPhone now.

I added this to the shellinabox installation I have on barwen.ch, and now its much prettier to use a browser-based shell on an iphone - you get a 30 character terminal thats at sensible font size, rather than a wide wide terminal at unreadable font size.

Two hardware OTP keys

I got a couple of OTP (one time password) keys to try out. These are hardware dongles that generate a unique code number every time you use them, which you then use in addition to a password when you log in to places (eg your server, some website).

The goal is to make things more secure by not having a password someone can steal.

The mechanics of this used to be hard to describe but enough people use online banking with security tokens now (at least in western europe) that the idea is pretty widely known already. (You can read about two factor authentication on wikipedia)

---

First, I tried a yubikey. Yubikey comes with a silly tagline, "the key to the cloud", but don't let that put you off. The yubikey plugs into a USB port on your computer and when you press its single button, it types in the next code in sequence as if it was a USB keyboard. I've tried this on a linux box and and an OS X box and it had no problem on either.

Pressing a button seems much less hassle than typing in a code off an LCD screen, but it does come with downsides: you need to have the device in a USB slot when you press the button. On a tower desktop, that's possibly down by the floor. Even worse, maybe you don't have a USB slot at all, in which case the device is useless.

yubikey has a number of different modes, and can store two configurations at once (yes, even though it has just one button).

There's a yubikey proprietary mode which generates a long key string which contains a bunch of stuff (for example, a device ID); an HOTP mode which generates a 6 or 8 character code (with programmable extra decoration); and a static mode which types in a preprogrammed fixed 64 character string. This is all configured by some software that you get for free off the yubikey website. I'm always a bit wary of vendor software to support hardware devices, because it often seems a much lower priority than the hardware device itself. But this worked well enough.

I only tried out the HOTP mode, because I wanted interoperability with other OTP implementations.

The two configurations are accessed by holding down the button either for less than 2s or for more than 2s. I've only used the first configuration, and I haven't had any trouble with accidentally falling through to the second one. But it sounds a bit cracked out and if I was giving it to the kind of user who would hold the button extra long "just to be sure," then maybe there would be trouble. I was hoping there would be an option to switch that second configuration off, but I didn't see one.

My only interaction with the supplier, yubico, was to order the key online, for $30+VAT. This arrived 24h after I order it, by regular mail (from the next village over!).

---

The second device I got was an OTP C200 token from Gooze. This has a more traditional user interface with a 6 digit LCD display and a button which turns on/off the display. The C200 is a TOTP token: HOTP, except the code changes every 30 seconds instead of when you press the button.

Gooze also makes a C100 which is regular HOTP. I haven't tried one of these, but the design of the C200 case makes me think that the button would get pressed a bunch randomly if you're carrying this with your keys in your pocket. With TOTP, that's not a problem - the code is not related to the button press. But I've encountered loss-of-sync troubles with other hardware tokens before due to this and I don't think the C100 solves that problem.

There is no configuration of the device itself needed - it only does TOTP, and the seed value is preloaded. You get sent that by Gooze. This was sent by GPG encrypted mail so I could cut and paste it into the configuration of my server easily. It means Gooze knows your secret key (although they claim to delete them after sending). I'm not too fussed by that because I'm not aiming for über-high security, but I'm sure some people will.

Worse though was that through some mess up in customer service, it took them over a week to get the codes to me after the devices arrived, and they were pretty silent during that week despite repeated enquiries. I think this is due to the company being pretty small. This is almost enough to make me not order from them again.

Because the C200 has a screen, you have to read the code and type it into your computer by hand. So some properties are inverted from the yubikey: it's a hassle to type in the code; but it doesn't matter if you have a USB port. Because of that, I think this is more appropriate than the yubikey for "I'm going on holiday but want to be able to access my email from public terminals" uses.

---

I wired both of these up to pam_oath to log into my linux servers; maybe I'll write about that side of things another time. Neither device has beaten the other in being my favourite - the yubikey has substantially higher geek value for plugging into a USB port and greater convenience in some circumstances, but the C200 feels more practical for other use cases such as connecting from unfamilar devices. I've only had a few days to form the above opinions and I expect I'll form more opinions over time.

server availability like uptime

I wondered if I could get a measure of server availability as a single number, automatically (for calculating things like how tragically few nines of uptime my own servers have)

So, I wrote a tool called long-uptime which you use like this:

The first time you run the code, initialise the counter. You can specify your estimate, or let it default to 0:

$ long-uptime --init

and then every minute in a cronjob run this:

$ long-uptime
0.8974271427587808

which means that the site has 89.7% uptime.

It computes an exponentially weighted average with a decay constant (which is a bit like a half life) of a month. This is how unix load averages (the last three values that come out of the uptime command) are calculated, though with much shorter decay constants of 1, 5, and 15 minutes.

When the machine is up (that is, you are running long-uptime in a cron job), then the average moves towards 1. When the machine is down (that is, you are not running long-uptime), then the average moves towards 0. Or rather, the first time you run long-uptime after a break, it realises you haven't run it during the downtime and recomputes the average as if it had been accumulating 0 scores.

Download the code:

$ wget http://www.hawaga.org.uk/tmp/long-uptime-0.1.tar.gz
$ tar xzvf long-uptime-0.1.tar.gz
$ cabal install
$ long-uptime --init

cuba

was in cuba for a month. internet is hard there (though possible) so no updates on this blog in that time. sorry.

https in cpanel

working with someone who has a cpanel server. they want https on it. cpanel doesn't do that by default. google doesn't reveal much in the way of tutorials for this, so here's a note for people to find.

1. generate a key pair and certificate using the Generate a SSL Certificate & Signing Request page. Copy the certificate onto your clipboard.
2. go to the Install a SSL Certificate and Setup the Domain page. Paste in the certificate. click fetch on the key text field and it should populate that field for you. Set the username to nobody so that all users can use this key pair.
3. When you save that page, apache will reload and you'll get https service on port 443, with a self-signed certificate (and so with consequent certificate mismatch error messages). But your existing domains won't work on that server - they'll go to the default cpanel parking page - cpanel only configures its virtual hosts on port 80... grr
4. So next I made an apache mod_rewrite rule in the VirtualHost directive for the port 443 virtual server. That causes all the internal sites appear on port 443.

       RewriteEngine on
       RewriteRule   ^(.+)          http://%{HTTP_HOST}$1 [P]
   

   That's an awkward hack to have to add to cpanel's generated config, but it seems to work (modulo invalid certificate warnings that all users ignore anyway)...

There's also a hole in the way that that rewrite rule is implemented: with a custom http client, you can probably make this server act as an arbitrary proxy for you, depending on mod_proxy configuration.

DHmPC

I have a server running inside EC2. It gets its network details using dhcp.

ubuntu@s0:~$ hostname
s0.barwen.ch
ubuntu@s0:~$ hostname --fqdn
hostname: Name or service not known

grr.

This happens pretty much every time the VM reboots. Its something to do with getting a new private IP address each time it reboots.

Although this manages to work:

hostname --all-fqdns
s0.barwen.ch polm.pl stacheldraht.it s0.barwen.ch

The autoconfigured resolv.conf looks like this:

nameserver 172.16.0.23
domain eu-west-1.compute.internal
search eu-west-1.compute.internal

and if I comment out or remove both the domain and search lines, then everything works...

Those lines are wrong anyway - this machine is in my barwen.ch domain. It just happens to be hosted in amazon's network...