ping reverse dns

Slightly unexpected hostname lookup on a CNAME.

maven.op is a CNAME to lulu; the reverse DNS points only to lulu.

Someone has already figured out the lulu hostname by the first line of output because it's shown there. But the ping lines for a few seconds show the name I gave and then turn to the "real" hostname (perhaps when a reverse DNS happens? rather than using the name that we looked up forward-wise to begin with?)

No big deal, but slightly unexpected.

benc@utsire:~$ ping maven.ops.xeus.co.uk
PING lulu.xeus.co.uk (46.4.100.47) 56(84) bytes of data.
64 bytes from maven.ops.xeus.co.uk (46.4.100.47): icmp_req=1 ttl=51 time=466 ms
64 bytes from maven.ops.xeus.co.uk (46.4.100.47): icmp_req=2 ttl=51 time=51.1 ms
64 bytes from maven.ops.xeus.co.uk (46.4.100.47): icmp_req=3 ttl=51 time=51.9 ms
64 bytes from lulu.xeus.co.uk (46.4.100.47): icmp_req=4 ttl=51 time=60.1 ms
64 bytes from lulu.xeus.co.uk (46.4.100.47): icmp_req=5 ttl=51 time=96.5 ms
64 bytes from lulu.xeus.co.uk (46.4.100.47): icmp_req=6 ttl=51 time=50.9 ms
64 bytes from lulu.xeus.co.uk (46.4.100.47): icmp_req=7 ttl=51 time=49.5 ms
64 bytes from lulu.xeus.co.uk (46.4.100.47): icmp_req=8 ttl=51 time=50.6 ms

benc@utsire:~$ ping -V
ping utility, iputils-sss20101006

containerisation of my own environment

I've encountered docker in a couple of work-related projects, and for a bit more experimentation I've begun containerising a chunk of my own infrastructure.

Previously I've had a few servers around which over the years have ended up with me being a bit scared to upgrade: too many dependencies for what should be separate services. Last time I rebuilt my main server, I looked at Xen but that machine was a little bit too out of date for doing decent virtualisation: having a bunch of different VMs was at the time the best way I could see for doing this.

I've ended up with an LDAP server for unix accounts, and a /home shared between all the containers that need home directory access (which is done with docker volume mounts at the moment, but there is scope for adding NFS onto that if/when I spread to more than one machine).

I've got separate containers for each of: inbound smtp, outbound smtp, an apache proxy that redirects to other containers based on URL, imap, webmail, ldap server, ssh server (so you are ssh-ing into a container, not the base OS).

The plan is that each of these is built and restarted automatically every week or so; along with a whole machine reboot at least once a month. I'm hoping that keeps stuff fairly up to date and helps me discover upgrade-related breakages around the time they happen rather than years later. It also forces me to pay attention to documenting how I set something up: all the stuff that is torn down and rebuilt each time needs to be documented properly in machine-readable form, so that the rebuild works. In that sense, it is a bit like automated testing of documentation.

I've also tried to set up things like port forwarding and http forwarding so that its not too reliant on using docker - so that I can spread onto other machines or use different virtualisation. That is, for example, how I intend to deal with upgrading the base OS in a few years time - by starting a new VM, moving the services across one by one, and then killing the old one.

balancing tests between 4 workers by external observation

We have a bunch of functional tests that are run by our buildbot every time someone makes a git push.

These tests were taking a long time. Obvious answer: run them in parallel.

This brought up some problems, the biggest of which was how to split the tests between the four test runners. Naively chopping the test list into four pieces turned out to be pretty imbalanced: one test runner was taking 4 minutes, another was taking 14 minutes.

Initially this sounded like something a task farm would be good for. But I didn't want to get into digging round in the groovy test running code, and making the test runners talk back to the buildbot to collect tasks.

So I took a less balancing approach with a simpler interface: A balance program picks some tests for each of the four test runners, runs the tests, takes the time of the whole run for each runner, and then iteratively updates its knowledge so that it will hopefully pick a more balanced distribution next time round.

I had a quick play with genetic algorithms but that didn't seem to be going anywhere. Next I implemented this model:

Assume there is a startup cost k, and for each test i, a time t_i that the test takes to run. These cannot directly be measured by the balance program.

Keep an estimate of k and t_i in a state file.

Make a distribution of tests over the four runners based on the estimates.

When each runner finishes, if it took longer than the estimate, nudge up the scores on k and the tests that where on that runner; similarly nudge down if the run time was less.

Run this lots of times.

After a while this rearranges the tests so that each runner takes about 10 minutes each (compared to the 4 .. 14 minutes with a naive distribution)

So we've saved a few minutes on the tests and are hopefully in a position where as we get more tests we can scale up the number of runners and still preserve reasonable balance.

This also copes with converging to a new balance when tests are added/removed; or when test time changes (either due to the test itself changing, or the behaviour being tested)

(The other problem was that it turns out that loads of our tests were secretly dependent on each other and failed when run in different order - this would occasionally cause problems in the naive distribution but was much more of a problem with the reordering that happens with this balancing approach)

Source code is https://github.com/benclifford/weightbal

fascist push master

I've been working with a customer to get some development infrastructure and procedures in place.

A problem we had was that, although we had got development branches, some developers were surpremely confident that their change definitely wouldn't break things, and so would regularly put some of their commits directly on the master branch; and then a few minutes later, the build robot would discover that things were broken on master. This was (is) bad because we were trying to keep master always ready to deploy; and trying to keep it so other developers could always base their new work against the latest master.

So we invented a new policy, and a tool to go with it, the fascist-push-master.

Basically, you can only put something on master if it has already passed the build robot's tests (for example, on a development branch).

This doesn't mean "your dev branch passes so now you're allowed to merge it to master" because often breakages happen because of the merge of "independent" work. Instead it means, that master can only be updated to a git commit that has actually passed the tests: you have to merge master into your dev branch, check it works ok there, and then that is what becomes the new master commit.

I wasn't so mean as to make the git repository reject unapproved commits - if someone is so disobedient as to ignore the policy, they can still make whatever commits they want to master. Rejection only happens in the client side tool. This was a deliberate decision.

The merge-to-master workflow previously looked like this:

git checkout mybranch
# do some work
git commit -a -m "my work"
git checkout master
git merge mybranch
git push origin master
# now the build robot tests master, with mybranch changes added

but now it looks like this:

git checkout mybranch
# do some work
git commit -a -m "my work"
git merge master
git push origin mybranch
# now the build robot tests mybranch with master merged in
fascist-push-master
# which will either push this successfully tested
# mybranch to master or fail

The output looks like this:

$ fascist-push-master
You claim ba5810edfc67be5118be6c02ab3ffbe215bbe898 on branch mybranch is tested and ready for master push
Checking that story with build robot.
Umm. Liar. Go merge master to your branch, get jenkins to test it, and come back when it works.

INFORMATION ABOUT YOUR LIES: BRANCHID=mybranch
INFORMATION ABOUT YOUR LIES: COMMITID=ba5810edfc67be5118be6c02ab3ffbe215bbe898

The script is pretty short, and pasted below:

#!/bin/bash

export BRANCHID=$(git rev-parse --abbrev-ref HEAD)
export COMMITID=$(git rev-parse HEAD)

echo You claim $COMMITID on branch $BRANCHID is tested and ready for master push

echo Checking that story with winnie.
ssh buildrobot.example.com grep $COMMITID /var/lib/jenkins/ok-commits
RES=$?

if [ "$RES" == "0" ]; then
  echo OK, you appear to be telling the truth.
  (git fetch && git checkout master && git merge --ff-only origin/master && git merge --ff-only $COMMITID && git push origin master && git checkout $BRANCHID) || echo "SOMETHING WENT WRONG. CONTACT A GROWN UP"
else
  echo Umm. Liar. Go merge master to your branch, get jenkins to test it, and come back when it works.
  echo 
  echo INFORMATION ABOUT YOUR LIES: BRANCHID=$BRANCHID
  echo INFORMATION ABOUT YOUR LIES: COMMITID=$COMMITID
  exit 1
fi

D'Hondt results table for London MEP elections

There was just an election in the UK for the European Parliament (to elect MEPs), and it was interesting in the news because of the rise of one particular party, UKIP. The European elections use a proportional representation method, which UK general elections don't, so this doesn't mean that there will be a corresponding rise in UKIP members of parliament (at present, 0) at the next general election.

I voted in the London constituency, which has 8 MEPs elected in one ballot.

Wikipedia has a list of number of votes and number of MEPs elected, for each party.

Both the Green Party and UKIP got the same number of MEPs: 1 each. But their vote count was very different (371,133 for UKIP vs 196,419 for the Greens), and the liberal democrats, traditionally the third party in UK politics, got no seats at all.

I found myself playing with some "what-if" scenarios to better understand how the results came out.

The vote works like this: each elector chooses one choice on the ballot paper from a list of chosen parties - there were 17 parties on the paper, mostly small, fairly irrelevant ones.

The votes for each party are tallied, giving a vote count for each party.

Then, it is necessary to convert that vote count into a set of 8 MEPs that broadly reflects the proportion of votes. This is done here with the D'Hondt method which as an intermediate step needs a two-dimensional table. I'm going to omit the smaller parties here because they don't have an effect on my scenarios.

Party	Count	/1	/2	/3	/4	/5	/6	/7	/8	/9	/10	/11	/12
Labour	806959	806959 (1)	403479 (3)	268986 (5)	201739 (7)	161391 (11)	134493	115279	100869	89662	80695	73359	67246
Conservative	495639	495639 (2)	247819 (6)	165213 (10)	123909	99127	82606	70805	61954	55071	49563	45058	41303
UKIP	371133	371133 (4)	185566 (9)	123711	92783	74226	61855	53019	46391	41237	37113	33739	30927
Green	196419	196419 (8)	98209	65473	49104	39283	32736	28059	24552	21824	19641	17856	16368
LibDem	148013	148013 (12)	74006	49337	37003	29602	24668	21144	18501	16445	14801	13455	12334

So the 8 seats were chosen as the top 8 "votes / seats" quotients. Those are coloured yellow in the table. I've also numbered the winning positions and the next 4 after that in order of "votes / seats".

So there's a difference there between the Greens and UKIP: UKIP was chosen 4th, with a solid block of votes to get its single seat. The greens were chosen last, and only just got a seat.

What do I mean by "only just"? Well, the next seat allocated if not for the Greens would have been a second UKIP seat (numbered 9 in the table) and to get that they would have needed 2 * 196419 = 392838 votes to beat the Greens: 21706 votes more than they actually got.

Or conversely, if the greens had got less than 185566 votes (so 10853 less than they really got), they would have taken 9th place, behind a 2nd UKIP seat taking the 8th seat. If that was the case, then the table would have looked like this:

Party	Count	/1	/2	/3	/4	/5	/6	/7	/8	/9	/10	/11	/12
Labour	806959	806959 (1)	403479 (3)	268986 (5)	201739 (7)	161391 (11)	134493	115279	100869	89662	80695	73359	67246
Conservative	495639	495639 (2)	247819 (6)	165213 (10)	123909	99127	82606	70805	61954	55071	49563	45058	41303
UKIP	371133	371133 (4)	185566 (8)	123711	92783	74226	61855	53019	46391	41237	37113	33739	30927
Green	185565	185565 (9)	92782	61855	46391	37113	30927	26509	23195	20618	18556	16869	15463
LibDem	148013	148013 (12)	74006	49337	37003	29602	24668	21144	18501	16445	14801	13455	12334

Another thing that I think is interesting is just how badly the Lib Dems did. To get a seat, all other votes being equal, they'd have needed to get up to that 196419 to steal the 8th place off the Greens: that is 196419 - 148013 = 48406 more votes, 32% more than they actually got.

A different way of looking at that is considering if there were more than 8 seats, how many more seats would there need to be for the Lib Dems to get a seat to represent their proportion (about 6%) of the electorate?

The numbers in the first table above beyond 8 show that: the Lib Dems are 12th in line for a seat, after This table shows the 11 yellow coloured seats ahead of the lib dems, one more seat for each of Labour, the Conservatives, and UKIP.

Party	Count	/1	/2	/3	/4	/5	/6	/7	/8	/9	/10	/11	/12
Labour	806959	806959 (1)	403479 (3)	268986 (5)	201739 (7)	161391 (11)	134493	115279	100869	89662	80695	73359	67246
Conservative	495639	495639 (2)	247819 (6)	165213 (10)	123909	99127	82606	70805	61954	55071	49563	45058	41303
UKIP	371133	371133 (4)	185566 (9)	123711	92783	74226	61855	53019	46391	41237	37113	33739	30927
Green	196419	196419 (8)	98209	65473	49104	39283	32736	28059	24552	21824	19641	17856	16368
LibDem	148013	148013 (12)	74006	49337	37003	29602	24668	21144	18501	16445	14801	13455	12334

So many election tea-leaves for staring into in this election!

You can get the Haskell code on GitHub that produced the above tables, if you want to fiddle yourself.

paytag wristband

A year or so ago, one of my credit cards sent me a PayTag: a sticker with the contactless payment bits of a regular credit card, but without the other stuff (contact chip, embossed number, etc).

Their stated usecase was for sticking on your phone, as a sort-of low tech upgrade for phones which don't have NFC.

I didn't find that use case particularly compelling, and aside from comedy ideas like putting it inside a fairy-wand, I've been waiting for a use.

A few days ago I made it into a payment wristband. My right-hand wrist has loads of bracelets on it already, and I took one of those, some plastic packaging and some superglue and made a payment wristband.

I was a little wary at deploying this in use at first. My initial test was deliberately in a stationer's shop which had unattended self-checkout terminals. (I've used that same branch for RFID fun as a teenager in the past, where I had a anti-shoplifting coil in my pocket and set the alarms off every weekend). For 75p I ended up with a new card case and a successful initial test.

Next I went to Waitrose to buy my groceries. I was wary here because the daytime staff are angry dinner ladies who have confiscated stuff off me in the past(!). They didn't seem to bat an eyelid at me waving my jacket arm at their payment terminal.

Thirdly I went to buy coffee. This was more awkward. Their payment terminal was stuff under a shelf and looked like it was probably quite awkward to use even with a regular contactless card. The dude was a bit confused at me putting my empty hand towards him and waving.

My final test was in a pub. The contactless user experience is a bit different in most pubs: they use hand-held terminals and usually you hand over your card, they notice it is contactless and then use that. So I had to work around that a bit. I handed over my regular payment card and when he noticed it was contactless and went to swipe I drunkenly shouted "WAIT! LET ME USE MY WRIST!" which he did. At that point the card reader decided it needed to do dialup verification so there was a tense few seconds where I hoped I didn't look like a knob. But it worked.

I'm surprised at how unsurprised staff are at seeing this. I need to figure out the right way to start a contactless payment in a hand-held reader environment. I'm looking forward to being able to use this on the London Underground ticket gates later in the year too...

later: someone sent me this article about a contactless payment suit.

Haskell numbers

I just put the (still being worked on) slides for a talk in a few hours that I'm doing at the London Haskell User Group, on the subject of "Numbers".

Range-weighted average-score voting at battle of the bands.

Went to a "Battle of the Bands" contest. It was in a bar and the audience got to vote. Everyone was given cards numbered 1 point, 3 points and 5 points. Voting happened after each band played. We'd all gone to see a friend of a friend so the obvious strategy was to vote 5 for that band, and not vote for the others.

Turned out they did the voting differently here: they computed the average score rather than the total score. In a fixed audience, both average and total would produce the same ranking. But in this situation the audience changed. Battle of the bands is really "who has the most friends?" and people turn up only to see and vote for their friend.

So normally you'd vote 5 points for your band, and ignore all the others (who might be playing when you haven't even arrived yet or after you've left) and that would be equivalent to voting 0.

Using an average vote penalises you for this, though: your absence didn't count as a zero vote because you weren't in the total vote tally. To downvote a band that isn't your friends, you have to be physically present.

That seemed a neat way for the bar to make you stay and drink more beer.

xkcd 320 compliant watchface for Pebble watch

I released my xkcd 320 compliant watchface for the Pebble e-ink watch: xkcd320-1.1.pbw

It looks like this:

And the source code is here on github.

libpebble on debian

Getting my pebble talking to my linux laptop took a while - mostly because I didn't know much about the linux bluetooth stack.

I'm running all this as root. I might eventually figure out how to do it as a normal user.

Can we see the pebble at all? This doesn't always show the pebble to me - i think because its not waiting long enough for the pebble to be active on bluetooth.

# hcitool scan
Scanning ...
        00:18:2F:CC:EC:A9       Pebble ECA9

In one window:

# bluez-simple-agent
Agent registered

In another window:

# l2ping 00:18:2f:cc:ec:a9
Ping: 00:18:2f:cc:ec:a9 from E0:06:E6:BA:33:AB (data size 44) ...
44 bytes from 00:18:2f:cc:ec:a9 id 0 time 8.54ms
Send failed: Connection reset by peer

It gives that connect reset message every time I try this ping... but carry on anyway.

# rfcomm bind 0 00:18:2f:cc:ec:a9

next:

./p.py --pebble_id ECA9 ping

will make the pebble buzz and show a ping message on screen.

Now run as many p.py commands as you like.

At some point when I did this previously, it got a persistent pairing with the pebble - I have no idea where in my hacking around the persistent pairing happened as at one stage I had to do it for every interaction...