xkcd 320 compliant watchface for Pebble watch

I released my xkcd 320 compliant watchface for the Pebble e-ink watch: xkcd320-1.1.pbw

It looks like this:

And the source code is here on github.

libpebble on debian

Getting my pebble talking to my linux laptop took a while - mostly because I didn't know much about the linux bluetooth stack.

I'm running all this as root. I might eventually figure out how to do it as a normal user.

Can we see the pebble at all? This doesn't always show the pebble to me - i think because its not waiting long enough for the pebble to be active on bluetooth.

# hcitool scan
Scanning ...
        00:18:2F:CC:EC:A9       Pebble ECA9

In one window:

# bluez-simple-agent
Agent registered

In another window:

# l2ping 00:18:2f:cc:ec:a9
Ping: 00:18:2f:cc:ec:a9 from E0:06:E6:BA:33:AB (data size 44) ...
44 bytes from 00:18:2f:cc:ec:a9 id 0 time 8.54ms
Send failed: Connection reset by peer

It gives that connect reset message every time I try this ping... but carry on anyway.

# rfcomm bind 0 00:18:2f:cc:ec:a9

next:

./p.py --pebble_id ECA9 ping

will make the pebble buzz and show a ping message on screen.

Now run as many p.py commands as you like.

At some point when I did this previously, it got a persistent pairing with the pebble - I have no idea where in my hacking around the persistent pairing happened as at one stage I had to do it for every interaction...

list monad character to html entity encoding

This isn't particularly fancy, but it struck me as interesting from the "trying to abstract more" perspective.

Given a string with some funny characters in them, change the characters into HTML entities.

Here's a puffing-out function that given a character c :: Char returns a string (a list of characters, [Char]) for it to be replaced with. In the default case, the returned list contains only the original character.

generalPuffer c = case c of 
  c | ord c == 163 -> "£"
  c | ord c == 174 -> "®"
  c | ord c == 153 -> "™"
  c | ord c == 188 -> "¼"
  c | ord c > 128 -> error $ "unknown high character code " ++ (show $ ord c) ++ " encountered."
  c -> [c]

So how to use this? concatMap will handle this, and that was my first implementation (after fusing map and concat in my head).

concatMap f "£1 Shop™"

but concatMap is monadic bind, restricted to the list monad...

So equivalently you could write:

"£1 Shop™" >>= f

or

f =<< "£1 Shop™"

or

 do { l <- "£1 Shop™" ; c l }

BST

I wired in a rugby MSF clock into my raspberry pi, and wrote my own software to interface it to NTP.

Unfortunately I wrote it in winter when they were transmitting approximately GMT. Now its summer time and so they are transmitting British Summer Time not GMT; this is indicated by a bit in the time message. That I ignored. Oops.

Bed time project today was to add that in. Now I've got my stratum 1 server back.

certificate transparency

I went to a certificate transparency hack day at google london.

Dodgy x509 host certs really annoy me - they were a hassle when I worked on globus where we pretty much have out free host certs without any checking, to anyone who asked.

An achievable goal seemed to be to get a nagios plugin to check for certificates issued against a given hostname. I sort of have that working, with both OK/CRITICAL and a graph:

For now, the plugin is looking for certificates with the substring google in the subject name - there are plenty of such certificates in the log

When new certificates are discovered, they count as suspicious: they appear in red on the graph and the nagios notification system sends me an email. When an administrator (i.e. me) approves of the certs (by running an appropriate script), they turn into OK certificates and go green on the graph.

The underlying python code I'm using has a terribly slow ASN.1 parser, and so is only getting through a few hundred of the 2 million certs in the log every minute (see the blue line on the graph) - in a few days time hopefully it will have caught up. At least gives a pretty graph over time. In real life I'd expect a much smaller number of green certificates and hardly any/zero red certificates, as a flat line over time.

My original intention was to use this for matching domain names, but someone pointed out that it could be used to matching eg. trademark names anywhere in a certificate for some anti-phising detection.

Plenty of flaws:

• doesn't check certificate alternate names (subjAltName)
• doesn't check domain names at all
• doesn't check consistency of data coming from the log server
• doesn't deal with multiple log servers
• doesn't deal with multiple domain name probes efficiently (eg by caching or sharing download/ASN.1 decoding between domain names) - this is perhaps better implemented by using Nagios's passive plugin interface where a monitor could push interesting results (for various domains) into Nagios, rather than the present active/polling style (whichI chose because its easy to do)
• doesn't deal with unknown certificate extensions (at the moment, it ignores them which I think is sometimes the wrong behaviour - if the extension is one that authorises the use of new names (such as subjAltName does...)
• its fairly synchonous which is a bad thing for nagios probes - spending 2 days to verify the initial log is not good for a probe that should take less than 10s

Timesort

Standing at bar. Older gentleman comments how when he gets new missed calls appearing way down the missed calls list. conversation continues. After a while comments that SMSes also appear way down the list. Eventually I jokingly comment "is your clock set wrong?" to while he replies "yes- is that what's causing it?"

Mostly I find that interesting because it seems an unbelievably obvious problem if you've grown up immersed in computers, but isn't if you have different abstractions: post doesn't stack up on your doormat in the order of the clocks in your house; it stacks up in the order that normal time flows.

STV with restoration of eliminated candidates

STV eliminates candidates (and transfers their votes) until someone hits quota and win a seat. Then that winner's leftover votes are redistributed according to the next preference on those vote papers, and the process is repeated until there are no seats left to win.

One of the complaints about STV is that someone who is no ones first choice can get eliminated, even when it might turn out that they would get many second choice votes later on.

What happens if, after eliminating candidates enough times for someone to win, then in the next round those candidates are put back into the race, so can receive next-preference transfers?

It seems to make the counting process more complicated, but if you're doing this electronically, not devastatingly so. It puts candidates back in to receive transfers even if they weren't good enough to stay in the first round election.

What led me to wonder about this is some previous musing about proportional representation in the hereditary peers bit of the House of Lords, which is a bit like a stretched-out-over-time system that puts candidates back in for each seat.

shift+break

At school we had lots of BBC Micros. When I was near the start of school, these were very new indeed and almost none of the teachers or classroom helpers knew how to use them.

When you turned it on, you got a beep and a relatively unfriendly command prompt along the lines of: Acorn MOS >

To load and run the default program off a removable disk, they had a shortcut key combination: press shift+break to get that behaviour.

Easy, right? Except very few people understood instructions along the lines of "press shift and break together" to mean "depress shift and keep it depressed. Press and release Break quickly as if typing a letter. Release Shift".

So instead, minutes of pressing shift and break together, trying really hard to get them at the same time, would ensue at the start of each session until accidentally shift got pressed before the break.

ping error

Got this while pinging Google DNS from my mifi:

64 bytes from 8.8.8.8: icmp_req=302 ttl=50 time=2095 ms
64 bytes from 8.8.8.8: icmp_req=303 ttl=50 time=2107 ms
wrong data byte #52 should be 0x34 but was 0x45
#8      8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27
#40     28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 45 0 0 54
64 bytes from 8.8.8.8: icmp_req=305 ttl=50 time=2127 ms
64 bytes from 8.8.8.8: icmp_req=316 ttl=50 time=11091 ms
64 bytes from 8.8.8.8: icmp_req=317 ttl=50 time=10083 ms

Interesting to me that i) ping is doing more checking than I thought, and ii) there are link layers around that corrupt data (rather than drop packets) in reality not just theory.

Proportional Representation in the hereditary peers section of the House of Lords

The (UK) House of Lords has 15 seats elected by the approximately 800 hereditary peers, with a lifetime appointment.

As elections only happen on a seat by seat basis as they become vacant, for example through death; and never through general elections, many electoral systems cannot apply. The present system used is AV/IRV (single seat STV).

I'm not particularly interested in reform of the way this particular bit of the House of Lords, but I started wondering how one would achieve proportional representation in such a system: where you have seats where candidates are elected for life and do not stand for re-election.

To start with, why do I think there wouldn't be PR in the system described above? Because each seat is elected individually, by the same set of electors, so whichever power bloc was dominant in those electors would win every election. (so very similar to the problem of using First-Past-The-Post in House of Commons elections). Potentially 49% of electors could form a power bloc but end up with no representation. That doesn't seem to happen too much in the lords, though - the by-elections that have taken place seem to return a few different parties, but the turnout is low (50%) and so you only need 150 of the 800 potential votes to get a seat.

Because the seats are elected one at a time, many existing PR systems can't work. But I have been pondering if such systems could be adapted to this lifetime appointment, one seat at a time model.

Some PR systems (STV and Reweighted Range Voting) operate basically as:

• Everyone starts with a vote worth 1.
• A round happens, in which a single seat is filled. Any vote which was not used to elect that seat keeps its present value. Any vote which was used to elect that seat is reweighted to a lesser value so that it will have less impact in the next round; and ranking/choice information for that elected candidate is removed from each ballot.
• Rounds are repeated until there are no free seats.

(Some other systems, such as CPO-STV, are not round-based and so not amenable to this treatment)

My idea for adapting any voting system which has that form into the Lords model is as follows:

• Everyone starts with a vote worth 1, at the initial election of the first members.
• In the initial election of first members, one round will happen for each seat (so 15 rounds)
• Subsequently only one round will happen at one time, as a seat becomes vacant.
• When a ballot is reduced in strength by a particular amount, because it was used to elect a candidate, then that reduction is recorded against the seat (so in the Lords case, 15 seats x 800 electors = 12000 numbers)
• When a by-election happens, voters are given the new choice of candidates, but retain their weakened vote strengths from the previous round (previous by-election) except that any vote fractions recorded against the now vacant seat are added onto the vote strength of each voter.
• This procedure continues forever, one round per vacant seat.

I haven't done any mathematical analysis of this. But it feels like it might give proportionality: two distinct power blocs A and B should end up with seats in proportion to their relative strengths. When an A seat comes up for election, it will release a bunch of votes to the A-bloc voters and they can elect a replacement, if 30 years later they're still in the same bloc.

How would you add this into the present system where the 15 seats are already elected? Give everyone a vote worth 1 and start as if the next by-election is the first. As time passes, more and more seats will be elected with this method until (assuming each lords is mortal) all the seats are elected this way.

It needs to store a lot of information between elections: the product of the number of seats and the number of electors. This is feasible in the above case because only about 12000 numbers need be managed. It does not seem so easy when the electorate is an entire population.

In the lords case, there are almost no new electors: the set of hereditary titles is basically fixed and vote strengths can be attached to those titles rather than to the individual with that title at any one time. In the case of changing membership, new members must be given an initial voting strength. Giving them an initial strength of 1 would mean that new members would be more powerful than existing members in their initial vote. Another approach would be to encumber their votes against the existing seats based on some kind of average so that their vote strength is released over time.

(modified with a few more notes on 2013-07-31) (modified with CPO-STV reference on 2013-08-12)

mailinator-like service for QA

We're testing email based registration validation on a project. We started by using mailinator but a few times this has failed apparently because screen-scraping an ever-evolving AJAX UI is not particularly stable.

I decided to hack something up using my trusty old linux toolset.

Make a user account, maildump.

Get sendmail running.

Turn on FEATURE(`virtusertable`) in sendmail.mc

Add @maildump.example.com maildump

Set up an apach2 virtual host to ~maildump/web/

.procmailrc:

:0
/home/maildump/process.sh

process.sh:

#!/bin/bash

FN=$(mktemp)

cat > $FN

TARGET=$(cat $FN | formail -z -x "To: " | sed 's/[^0-9a-zA-Z\@\.\-]/X/g' )

chmod a+r $FN

mv $FN /home/maildump/web/$TARGET.html
exit 0

Now I wonder what security holes this has.

5 layer NAT

in a hotel I'm staying in, looks like NAT 5 layers deep... (yes they could be just routed with a final single NAT layer...)

 1. 192.168.99.1                                                      0.0%     6    1.2  15.4   1.1  84.9  34.1
 2. 192.168.20.1                                                      0.0%     6    2.4  36.4   2.3 206.1  83.1
 3. 192.168.1.1                                                       0.0%     6    3.8  28.7   2.9 154.5  61.6
 4. 192.168.10.254                                                    0.0%     6   86.7 107.1  77.9 195.2  43.9
 5. 192.168.219.250                                                   0.0%     6   28.2  56.0  27.2 141.7  46.8
 6. 195.14.158.85                                                    16.7%     6   26.6  55.1  26.6 115.7  35.0
 7. 195.14.158.145                                                   25.0%     5   31.1  33.7  30.9  39.1   4.7

Nominet .uk proposals again...

Nominet are proposing a second consulation on their wacky plan to open up second level domains in .uk - so I could have hawaga.org.uk or cqx.uk (assuming someone didn't get there first).

They talk now about owners of existing *.*.uk domains being given first dibs.

One of my concerns is that it fragments the UK namespace - there is already a "default zone" of co.uk and so that makes me wonder if (in the long term) co.uk should be abandoned entirely, and in the mean time, the co.uk and second level uk should be the same: one registration fee gives the same name registered in both zones. If you have the .co.uk, you get the .uk. If you don't have the .co.uk, you don't get the .uk.

That at least gives a bit of an easier user explanation "a .uk is the same as .co.uk used to be"; though its still loads of technical work on the site side for people who want such domains.

And it still destroys the history of having decent information about what kind of entity you are interacting with.

game theory

We played a new wide game with our Scouts.
Each kid in a team got a number written on their hand. To take someone prisoner, you needed to have a bigger number than them, either yourself or in total as a group of captors.
One group, C, cheated and increased several of their members numbers to 8, the maximum.
However, that backfired when we made the final score for each team be the number of points of their prisoners - team C had made its members so valuable compared to everyone else that another team, B, with just three team C prisoners scored more than everyone else.
It was satisfying that punishment for cheating was built into the rules.

mis-XML

There is a style of XML which I find quite frustrating. Here's a fragment:

    <mbeanNode>
      <name>java.lang:type=Memory
      <attributes>
        <attribute>
          <name>HeapMemoryUsage</name>
          <formattedValue>{committed=442433536, init=257990592, max=3670540288, used=103328176}</formattedValue>
         </attribute>
         <attribute>
           <name>NonHeapMemoryUsage</name>
           <formattedValue>{committed=57802752, init=24313856, max=224395264, used=57610720}</formattedValue></attribute>

This particular example came from XML output of javamelody, although I've encountered this misuse on a number of other occasions.

Most annoying is the guerilla formatting inside the <formattedValue> tag. XML already provides plenty of ways to pair up names and values, so why make a guerilla format that I now need to parse myself?

A smaller annoyance is the use of <attribute><key>X</key><value>V</value></attribute> style. XML already provides a mechanism for tagging a named key onto a value: <key>value</key>. The only defence of this I've heard is that it makes it easier to comply with a schema; but alas you miss the point and are also wrong. Wrong, because I can make an XML schema that accepts arbitrary XML tags; and missing the point, because the schema doesn't help much if all its doing is enforcing a skeleton.

I think all of this comes from fairly naive serialization of improperly defined application-internal data structures. suck.

Previously I wired a radio clock via a gertboard into my raspberry pi.

Since then, I figured out that I didn't need the gertboard there, and so now the clock is connected to my other "production" Pi directly onto the GPIO pins.

I've also put the C code I use to interface the radio into ntpd onto github. I needed to copy it from one Pi to the other, and doing it via github means you can download it too now. (yes, I know there are other implementations of MSF decoding around, but I wanted to write the code myself - this is supposed to be fun!)

radio clock NTP server

I've got a couple of Raspberry Pi(e?)s. One I'm using for real server stuff like DNS. The other is for hacking around on. I got a gertboard for it, but hadn't wired anything else up. Then I saw this MSF radio time receiver (for 9 GBP) which is intended to receive 60kHz time broadcasts from the UK National Physical Laboratory.

When I was a teenager, a radio clock was a cool thing that I never had. Then later I got an internet connection, and with it NTP, and so I got geek-time a different way. So I never got an MSF clock until now.

The receiver plugs into the Gertboard (actually it can maybe plug directly in to the Pi's GPIO pins without a Gertboard in the way - I'm not sure) and presents a 1-bit serial signal that is the demodulated 60kHz signal. This is a slightly strange looking signal mostly consisting of 100ms of no carrier, two bits at 100ms each then 700ms of carrier, with 60 seconds of data being enough to transmit the current time.

There was some fiddling soldering to do to get the pins connected - turns out 10y old crocodile clips don't cut it. My father, being better than me at soldering, put those on for me. I cut two of the Gertboard's jumper cables in half so that I'd have wires which connect straight onto Gertboard pins.

The layer above that needs to run in software, at least in my setup.

The first program was a scope program that output * for 1 and . for 0 to the console 80 times a second or so, with a CR every second, to give a line-by-line view of the time. That showed the signal pretty clearly, like this:

.*******************............................................................
.**********.....................................................................
.*********************..........................................................
.**********.....................................................................
.*********......................................................................
.******************.............................................................
.******************.............................................................
.************...................................................................
.***********....................................................................

The short lines are when there's a sync pulse only, and the long lines are when there's a 1-bit immediately following the sync pulse.

The second program used a minute-long circular buffer as a shift register (thanks to Dave_H from the #a&a for that suggestion), and tries to decode time when one end of the buffer looks like the start-of-minute sync pattern. This was able to pretty accurately decode the time. I didn't implement parity checking or other error checking (for example, checking the sync pattern is exactly right) because of that accuracy, which gave trouble later on when I moved the antenna around. Thats something I could implement, and give out some accuracy metric. This took a few hours to code in C (yes, I wrote something in a language that wasn't Haskell). The code is pretty inefficient because it hard polls the GPIO pin. There is some edge triggering available in the Pi's GPIO implementation so I could investigate that. So that ended up as a program which outputs the current time every minute, to the console.

Next, I wanted to connect this with ntpd, so that I could compare the time with other internet time sources, and share the MSF signal with other hosts on the same LAN. ntpd has a facility for connecting "reference clocks" which are sources of time that are not other ntp servers. One of those, the SHMEM driver, communicates with a separate process using a shared memory buffer. So I got to learn about Linux shared memory (indexed using integer keys, I discover, rather than as nodes the file system). For that protocol, basically you stuff the current system clock time and the current received radio time into two fields in that shared memory, every time you get a clock reading. That was a straightforward addition to my second program.

So here's what the hardware setup looks like. The big board is the gertboard. Under it with the big wires is the Pi. The tiny tiny board held in the air by the jumpers is the MSF receiver board, and the big metal rod at the front is the antenna.

And here's what I see in ntpd:

 $ ntpq -pn
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+2001:8b0:0:53:: 195.66.241.3     2 u   38   64  177   27.872   -4.596   8.680
-2001:8b0:7c:1:2 61.69.233.69     3 u   39   64  177    0.597   -2.285   8.710
+2001:8b0:7c:1:b 90.155.53.93     3 u   37   64  177    1.000   -3.868   8.395
*127.127.28.2    .MSF.            0 l   44   64   17    0.000  -23.730   6.885

where NTP has recently synced to the MSF driver (with fake IP address 127.127.28.2). Its about 20ms different from the network-connected NTP servers (see the offest column). Its difficult to know how much of that is from NTP over-the-network inaccuracies and how much is from my code, but I suspect the bulk of it is from my MSF code - my polling just isn't accurate enough at the moment, and my parsing has some weird time alignment stuff in it.

All in all, good fun and blinkenlights!

p.s. you're welcome to point your own ntp servers at this for fun. Add server tyne.cqx.ltd.uk to your /etc/ntpd.conf

p.p.s. is it sad that I can decode those octal reachability fields in my head.

scripting languages

another half-serious entry to the scripting languages vs programming languages debate: a scripting language is one which has decent string handling, rather than telling you to man-up.

brief facebook commentary in response to I'm supposed to give a talk on software stuff to college students Wednesday. Kind of a "things that would have helped to know before my first job." What would you put in there?:

possibly they already read "the mythical man month". now they just have to learn to actually believe that most of what he says in that book is relevant; most of what you do in colege is one person doing a project. almost nothing in paid software is that, even when you're the only programmer; use version control goddamnit; if 5 different people/groups have written some thing, and you don't like any of them and are going to write your own thing but "do it right this time", well there's 5 examples of people who tried to do it right and failed so you're feeling pretty cocky, aren't you? when your boss asks you "can we do X" and your answer is "yes (given 10 years and 50 developers)", your boss heards "yes (i can have it tomorrow)".

also, accept that you will not write bugfree code. its not a sign of being a pussy. its how code is. write tests. or, if you think you can write bugfree code, bet someone your next paycheck that its bugfree, if you're so confident. use the machine as your helper for that , and ge tin the habit of 1000 unit tests run while you go take a shit. i mean, before you commit your code.

(run the unit test son the other pussy's commits at least, and find the bugs they committed and make it seem like you are amazing)

london traffic pirate announcement

In-car radios in the UK often have a feature called Traffic Announcement, transmitted as part of RDS, which lets the current channel (or another non-radio input such as CD/mp3 player) be interrupted with a traffic programme being broadcast on another channel.

Driving on the M25 and listening to BBC Radio 1 the other week, the traffic flag clicked on and switched me to receiving some dodgy pirate radio station!

I wondered if this was accidental. Maybe not so accidental though - it turns out many of the pirate radio stations round London transmit RDS.

nagios and andrews and arnold SMS notification

Andrews and Arnold sells me a VOIP service which also lets me send SMSes over HTTP.

I wanted to make Nagios send out notifications this way, in addition to the existing email-based notification.

First I need a Nagios command definition to send out the notifications:

define command {
  command_name cqx-notify-service-by-sms
  command_line /var/cqxmon/secret-aa-curl 0781234567 "$NOTIFICATIONTYPE$: $HOSTALIAS$/$SERVICEDISPLAYNAME$ is $SERVICESTATE$"
}

That will invoke my sender script to send a message to phone number 0781234567.

Then we need the sender script. It has my password hardcoded in it (ick! but that could be replaced with a < redirect).

#!/bin/bash

# DO NOT COMMIT THIS TO VERSION CONTROl!

TARGET=$1
shift

TMPFILE=$(mktemp)

echo $@ > $TMPFILE
curl --get --form-string username=0300300300 \
              --form-string password=XYZ123 --form-string destination=$TARGET \
              --form message=\<$TMPFILE \
              http://sms.aa.net.uk/sms.cgi

rm $TMPFILE

The redirect via TMPFILE is because I was having trouble with spaces on the commandline that I couldn't seem to escape around.

Then I need to modify the Nagios contact definition for myself to call the cqx-notify-service-by-sms command in addition to the regular email notifier. In the contact definition for myself:

service_notification_commands cqx-notify-service-by-email, cqx-notify-service-by-sms

That will work for services. For hosts, you'll have to create a cqx-notify-host-by-sms command which does something similar, and add that to the contact notification_commands.

Nagios has a number of other contact fields that could be used to allow the target number to be configured per contact. I'll get round to that at some point; without making those changes, the above will only let you notify a single SMS number.

pronunciation of Haskell applicative operators.

I gave a talk on parsers in Haskell yesterday, and asked the audience how they thought I should pronounce some operators.

They came up with:

<* left looking sparrow
*> right looking sparrow
<*> goatse

If you don't know whatis goatse, you don't want to know. Really. But it made me laugh.

parser combinators talk at London HUG

I'm doing a talk on parser combinators in Haskell, in London on 23rd Jan 2013. Come along if you like.