mysql rsync backup doh!

Well, I moved a big live database from one server to another for a customer. I was planning on taking the site offline first and doing an sql dump/move/restore, but someone accidentally deleted the old server before they were meant too (oops) - leaving me with some rsync backups that I'd made of the whole server over the previous days, to guard against such an eventuality.

Turns out rsyncing a live mysql database is not the right way to do things - I ended up with a database that managed to get mysqld to actually crash when I tried to dump out the database contents. I managed to fix this by dropping the tables that it was crashing on: it conveniently told me which ones it was crashing on, and it let me drop them, just not dump them. Luckily they were all cache tables so there didn't seem to be harm in dropping them and recreating them empty.

It also turns out (obviously, post facto) that mysql databases don't play nicely with rsync's --link-dest option, which is supposed to reduce disk space on the target file system when making multiple snapshots. Because it only optimises away duplicate data for identical files, but mysql stores the entire database (to a first approximation) in a single file, hiding its internal structure from rsync, and so the whole db gets duplicated each time.

Taking an SQL dump should solve the first problem. Not entirely sure what the right way to do an incremental backup of the database is. Maybe I shouldn't even try.

batch checking DNS delegation

I'm working with someone who has ten or so domains. All the domains are registered in different places, and with slightly different DNS settings. As part of tidying that up to get everything consistent, I wrote the following bash+dig script to display the delegation of each zone from its parent.

That is importantly not the same as what the name servers for the zone return as NS records. The "authoritative" source of NS (nameserver) records for a zone is that zone itself. Using dig to query the NS records seems to be returning those, unsurprisingly.

However, in order to query a zone, there is a second place where name servers must be configured, separately from in the zone itself. That is in the parent zone. If those are wrong, then you can get awkward to diagnose problems: you can see from dig that the nameservers are right, yet lookups go to the wrong place.

Hence my script.

You can see even on my own domains there's a slight misconfiguration: barwen.ch claims to have s0.barwen.ch as a server, but the .ch registry isn't delegating to it. That won't cause bad DNS lookups but will cause s0.barwen.ch to not be used as a nameserver sometimes. Worse is when the delegation points to an old server that then returns a new servers DNS, giving the illusion that all is well, until you turn off the old server (which is the problem I have on other zones)

$ ./list-domains-NS.sh 
ZONE hawaga.org.uk
NAMESERVERS ACCORDING TO GOOGLE DNS
dildano.hawaga.org.uk.
paella.hawaga.org.uk.
NAMESERVERS ACCORDING TO org.uk
hawaga.org.uk.  172800 IN NS paella.hawaga.org.uk.
hawaga.org.uk.  172800 IN NS dildano.hawaga.org.uk.

ZONE barwen.ch
NAMESERVERS ACCORDING TO GOOGLE DNS
paella.hawaga.org.uk.
s0.barwen.ch.
dildano.hawaga.org.uk.
NAMESERVERS ACCORDING TO ch
barwen.ch.  3600 IN NS dildano.hawaga.org.uk.
barwen.ch.  3600 IN NS paella.hawaga.org.uk.

So here's the script:

#!/bin/bash
cat domainlist.txt | while read d ; do
  echo "ZONE $d"
  echo "NAMESERVERS ACCORDING TO GOOGLE DNS"
  dig @8.8.8.8 -t ns $d +short

  PARENT=$(echo $d | sed 's/^[^.]*\.//')
  echo "NAMESERVERS ACCORDING TO $PARENT"

  PARENTNS=$(dig +short -t NS ${PARENT}. | head -n 1)

  dig @$PARENTNS -t NS +noall +authority +norecurse $d

  echo
done

country selector

I've had a rant queued up on this blog but never completed about various forms of internationalisation/localisation.

One part of the rant is about web forms that ask for your country. A common way to do this is a drop down list of "every country" (deliberately in quotes). That's awful.

Its especially awful when you are from simultaneously from: Britain; United Kingdom; Great Britain; and England, all "countries" in someone's definition or another - not only do you have to scroll down to your country, you don't even know which of the four countries you are meant to scroll to until you get there and find that country missing.

I saw (on hacker news) this redesigned country selector: http://baymard.com/labs/country-selector which presents an autocompleting text box.

Type "eng" and it brings up "United Kingdom". woo!

It also converts "deut" into "Germany", but fails to convert ”日” into Japan.

beep is not useful.

sometimes I get a beep on my laptop. my console has ~10 applications on 9 desktops, including being remotely connected to ~5 machines. a beep without any context is not useful.

POSSIBLE BREAK-IN ATTEMPT (not really)

SSH gives out error messages like this:

Sep 28 09:50:09 s0 sshd[27967]: reverse mapping checking 
                  getaddrinfo for adsl86-34-217-144.romtelecom.net 
                  [86.34.217.144] failed - POSSIBLE BREAK-IN ATTEMPT!

Why does it label it as POSSIBLE BREAK-IN ATTEMPT!? How is it more of a possible break-in attempt than a user attempting to connect more than a few times with a wrong password? This has bugged me a bit recently when helping a few people who aren't really used to linux - its shouting at them that something is SERIOUSLY WRONG!!! and when they look through their log files, they've fixated on this (as far as I can) relatively minor misconfiguration of a remote user's network.

♫ if i could turn back time ♪

Cooker has a one-way-only knob for setting time. You wind forwards until you reach now. If you miss now, you wind forwards another 24h (luckily the cooker doesn't have a date too) until you reach now again. On TZ shift (i.e. end/start of summertime / daylight savings), once per year, there's a wind-forward 1h event. fairly easy. you make the clock run faster than real time. but once per year there's a once per year wind-forward 23h event. BUT! Turns out if you can stop the clock, then you can let real time wind forwards faster than clock time. and wait an hour. and then start the clock again. It involves more real time (you have to wait an hour) but a lot less tedious winding.

wikipedia find-of-the-day: electronic calculator abacus

linked on the wikipedia page for Soroban

5 centuries ago...

I just queued up jobs for a physics simulation that are 5 compute-centuries long:

6 jobs x 11.5 hours x 16384 nodes x 4 cores/node = 5.16 centuries.

I'm expecting the runs to be finished in about a week

If this ran on a single core and I wanted it to be finishing in about a week, I would have had to start them in the year 1495 (the most interesting wikipedia note for that year is Friar John Cor records the first known batch of Scotch whisky).

frustrated-with-slow-simulation haiku

microseconds pass
simulation time stretched
long as winters night

I can't complain too much though - this was 65536 cores on an IBM BlueGene/P

today is blog day

Today is blog day (well, today isn't, it's the 19th of May, but hurrah for autopost)

I'm supposed to recommend 5 blogs to you.

I'm also supposed to find *new* blogs. But FTS, I'm too lazy. Plus I know the ones listed here are ones that I know I go back to day after day and find useful.

Oglaf is some dude's NSFW comic that started out as an attempt at cartoon porn but now is mostly funny. The shield maiden is my favourite post.

Hacker News from ycombinator - kinda like slashdot in that its a lot of interesting geek links. More links. Less discussion. Feels more intimate, and more focused around the bay area startup crowd.

Lambda the ultimate - a programming languages weblog. A cross between programming and maths. If you like Haskell, you'll like this, though all sorts of language stuff (theory, design) is talked about. You can also laugh at the "amusing" paper titles that authors come up with.

In a bit of shameless self promotion, I'll point you at the drupal news feed for my hobby project, barwen.ch - this is partly blog, partly just a web page change tracker.

Finally, Bruce Schneier's security blog. Aside from his tendency to ramble on about squids, this is pretty interesting. It evolved out of his company newsletter, and talks mostly about computer security but also about other related topics such as social engineering and terrorism. And squid.