"The Butler did it": Forbrydelsen and the problem of depth-first search in a data-rich world

(going to try and put proper caps in here)
(plus, SPOILERS!)

So Forbrydelsen (the killing) is over, and guess what? The butler did it.



Well almost; Vagn, the best friend, the persistent companion, handyman, friend to the children, in every episode, always hanging around, never suspected (until the twisty ending)... it couldn't be him, could it? Why, he had an alibi, and no motive*, and didn't fit the psychological profile. Never mind all that - looking at the structure of the story, he's the only person it could have been**. This hurts, actually; for its 20 episodes, forbrydelsen was stuffed full of believable, well-drawn characters, performed sensitively; and they sacrificed them for the sake of a hack ending that jarred. I spotted it was going to be Vagn around episode 10, based soley on the fact that he was the only character to be always there, and my mum claims an impressive episode 2 (she's read a lot of detective books). You could tell it was going to be him, not because of how he acted, but because of his position in the structure and narrative, and that is annoying.

I don't think there's been such a good dissection of life for a grieving couple since 'don't look now', even if the stages of grief seemed to be compressed into three convenient weeks. This is at odds with the contrived structuring of the program, where the car she had been found in appears to have been driven by no less than four people that night, and everybody has a secret alibi, rather than a motive, as would be the case in a classic poirot investigation. Interestingly, "In Germany they liked it as a whodunnit, but you guys seem to be more into the characters."

Now that that's off my chest, I want to talk about something else: the methods of the detectives. not from a police procedure point of view, but in terms of data search.

In search analysis, there are two basic methods: breadth-first and depth-first. A problem space is a network; each node is a piece of evidence and you're looking for the node that gives you the critical evidence that links a particular person to your starting node, which is 'there has been a murder'. You don't know where this node is, so you need to explore the problem space. So you have a choice of how you go about exploring it.

A depth-first approach says

1: Take the first unexplored node attached to your current node.
2: Check if it's the end state.
3: if yes: exit with success! otherwise, goto 1.
4: if there are no other nodes left to explore: backtrack to the previous node and goto 1.

In this diagram, the red lines show which nodes are connected and the black arrows show what order the nodes are investigated.


This recursive algorithm produces a burrowing action that drills down as fast as it can to the bottom of the data. It's unlikely to produce a correct answer straight away (except by luck), but produces more interesting television; it's the technique used by Lund and Meyer on Forbrydelsen. They picked a suspect, assumed they did it, harangued them to death, and ultimately found out the were innocent. And OK, I think the detectives generally handled themselves pretty well given the absolute bat-shit-mad number of people who didn't come forward with vital information because they had something really really important to hide. Then again, the convoluted plot somersaults need to keep the series going for 20 episodes are pretty shameful. For just one example - why did the initial driver of the car not know that it had been picked up when he went home ill? why did the security guard not come forward with blatantly vital information, which would have cut out three suspects (by my count) and about 8 days of police work.

And worse: why did Meyer cryptically say 'Sara 84' on his death bed, rather than 'Vagn shot me'? he knows what vagn looks like and the implication that vagn goes round with his face covered is just to silly contemplate.

Also Vagn - if you're going to steal a vitally important photo from a photo album, don't leave the page saying which photo has been taken in the book: Take the whole book. It's quicker and easier and it's what you would have done without even thinking about it.

Ok, so that was more than one example but so much of the plot was too silly.

So I'll move onto breadth-first search, and in doing so, talk about why I think it's relevant. Breadth-first involves checking nodes for success depending on how close they are to your start node, as illustrated here:



So notice that in this case, the search procedes layer-by-layer.

I like to think about these searches as 'lines of inquiry'. From your starting point, you have several suspects, and as you open up these lines of inquiry, you have different branching facts of evidence, motive, alibis and inconsistencies, &c.

Take the classic Poirot scene: a lounge, with all the suspects gathered, a few days after the event, as Poirot reveals the sum of his findings. The detective has spent days combing the area and suspects, accumulating evidence, and let the facts speak for themselves. With a little creative thinking, he is able to read between the facts, and given a room full of people who all have motives and windows of opportunity, is able to pick out the murderer. He is a breadth-first searcher; he does not accuse anyone until all the facts are out.

I read PD James' "a mind to murder" not long ago, and it's a great example of breadth-first detecting: virtually the first half of the book is police interviews with occupants of the house where the murder took place. That's how police work should be done, but to be fair, I didn't enjoy the book. It was too long before any actual, you know, detecting went on.

This is where the 'hunch' comes in and rescues us from overly-procedural police work. once the basic facts of the scene are established, the detective goes off on their own, follows their hunch, maybe makes a few illegal short-cuts and compromising mistakes. I didn't get the sense of any hunches in Forbrydelsen; it was all or nothing.

In the diagrams above, if the solution to the murder was at node 10, both methods would be as inefficient as each other. But in Forbrydelsen, it was at node 3: the best friend (in this analogy, 1 is Nana, 2 is Theis). And it took the detectives over two weeks to even question him. In fact, in every case, they only took people in for questioning when they suspected them - and this is why I felt, in 2011, let down by the series (which was, in its defence, made in 2007).

We live in a data rich age. The Petabyte Age, to quote Wired. They talk about the end of theory: living in a time when you don't need models and theories because we have so much information. Who needs a theory of evolution when we have so much data on so many different species? The argument goes, we don't need a model, we have the real thing. The first task in police work should be to accumulate as much evidence as possible - regardless of meaning. Once you've got everything laid out in front of you, the suspect should simply emerge.

The point is, detectives have two resources: facts and hunches. The two build on each other, with facts leading to hunches and vice versa; you start with facts, you need to end with facts, but along the way you can develop a hunch to get you through the dry times and give you some direction - called a heuristic in the search lexicon. Too much fact and not enough hunch, and you end up in staid 'a mind to murder' territory. Too much hunch and not enough fact, and you get Armstrong & Miller's 'Force on the Case', where a raving alcoholic ex-policeman repeatedly accuses his book-shop rival of being a murderer.


But in the Petabyte age, we don't need hunches or theories. we just need to accumulate data. That's what I think is sad about forbrydelsen: they don't use facts or hunches. They just blindly follow and exhaust linear enquiries, without considering the whole picture. It's one thing to be overly-procedural, and it's one thing to be a loose cannon; but the detectives in Forbrydelsen are neither, with a paucity of data and no hunch to give them direction. Their only strategy is pure depth-first search.

So the problems with Forbrydelsen are threefold:

1: the answer felt cheap (like they clumsily lumped together a crime of passion carried out by a serial killer),
2: there was the fact that anyone of about 10 people should have come forward with information, but had something else to hide (which is pushing it, and just for the sake of throwing in red herrings and prolonging the series), and
3: the detectives just didn't seem very good at their job - relying on too little data and accusing people to readily. But then, I suppose this follows from point two, as I said earlier, what could they do, as rational actors with a plot this convoluted?

So I'd like to see a 21st century, web 2.0 detective, totally data-led in his methodology. In short: Ben Goldacre PI.


*Yes, he had motives to stop Nana leaving the country, but not a motive to ritually rape and murder her. The killer was described as methodical and not impulsive, while Vagn's actions are totally off the cuff and unplanned.

**The only other person it could have been was Morten, for the same reason of being a 'big bad friend' of another protaganist. It turned out he had his own twist.

two new demos

there is a funny story attached to these songs.
I bought a new mixer, and as well as learning to use Reaper, i've been learning to get the best out of it.
I had problems with latency at first, because i'd created an awkward setup , and it's got two speaker outputs - monitor and control room, with importantly different mixes.

Recording through it, i've got both line out and microphone from my guitar amp. but on closer inspection, both the outputs recorded to disk from these inputs had been identical - and what's more, hugely hissy, with only a 12 dB signal to noise difference. i couldn't believe my equipment was this bad. having a fiddle around, i starter to notice something strange, and eventually confirmed this - even with all the levels zeroed, the gain set to zero, and finally, desperately, all inputs to the mixer unplugged - the computer still recorded this awful sounding input. i didn't for the life of me understand.

the culprit: a webcam with built in microphone, using identical USB audio codec drivers, therefore coming in on the mix. with the quality sound.

on 'yentl' below, you can hear the difference in sound quality with the piano sound, which was added after i fixed the problem.

anyway, here's a couple of lo-fi tracks based around picking patterns i wrote recently, having taken paternity leave more than a week before i should have done. but waiting was doing my head in at school. oh my god, a feeling. i haven't expressed one of those on here for a hugely long time. i'd better log off and let you listen to these tracks.

"Info for nerds":striking similarities between the packaging of bruised pilgrim and hello sailor..!

1. Use of the 'exclamipsis'

2. Black cover with iconic image centre.

3. Gatefold card CD case.

4. Band picture has some members obfuscated/not included.

5. Tracklisting (centre-aligned) makes reference to two-sided nature of vinyl.

6. One band member standing to the right of the frame, looking left.

7. Surprisingly colourful interior.

8. Sleeve notes in light, serif, font, aligned to edge of packaging, on black background. Sleeve notes have several similarities of content, including the mention of the name 'Andrew Gardiner' at least three times, in mixing, producing, and additional vocal and percussion capacities, plus 'Colerabby Studios'

"Lumious beings are we, not this crude matter": towards a new RPG paradigm

(i have a lot of problems with this video, e.g. if size matters not, why can't yoda just crumple the death star into a tiny black hole? if 'try not! do or do not, there is no try', how do i deal with uncertainty - somethings just end in failure no matter how hard you try, through no fault of your own, because there were factors you couldn't know about. but anyway, onwards to my new paradigm)

while waiting for a baby that simply doesn't want to drop, i've been playing a lot of desktop dungeons. it's a lot of fun, and very easy to get far too obsessed by. but while wishing that i could throw 'half a fireball' at a pesky enemy who only has a couple of hp left, i am reminded of an old idea i had that i never got around to developing.

imagine a plane of luminous beings. beings with no corporeal element - their spiritual side is their life force. their energy is their blood. they spend their own power, diminishing themselves, in order to hurt or heal others.

i'm talking about a two-fold change to the typical rpg system -

1: a spell's power is determined by how much power the player chooses to invest in it;
2: no mana bar, no hp bar, no xp bar - just one bar of fluid energy.

balancing these ideas would be tricky. it would be very important not to make it a zero-sum game: if i spend 14 energy points on a spell, then it's no good if it just takes of 14 of your energy points, because then it's just a case of 'who's got the biggest health bar?'.




so from the first point: when a player starts the game, all spells are available to them, but being a noob, they can only use them lightly. there would be a bell-curve effect to the strength of the spell - not a deterministic one. so investing a little power would produce anything from a fluff to (rarely) a medium sized explosion, while increasing the power would stretch - not move - the probability distribution, making fluffs rarer and opening up the potential for devastating explosions of energy. they could try to reach beyond their means, overspend their energy, but the more they overspend by, the more likely it is that it'll be a complete disaster.

from the second point: you don't spend your mana, you spend your you. in order to damage your opponent, you must spend the one gauge you have. there's no need for levelling up, as xp extends the size of the gauge; and receiving damage costs you the ability to damage people back. so there's no real difference between a tooled-up beginner and a wounded veteran.

and this may sound like a problem, but maybe it's another strength. being wounded actually hurts you - and i think we have to back to deus ex to find another example where being hurt actually impairs your character's performance. hurting your opponent means they're less able to hurt you back, and vice versa, making battles struggles to the death as characters fire off at each other with less and less energy. now, being low on health doesn't make strong attacks impossible, it just makes them riskier. the powerful energyblast you need to finish off your opponent looks like a safe bet, until they bring you low. you can still attempt it, but what's changed is the probability that you'll get a positive effect.

therefore there needs to be no absolute zero to the gauge - but past a certain point your attacks become entirely impossible and while still alive, you are effectively neutralised. do this to an opponent, and you've won the battle and can receive your xp (i.e., your gauge goes up). lose the battle, and you need to go and find somewhere to regenerate.

spending a small fraction of your power would produce a confident, low-power attack - low impact, but low standard deviation. therefore a powerful being could cast weak spells more confidently than a noob. the larger the proportion of your energy bar, the more risky - postively skewed - your attempt becomes.

here, let me explain with graphs:
So let's consider two characters - a low-power one (Gilgamesh) and a high-power one (Enkidu). they both attempt to cast a green-powered spell, meaning they both need to put in the same amount of energy. the amount of energy is a greater proportion of Gilgamesh's force bar than Enkidu's, so while the maximum output is the same, Enkidu's probability distribution is more skewed towards this maximum than Gilgamesh's. therefore the mean value of the spell would be higher.

In the second half of the picture, we see Gilgamesh over reaching himself to compete with Enkidu, who confidently attempts the high-power spell. There's a very small chance Gilgamesh reaches the heights that Enkidu easily gets to. What I can't easily illustrate here is the negative affects that a player would be risking by living beyond their means; from this illustration it would appear that it's really worth going over the top, but that's not my intention. it's just really hard to illustrate negatives on the electro-magnetic spectrum.

it has to be balanced so that the risks of over-reaching your power outweigh the benefits, effectively cutting off anything beyond what you should be capable of at any point in the game. this basically brings the system back in line with more traditional methods of 'you can't cast that spell because you're only a level 3 wizard' style mechanics (hell, even characters in the book 'nightwatch' complain about that kind of thing) but i like the expandability of it, the fact that what's stopping you isn't the ability to cast it, just the unlikeliness of it having the effect you want, which is more like how our universe works: it's not held in place by absolute laws, but by the sheer weight of probability.


These two ideas could be combined with a modular, diy spell building system like that of magicka, meaning that you could still restrict the kinds of spells users could cast by not letting them have some key rune, glyph, grimoir, or other way of encoding knowledge. thus maybe you had to learn or subscribe to different elemental powers.

in conclusion, i guess what i'm after here is a way of simplifying rpg tropes into one measure of success, plus a way of scaling magic so that it's accessible to all. back an opponent into a corner, beat them down and they'll either flee or try something super-risky, just when you're at your weakest. personally i think this mechanic has a lot of potential, especially for a computer-based game, where the maths doesn't have to be done by hand. it just strikes me as a really fair way to build a universe.

OGRY

Ogry by Dj Gallowslutt on Mixcloud

Ogry.mp3

wtf, google?

reviewing the king of limbs after one listen

most people threw up some opinions on the king of limbs after just one listen, the day it was released. i'm going to happily share my opinions a few weeks after release, still having only listened to it once.

i listened to king of the limbs in it's entirety on headphones on the way to work, having never heard it before (except for putting it on in the background when we had guests, after 4 other new albums, so it was kind of lost in the mix)

I got about half way through, and realised i was still waiting for something to happen. by the end, i'd heard a couple of fairly nice songs (in a supra-keane sort of way) in the second half, but wasn't satisfied.

listening to an album once is never enough to know it; none of my favourite albums have been so highly regarded after just one listen, and that's especially true of radiohead, whose albums i've always found to be growers. but king of limbs just annoyed me; after one listen, it seems closest to thom yorke's pitiful 'feeling pulled apart by horses' single, while also harking back to the absolutely classic amnesiac b-sides. but four tracks in and i dunno, i just hadn't heard any songs, or riffs.

'In Rainbows' was fantastic. I might not have realised how good it was on the first listen, but it didn't put me off. I don't mean put me off in a 'slipknot' kind of way - hearing that when i was 17 was thrilling, but too much - or in a 'trout mask replica' kind of way - as in, the genius could initially only be drank in in small quantities. or Race Horses' 'goodye falkenburg', which i initially thought was a limp version of their live set, but after a few listens i absolutely adored. not like those; this feels more like ephel duath's 'through my dog's eyes', an album I never found a space for. listening to 'limbs' was a frustrating, empty experience. I want to go back to it because radiohead have never let me down before, and others have said it's good, but i honestly don't look forward to the experience or listening to it again.

So, how did listening to it actually make me feel? like a twerp. The lack of publicity just underlined the disappointment for me, it's like they just weren't bothered by it. they'd done it, they were going to shit it out regardless. it's all puffy, rich beats and nothing else; yorko's warblings come across with no lyrical warmth, no harmony, no clever instrumentalism. i don't remember anything from the album except some piano on track 6. i think i found it irritating, which is often a sign that i'm grumpy about something, perhaps something i can't deal with.

So, i guess i'll give it a few more listens, then maybe give up, then do another 'disappointing albums' post in a couple of years. i hope i realise i was wrong, but part of me just wants to dislike the record. is that because i've got an unreasonable grudge, or is it because i feel dumped by it?

edit:
i think one of the things i didn't like about it is that all the songs are about 5 minutes long and go nowhere. it's like... if you do a two minute track that goes nowhere, well fine, it's just like a flavour, a taster, or something; extending it to five minutes is a trick, a smokescreen, like saying 'there's something in this track that deserves to be 5 minutes long' when there isn't. like blowing a doodle up to dramatic proportions in order to make it look like a work of art, which gives it some meaning it didn't originally have. now, i know music is different; a groove can take hold when jammed out. or it can become annoying. i mean, the first discernable riff on the album is track 3, and that's just such an arbitary melody, like they just found some notes and said 'fuck it, that'll do', without considering whether it was catchy or just sounded like a shit version of 'go to sleep'.

the black swan

I am angry with the world of publishing.


A few years ago, a saw a bunch of posters for 'the black swan' (the book, you dolt), and was basically suckered into buying it. it sounded like freakonomics, but more high-brow, and obscure, or something. It looked like exactly the book i wanted to read.

I didn't finish it because it's basically a made up pack of shit, with a very insightful basic idea, which i think i can summarise like this:

let's say there's a power law that relates how things that happen frequently don't change the world very much, and as things get more important, they get more rare. so far, so obvious. Taleb marries this to the long tail, and a pinch of chaos theory - the rarer, and therefore harder to predict, something is, the more it affects the world. right at the end of the tail, you get things that almost never happen, that totally change everything, that you could never predict.

i really like that idea, even if it is pretty pessimistic. but Taleb runs with, makes up a load of fictional evidence - i'm not kidding, he tells a story that he cites as evidence and then admits he made it up - and furthermore, the idea is fairly self-defining and tautologous. but what really irks me is beyond the book, and goes back to penguin's marketing for the book.

my excellent brother Dan recommended i read Iain M. Banks*' books "excession" as grist for the conceptual piece we're working on. i've just finished it, and it's a pretty complex book, but very good if you can keep up with it, or just surf along not understanding, knowing that what's going on will probably make sense when you get a few more chapters in. The plot deals with an "outside context problem", the kind of problem "most civilisations would encounter just once, and which they tended to encounter rather in the same way a sentence encountered a full stop."



i was reading the wikipedia page on excession, and guess what i read: "The situation has been more recently described as black swan theory." The link leads to a page on black swan theory, not the book itself, as if Taleb had any kind of formal definition of the theory, or had written an academic paper not a straight-to-paperback pop-science book.

edit: the link to the black swan theory page is also in the opening paragraph of the article. which is wierd.

this page is basically just an advert for the book (there is some discussion on the talk page as to why it exists and how it should be merged into the page for the book); and what are the chances of me coming across this link - unless it's not as uncommon as i expected. so i started thinking - have they just plastered the link all over wikipedia? what about the rest of the internet?

a quick site search brings up 113 pages that mention 'black swan theory' - many of which are talk pages or user pages and so on. but that's still a lot of pages directing wikipedians towards this page. a broader search brings up 259 pages across the internet that link to it. I'm unable to tell how many of these are people using the term in general parlance, and how many are what i suspect - deliberate insertions of the link across the interweb to boost sales of the book by viral marketeers. I say 'viral marketeers' because they're linking to the 'theory' not the 'book' - they're selling an idea, not a product. they're trying to turn wikipedia, which is freely editable, into a web of adverts. I hate that.


*nothing to do with 'rosie m. banks', the author bertie wooster pretends to be in jeeves and wooster.

On a different tack, the recent controversy over the Old Man Murray page on wikipedia really reminds me of arguments between Minds in Excession. there's something about the anarchy - as in, self-organisation - the independence, the distance between people. we depend upon them, and when individual ones become corrupted by the power, the group steps in to set things right. the user who suspended the OMM page has been reprimanded, and the comments here are just like dialogue between various characterful craft. the thrill of peeking behind the curtain is just the same in excession (where the human's plotline is completely pointless) as with wikipedia.

Ivor cutler's similar artists

I think this is pretty fascinating, and self explanatory.

cancer link dump

cancer keeps cropping up. i mean, in the news.

well, you can always draw a straight line between two points, but we've got carl zimmer on cancer in whales (or lack of) and the prevalence of a transmissible form of face cancer in tasmanian devils - surely the only case outside dogs, and possibly Henrietta Lacks.*

speaking of which, i might as well throw adam curtis' documentary about HeLa cells because it's brilliant, real *must watch* stuff.

I just think it's fascinating that we've got those two extremes out there at the moment - whales with their inexplicably low rate of cancer, and devils with a form of cancer so virulent that it's threatening their whole species. the killer quote: "Tasmanian devils are so genetically similar to one another that their immune systems don’t recognize infectious cancer cells from another individual as foreign"

i suppose it humbles me, how much we have yet to learn.

anyway, here's cancer: the musical, because mansun say it better than i ever could.

*edit: apparently, syrian hamsters too. surprised that the wikipedia article hasn't picked up on hela cells - while they were not transmissible to other humans, they seemed awfully virulent.