Why you get “results” when you pay for results

Sometimes a story is so breathtaking in the wrongheaded use of metrics that it barely requires commentary. Take the example of the Troubled Families’ Program from the UK:

… as far as we can tell from extensive and voluminous analysis of tens of thousands of individual records … the Troubled Families Programme had no impact on the key outcomes it was supposed to improve at all. It didn’t make people more (or less) likely to come off benefits. To get jobs. To commit fewer crimes. And so on … the government’s deliberate misrepresentation of the data and statistics [led] to badly formulated targets, which in turn translated into a funding model that could have been designed to waste money. Bad stats meant bad policy.

In a classic example of incentives leading to bad outcomes, the program gave districts a “target” number of families (without knowing who they were or what help they needed) and then paid for them being “turned around”. As Stephen Crossley drily observed:

Manchester, for example have identified, worked with and turned around a staggering 2385 ‘troubled families’. Not one has ‘slipped through the net’ or refused to engage with the programme. Leeds and Liverpool have a perfect success rate in each ‘turning around’ over 2000 ‘troubled families. By my reckoning, over 50 other local authorities across the country have been similarly ‘perfect’ in their TF work. Not one single case amongst those 50 odd councils where more ‘troubled families’ were identified or where a ‘troubled family’ has failed to have been turned around. It is also staggering that work with some of the most disadvantaged families who have allegedly been immune to all previous policy interventions and whose ‘troubles’ have existed ‘for generations’ has been so successful at a time of wide-ranging and deep cuts to local authorities and others public service.

In case his sardonic message isn’t clear enough: it should not have been surprising that local councils, in the midst of historic budget cuts, presented with a bucket of money if they could “find” troubled people, would have such success in finding them. Financial incentives have unprecedented ability to strongly distort behaviour, often in undesirable ways: from the TAFE students who signed up to large student loans with promises of free tablets, to the Wells Fargo salespeople creating fake bank accounts.

However, any system with incentives to distort the truth (lie) to hit an easily manipulated metric will encounter similar ‘remarkable’ success. Managers and knowledge managers beware!

Header image source: Money by 401(K) 2012 is licensed by CC BY-SA 2.0.