Randomised Policy Trials

Good paper from Australian economist Andrew Leigh on the merits of ‘Randomised Policy Trials’.

What’s the idea?

Essentially the idea is to use the more scientific approach to experimentation in the design of policy. So, rather than just rolling out an idea, or doing simple before-after analysis, policy makers would subject an idea to the scrutiny of controlled tests. For example, a new approach to school exams might be trialled in several schools in a way that attempted to hold constant for other factors that influence performance before extending on a national scale (of course only if it proved effective).

If it’s so simple why don’t we do it?

Andrew debunks a number of arguments made against randomised policy trials in social sciences. My take on the most compelling – but ultimately wrong – arguments against randomised trials:

• Because they might involve denying treatment to worthy individuals. Some would, for example, argue that it is unethical to deny all the opportunity to the new school exams if they were thought to be superior to what everyone else got. But try the counterfactual: surely it’s more unethical to try the new school exams on everyone, rather than a small group first.
  


• Because there are other alternatives. Yes there are other ways to test the effectiveness of policies. But all too often we get stuck in a simple before-after analysis. So the government claims success on the new school exams because results turn out to be higher than they were ten years ago. But, of course, it may not be the new exams that actually caused the increase in results, instead being a consequence of other things like school investment, better teaching or inflation in marking. Randomised policy trials are the most scientific way to isolate the causal effect of policy.
  


• Because qualitative results are better. A simple rejoinder to this is that doing quantitative trials doesn’t preclude doing the qualitative analysis too – surely we’ll get a better result from doing both together?
  


• Because politicians are interested in re-election, not results. I think this is perhaps a slightly harsh phrasing that Leigh uses but probably gets to the heart of the problem. There are strong incentives for politicians to roll out and start doing. After all, elections come around pretty quickly and no-one is interested in a small scale pilot if the rest of the system is failing. Yet clearly over the long-term this leads to sub-optimal policy paths being taken.

Anyhow it turns out that it is in the US where most randomised policy trials occur, with Canada and the UK a little behind. Would be interesting to see this as a proportion of all new policies though – I suspect it’d be pretty low…