Evaluating Policy Impact: Working Out What Works

• Published date: 01 December 2023
• Author: Andrew Leigh
• Date: 01 December 2023
• DOI: http://doi.org/10.1111/1467-8462.12530

The Australian Economic Review, vol. 56, no. 4, pp. 431–441DOI: 10.1111/1467-8462.12530

Evaluating Policy Impact: Working Out What Works

Andrew Leigh*

Abstract

Randomised trials frequently produce surprising

findings, overturning conventional wisdom.

During the twentieth century, randomised trials

became commonplace within medicine, saving

millions of lives. Randomised trials within

government can now be conducted more

cheaply, using administrative data. Just as it

might be considered unethical to conduct a

randomised trial if a program is indisputably

effective, it might be considered unethical not to

conduct a rigorous evaluation if a program

lacks evidence. Developed within a robust

ethical framework, and alongside community

consultation, better evaluation can help

governments save money and address social

disadvantage.

1. Introduction

Social workers in schools always boost

student outcomes. Drug offenders should not

be treated differently. Malaria bed nets are

more likely to be used if people pay for them.

Seeing inside a jail will deter juvenile

delinquents from becoming criminals.

All four statements sound perfectly sen-

sible, don't they? Unfortunately, randomised

trials suggest that all four are perfectly wrong.

Let me explain.

In Britain, teachers, social workers and

students all liked a pilot program placing

social workers in schools. Then researchers at

Cardiff and Oxford Universities ran a two‐

year randomised trial across 300 schools to

test the program's impact. The results, re-

ported this year, show no significant positive

impact (Westlake et al. 2023). As a result,

the planned national rollout has now been

scrapped (Molloy 2023).

In New South Wales, a randomised trial of

a specialised drug court shows that the

tailored approach to drug offenders reduces

recidivism (Lind et al. 2002). By treating their

addiction, drug offenders’became much less

likely to reoffend than if they had been

sentenced through the traditional criminal

justice system. Drug courts do not just help

addicts—they also make the streets safer.

In Africa, some economists argued that free

anti‐malarial bed nets would not be valued by

villagers, and might be used instead as

makeshift fishing nets. So a randomised trial

tested the take‐up and use of free versus cheap

bed nets. It turned out that free bed nets were

far more popular, and equally likely to be used

*Parliament of Australia, Canberra, Australia. email:

<andrew.leigh.mp@aph.gov.au>

This is an edited version of an address to the National

Press Club in Canberra on 29 August 2023. Thanks to a

range of experts, including Elisabeth Costa, David

Halpern, Jon Lavis and officials in the Australian Centre

for Evaluation for valuable feedback on earlier drafts, and

to Ross Williams for his deft edits.

© 2023 The University of Melbourne, Melbourne Institute: Applied Economic & Social Research, Faculty of Business

and Economics.

Published by John Wiley & Sons Australia, Ltd