Let’s Get Real About Evidence Based Policy

One of Kevin Rudd’s favourite mantras is that he wishes to develop “evidence based” policy. That is, he backs policies which have enough supporting evidence to suggest that they will produce their desired effect. However, it is well known in scientific circles that there is one gold standard technique for discovering such a causal relationship between cause and effect– the randomised trial.

A randomised trial starts with a hypothesis – a statement of fact which is to be tested in the trial. Stating a hypothesis forces policy makers to move from a vague statements of intent to a specific measurable outcome they wish to achieve. The hypothesis is then tested by randomly selecting two groups of people. One group receives the treatment and one group does not. The power of the randomised trial lies in the fact that the two groups are as alike as possible in every respect except for whether or not they receive the treatment. If after the treatment has been applied, there is found to be a significant difference between the two groups, that difference can be attributed to the treatment and to the treatment alone. The researcher can conclude that the treatment caused the difference and therein lies the power of the randomised trial.

If policy makers want to be able to claim that their policies will work, they should subject them to randomised trials beforehand. Randomised trials present the policy maker who genuinely wants to make a difference with a powerful and irrefutable tool to put his theories to the test and to draw fact based conclusions from them.  Apart from trivial and self evident cases, the randomised trial is the one and only means of establishing a cause-and-effect relationship between one phenomenon and another and should be utilised by those who are serious about evidence based policy.

There is a growing recognition internationally that randomised trials should play an important part is assessing the efficacy of social policy.  One of the most outstanding examples of randomised trails in social reform is the Progressa (later known as Oportunidades) Program in Mexico.  The aim of the program was to “close the gap” between rich and poor in Mexico in terms of nutrition and education. Funding was made available to poor rural familles for education and improved nutrition. However the funding was conditional on attendance at both school and a government funded infant health clinic. The results have been encouraging and the program has been expanded into urban areas and extended to target youth up to the age of 22.  

Randomised trials have only been used only twice to assess social policy in Australia. Between 1999 and 2001 the Department of Family and Community Services conducted two randomised trials on the Job Network examining the effect of interviews and follow up contact from professional staff on workforce participation by the long-term unemployed. They found that the intervention led to a reduction in the number of hours worked but an increase in the number of hours spent in studying or training. In 2002 the effectiveness of the NSW drug court in reducing recidivism was tested in a randomised trial. Drug Court reduce recidivism, it was also more cost effective when measured by cost per offence averted.

Kevin Rudd speaks often about his preference for “evidence based policy”. The government’s responses to the 2020 summit are to be built on “a strong evidence base”. A raft of new policies is being introduced by the recently elected government and many of these policies are candidates for testing by randomised trials. If Rudd is serious about evidence based policy he should insist these initiatives are subjected to scrutiny by randomised trial.

In the financial year 2005-2006 the Federal government spent an around $90 billion on programs which purported to benefit Australians one way or another. None of these programs were tested by randomised trial.

There are a number of factors which make randomised trials unattractive to politicians. Making a real difference is hard and many randomised trails often show that the treatment made no difference. However, our elected representatives like to be seen as decisive, energetic and positive; especially when there is a crisis. They like to be seen as doing something about the problems and demonstrating leadership where others will not. Randomised trials on the other hand take time to implement, require an investment of time and money, show no immediate results and are based on the premise that no one actually knows what will work. Nonetheless, if the government wants to be taken seriously when it speaks of evidence policy, it cannot ignore the randomised trial.

Ross Farrelly’s article Policy On Trial appears in the latest edition of Policy magazine