Report finds only 6% of Indigenous evaluations use robust methodology

Indigenous program evaluations are marred by a lack of rigour, with only three of 49 evaluations analysed using robust methodology, according to a new research report, Evaluating Indigenous programs: a toolkit for change, published by the Centre for Independent Studies (CIS).

“Previous research found only 8% of 1082 Indigenous-specific programs had been evaluated, but our latest research found, of those evaluated, only 6% were of high quality,” says report author Sara Hudson, manager of the CIS Indigenous Research Program.

“Not only is there an absence of evaluations, but only a minority have been conducted properly. None of the evaluations analysed used what is considered the ‘gold standard’ of evidence: random controlled trials. Overall, the evaluations were characterised by a lack of data and an over-reliance on anecdotal evidence
“What’s more, even when programs have been evaluated, government agencies have ignored them when making funding decisions or implementing new programs,” Ms Hudson says.
“A recent audit of the New South Wales Evaluation strategy found the state’s Treasury and Department of Premier and Cabinet were not using evaluation outcomes to make funding decisions, while other reports showed many organisations continued to receive funding to deliver programs even after evaluations had identified ‘serious deficiencies’ with them.
“In the 2017-18 Budget, the federal government announced it will allocate $40 million over four years to strengthen the evaluation of Indigenous programs, but given that the average cost of an evaluation is $382,000, the extra $10 million a year will not go far. In fact, only 26 of the 1000 or so Indigenous programs funded by the federal government will be able to be formally evaluated.
“Surely the government’s intention behind this increase in funding for evaluation of Indigenous programs is not to feather the nests of evaluators.
“If this is the case, the government must change the way it evaluates and monitors programs. Instead of getting contractors in and only evaluating a few programs it would be better to encourage self-evaluation and for evaluation to be embedded into a program’s design as part of a continuous quality improvement process,” Ms Hudson says.
The report recommends adopting a co-accountability approach to evaluation to ensure that both the government agency funding the program, and the program provider delivering the program, are held accountable for results.
An overarching evaluation framework would assist with the different levels of outcomes expected over the life of the program and the various indicators needed to measure whether the program is meeting its objectives. Feedback loops and a process to escalate any concerns would also help to ensure government and program providers keep each other honest and program learnings are shared.
Sara Hudson is a Research Fellow and Manager of the Indigenous Research Program at the Centre for Independent Studies