By Howard White, CEO, The Campbell Collaboration
Suppose you, or you child, were recommended to undergo an operation. Would your response be to say, “sure, go head and stick the knife in and see what happens.” Unlikely. More likely you would seek reassurance that the operation has been done before, what the success rate is, and if there have been adverse side effects. If the operation were for your child, you would ask specifically if there was any data showing the operation to be safe for children.
Why then is it, that new programmes get started, or existing ones reformed, without looking to see what evidence, elsewhere, has to say about their likely effectiveness? Why are we not using the wealth of knowledge available from global experience?
This is starting to happen in places, but it still is not enough. Promoting systematic use of existing global evidence is the third wave of the evidence revolution.
The first wave was the ‘results agenda’ under new public momentum, which started in the 1980s and picked up momentum in the nineties as it was endorsed by both Clinton and Blair governments. The results agenda resulted in comprehensive tracking of indicators across the causal chain, with a particular focus on outcomes. This shift from measuring success by focusing on outputs – how much money had been spent – to outcomes – such as children lifted out of poverty – was welcome. But too many agencies mistook the factual account of what had happened to outcomes as telling them what impact their policies and programmes were having. But knowing a programme’s impact requires the counterfactual analysis provided by a rigorous impact evaluation.
The rise of rigorous impact evaluations, notably randomized evaluations of social programmes, has been the second wave of the evidence revolution. More and more countries around the world – from Mexico to Mozambique - are subjecting their programmes to this scrutiny. The challenge is to utilize this growing body of evidence not just to assess the programmes being evaluated but to help shape policy elsewhere.
This is the third wave of the evidence revolution: using global evidence to inform national policy. It is most heavily institutionalized in health, promoted by Cochrane. WHO policy on guidelines state that they are to be based on high-quality systematic reviews. In the UK, the National Institute for Clinical Excellence (NICE) produces guidelines and determines which treatments are eligible for public support through the National Health Service. This approach is spreading to other sectors in the UK through the What Works Centres. The Nordic countries have government-funded research centres whose job it is to produce reviews of global evidence to inform government policy.
So, doesn’t it sound like the third wave has landed? Partly, but not without some problems.
The various national efforts to learn from global evidence are not done in a coordinated way. This can lead to several agencies commissioning separate reviews of the same global evidence. At best this is unnecessary duplication. But often it leads to low quality reviews being undertaken, which are at a higher risk of bias, that is giving misleading findings. It would make more sense for these agencies to join forces, producing one global review, which they then help their stakeholders use in that specific context.
And the revolution is far from global. Even in the countries where it has gone farthest penetration remains uneven across professions. And in many countries – notably in the developing world – new policies and programmes are designed with little or no regard for existing evidence.
The reasons for this lay partly in the nature of the product. Evidence synthesis reports are technical documents, neither discoverable by or accessible to busy policy makers and practitioners. The range of evidence they provide may sometimes be of limited value. And, finally, there remain large gaps in evidence synthesis.
There is a solution to all these shortcomings, and the Campbell Collaboration offers it. The Campbell Library is a global repository of knowledge in high-quality systematic reviews. To ensure quality, to avoid duplication, and to promote broader use, all commissioners and producers of reviews are encouraged to register their reviews with Campbell. But Campbell is more proactive still in tackling these issues. The global funds for child welfare and children at risk tackle the problem of coordination failure, offering stakeholders a mechanism to have a say in the commissioning, say, six high quality reviews rather than just commissioning one of their own. And Campbell supports the production of policy-friendly derivative products: drawing out the key messages from a review and presenting them in a more accessible manner.