A short reading list on Payment By Results, from our Evaluation Team

Lucrezia Tincani (Evaluation Team for WASH Results) shares a compilation of guides and best practice reviews on Payment By Results (PBR).

PBR within the Water, Sanitation & Hygiene (WASH) sector

  1. This review for the Gates Foundation (2015) examines 30 PBR WASH programmes, predominantly implemented by the World Bank, all relatively small-scale. It outlines the minimum conditions that should be in place before choosing a PBR design. It suggests that PBR is better suited to private than to public providers, and works well in developing countries with low government capacity. However, it concludes that there is insufficient evidence to determine whether PBR WASH programmes are more efficient or more sustainable than non-PBR WASH programmes, also cautioning that there is limited evidence that PBR is effective at achieving behaviour change in community sanitation projects.
  2. Output-Based Aid is a form of PBR where aid is disbursed to governments who in turn contract a public service provider. These World Bank reports – (2010) and (2014) – outline the World Bank’s experiences of the Global Partnership on Output-Based Aid (GPOBA) with sanitation and water, respectively. The reports outline the preconditions necessary before choosing PBR approaches and outline case studies of the World Bank’s work.

PBR more broadly

  1. This paper by the Center for Global Development (2015) presents four major theories explaining the rationale for using PBR interventions and then explores which ones hold, through four case studies of Results-Based Aid in education and health, which were implemented alongside conventional aid programmes. It provides useful background on the different views regarding how and why PBR can act, and provides rich insights of how PBR has operated in the four case studies.
  2. The World Bank’s 330-page toolkit on performance-based financing (2014) contains a wealth of knowledge on design of PBR schemes, including selection of indicators and design of the verification process. Their GPOBA team also produced a useful overview note on approaches to independent verification of PBR programmes (2012). Though aimed at PBR schemes for health facilities, the lessons in the toolkit are generalisable across other sectors. The World Bank also produced a guide specifically for PBR in the energy sector (2013).
  3. This note (2014) by Clist & Dercon proposes 12 principles which should be considered when designing a PBR programme. It is relevant both for the implementers and evaluators of PBR programmes. It is a summary of their theoretical paper (2014) which outlines the likely costs and benefits of PBR programmes according to behavioural economics theory.
  4. This Cochrane Review of PBR programmes in health (2012) reviewed nine health PBR programmes and concluded that they were not uniform enough to be able to determine whether PBR programmes are more effective at improving health care than non-PBR ones. The review calls for more rigorous evaluations of PBR programmes, particularly to assess whether they are cost-effective. The German Development Institute (2013) also reviewed Results Based Financing experiences in the health sector.
  5. A DFID review (2013) synthesises 33 evaluations of PBR programmes: mostly health, with some education and one WASH programme (in Indonesia). It concludes that the added-value of PBR is, as of yet, unproven, highlighting that many evaluations have weak ‘external validity’, meaning that their conclusion are not generalisable to other contexts. This reinforces the premise that the success of PBR programmes is highly context specific. This report for NORAD (2008) came to a similar conclusion, as did this DFID review (2010) of Results Based Aid and Results Based Financing schemes, which also provides a useful overview of different types of PBR.
  6. A 2012 review by the European Network on Debt and Development (a network of 47 European NGOs) synthesises the experiences across six PBR multi-country initiatives (including GPOBA) and analyses how the amount of PBR financing has increased over time.
  7. MANGO (a UK NGO network) produced a report (2015) specifically highlighting the contractual risks of PBR contracts.
  8. This much-cited blog by Robert Chambers (2014) also highlights some of the risks of implementing PBR programmes.
  9. A SIDA review (2015) outlines the factors to be considered when designing a PBR programme, and outlines SIDA’s experience with different types of PBR so far (albeit limited). Similarly, this DFID ‘Smart Guide’ (2014) gives a brief overview of the factors to be considered when designing a PBR programme, and provides a list of all DFID PBR programmes operating at that time.
  10. DFID’s PBR evaluation framework (2015) provides useful guidance for those designing an evaluation of a PBR programme, including a detailed set of possible evaluation questions. These are also important for implementers when considering whether or not to implement a PBR programme.
  11. A review by the UK National Audit Office (2015) cautions that PBR schemes are “hard to get right, and are risky and costly for commissioners”. Thus, most UK PBR programmes only link 20% of payments to performance. The review covers work done by the whole of the British Government, not just by DFID.

The websites of the Suppliers implementing the WASH Results Programme (SWIFTSNV and SAWRP) also provide useful insight into the reality of implementing a PBR programme.

Given the absence of clear evidence about what works under different contexts, this reading list can never truly be complete or offer simple answers to questions about how best to implement PBR programmes. A key challenge is isolating the success or failure of the PBR modality itself from the success or failure of the technical intervention, which can be affected by external factors unrelated to PBR. Thus, there is still little evidence of the added benefits of PBR, because an accurate counterfactual cannot easily be established (rarely is a non-PBR programme with almost identical activities implemented alongside the PBR programme).

Our Evaluation Team would be glad to receive your suggestions of resources to add to this list, either through the comment section below or you can tweet them to us via @WASHResultsMVE

Lucrezia Tincani (OPM) leads the evaluation of the WASH Results Programme.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s