Category Archives: Uncategorized

A short reading list on Payment By Results, from our Evaluation Team

Lucrezia Tincani (Evaluation Team for WASH Results) shares a compilation of guides and best practice reviews on Payment By Results (PBR).

PBR within the Water, Sanitation & Hygiene (WASH) sector

  1. This review for the Gates Foundation (2015) examines 30 PBR WASH programmes, predominantly implemented by the World Bank, all relatively small-scale. It outlines the minimum conditions that should be in place before choosing a PBR design. It suggests that PBR is better suited to private than to public providers, and works well in developing countries with low government capacity. However, it concludes that there is insufficient evidence to determine whether PBR WASH programmes are more efficient or more sustainable than non-PBR WASH programmes, also cautioning that there is limited evidence that PBR is effective at achieving behaviour change in community sanitation projects.
  2. Output-Based Aid is a form of PBR where aid is disbursed to governments who in turn contract a public service provider. These World Bank reports – (2010) and (2014) – outline the World Bank’s experiences of the Global Partnership on Output-Based Aid (GPOBA) with sanitation and water, respectively. The reports outline the preconditions necessary before choosing PBR approaches and outline case studies of the World Bank’s work.

PBR more broadly

  1. This paper by the Center for Global Development (2015) presents four major theories explaining the rationale for using PBR interventions and then explores which ones hold, through four case studies of Results-Based Aid in education and health, which were implemented alongside conventional aid programmes. It provides useful background on the different views regarding how and why PBR can act, and provides rich insights of how PBR has operated in the four case studies.
  2. The World Bank’s 330-page toolkit on performance-based financing (2014) contains a wealth of knowledge on design of PBR schemes, including selection of indicators and design of the verification process. Their GPOBA team also produced a useful overview note on approaches to independent verification of PBR programmes (2012). Though aimed at PBR schemes for health facilities, the lessons in the toolkit are generalisable across other sectors. The World Bank also produced a guide specifically for PBR in the energy sector (2013).
  3. This note (2014) by Clist & Dercon proposes 12 principles which should be considered when designing a PBR programme. It is relevant both for the implementers and evaluators of PBR programmes. It is a summary of their theoretical paper (2014) which outlines the likely costs and benefits of PBR programmes according to behavioural economics theory.
  4. This Cochrane Review of PBR programmes in health (2012) reviewed nine health PBR programmes and concluded that they were not uniform enough to be able to determine whether PBR programmes are more effective at improving health care than non-PBR ones. The review calls for more rigorous evaluations of PBR programmes, particularly to assess whether they are cost-effective. The German Development Institute (2013) also reviewed Results Based Financing experiences in the health sector.
  5. A DFID review (2013) synthesises 33 evaluations of PBR programmes: mostly health, with some education and one WASH programme (in Indonesia). It concludes that the added-value of PBR is, as of yet, unproven, highlighting that many evaluations have weak ‘external validity’, meaning that their conclusion are not generalisable to other contexts. This reinforces the premise that the success of PBR programmes is highly context specific. This report for NORAD (2008) came to a similar conclusion, as did this DFID review (2010) of Results Based Aid and Results Based Financing schemes, which also provides a useful overview of different types of PBR.
  6. A 2012 review by the European Network on Debt and Development (a network of 47 European NGOs) synthesises the experiences across six PBR multi-country initiatives (including GPOBA) and analyses how the amount of PBR financing has increased over time.
  7. MANGO (a UK NGO network) produced a report (2015) specifically highlighting the contractual risks of PBR contracts.
  8. This much-cited blog by Robert Chambers (2014) also highlights some of the risks of implementing PBR programmes.
  9. A SIDA review (2015) outlines the factors to be considered when designing a PBR programme, and outlines SIDA’s experience with different types of PBR so far (albeit limited). Similarly, this DFID ‘Smart Guide’ (2014) gives a brief overview of the factors to be considered when designing a PBR programme, and provides a list of all DFID PBR programmes operating at that time.
  10. DFID’s PBR evaluation framework (2015) provides useful guidance for those designing an evaluation of a PBR programme, including a detailed set of possible evaluation questions. These are also important for implementers when considering whether or not to implement a PBR programme.
  11. A review by the UK National Audit Office (2015) cautions that PBR schemes are “hard to get right, and are risky and costly for commissioners”. Thus, most UK PBR programmes only link 20% of payments to performance. The review covers work done by the whole of the British Government, not just by DFID.

The websites of the Suppliers implementing the WASH Results Programme (SWIFTSNV and SAWRP) also provide useful insight into the reality of implementing a PBR programme.

Given the absence of clear evidence about what works under different contexts, this reading list can never truly be complete or offer simple answers to questions about how best to implement PBR programmes. A key challenge is isolating the success or failure of the PBR modality itself from the success or failure of the technical intervention, which can be affected by external factors unrelated to PBR. Thus, there is still little evidence of the added benefits of PBR, because an accurate counterfactual cannot easily be established (rarely is a non-PBR programme with almost identical activities implemented alongside the PBR programme).

Our Evaluation Team would be glad to receive your suggestions of resources to add to this list, either through the comment section below or you can tweet them to us via @WASHResultsMVE

Lucrezia Tincani (OPM) leads the evaluation of the WASH Results Programme.

Latest views and news on Payment By Results

Our last post shared our lessons on Payment By Results (PBR); this time we’re taking a look at what other people are saying about it.

An over-looked benefit of PBR is sharpening the minds of donors and recipients, observes Joseph Holden, deputy lead of Monitoring and Evaluation (M&E) for the Fund Manager of DFID’s £350 million Girls’ Education Challenge (GEC) Programme. Writing at the end of 2015, Holden highlights PBR’s advantage over other funding models: reducing “fuzzy thinking” and over-statement of results by replacing self-reporting with independently verified, reliable and robust evidence. Writing about the approach and experience of the GEC, Holden comments “This is not cheap, and complexities have meant the application is far from perfect, but the evidence produced is of a higher quality than for the vast majority of other development programmes”. You can read more about the GEC’s experience of verifying PBR in the report on our chat show at the UK Evaluation Society

A new literature review [PDF]  intends to help PBR stakeholders: clarify the purpose of an individual PBR scheme, identify critical success factors, identify common issues which cause problems and difficulties, and be aware of how they might address or mitigate them. The review’s author, Russell Webster, is currently exploring the main themes of the review through a series of blog posts – the most recent ones question if PBR can improve outcomes and if PBR can save money.

Finally, even as this post is being written, delegates at the BOND conference are discussing ‘How to crack Results 2.0‘. We look forward to finding out what they had to say…

Cheryl Brown, Communications Manager for WASH Results MVE

What have we learned about Payment by Results (PBR) programmes from verifying one?

After 19 verification rounds, the WASH Results Monitoring and Verification team shares its suggestions for how to design future PBR programmes.

Martha Keega assesses a latrine in South Sudan

Verification in action: MV team member Martha Keega assesses a latrine in South Sudan

Verification is at the heart of the WASH Results Programme. Suppliers only get paid if we, the Monitoring and Verification (MV) team, can independently verify the results they are reporting. Usually we can: results are reported by Suppliers, verified by us and Suppliers are paid by DFID to an agreed schedule. However, all Suppliers have received deductions at least once, which, although painful for everyone, is testament to the rigour of the verification process. Overall, the system is working and the results of the programme are clear. But the demands of verification are also undeniable, leading to some aspects of verification being labelled “Payment by Paperwork” and like any process, it could be improved.

In January 2016 the team* came together to reflect on what we have learned so far from conducting 19 rounds of verification across the three Suppliers. Our discussions focused on verification but inevitably considered wider issues around design of a PBR programme. Here we share some suggestions for design of future PBR programmes, from a verification perspective.

  1. Ensure targets and milestones reflect high level programme objectives
  2. Be clear on targets and assumptions about their measurement
  3. Think carefully about enabling alignment with local government and other WASH stakeholders
  4. Reconsider the 100% PBR mechanism to avoid verification inefficiencies
  5. Consider payments for over-achievement of outcomes, but not of outputs
  6. Include provision for a joint Supplier and Verifier inception phase that will streamline verification
  7. Consider pros and cons of relying more on Supplier-generated evidence as opposed to independent evidence generation

1. Ensure targets and milestones reflect high level programme objectives
The WASH Results Programme has ambitions with regard to equity, gender and disability and overall health benefits that are not universally built into targets and payment milestones agreed between DFID and Suppliers. As a consequence, these ambitions are not explicitly incentivised. Any future programme should think carefully about how the design of the programme, especially the targets set in the tender and agreed with Suppliers, uphold objectives based on good practice within the sector.

2. Be clear on targets and assumptions about their measurement
We have found that when payment decisions are riding on whether targets have been met, the devil is in the detail. During implementation, some discrepancies have emerged over targets and how to achieve them. Discussions have taken place about minimum standards for latrines (DFID or JMP definition) and hygiene targets (what does ‘reach’ mean?). In addition, there was occasionally lack of clarity on how achievement of targets would be measured.

When working at scale, assumptions made about the average size of a household in a particular area, or the best way of measuring the number of pupils in a school become subject to intense scrutiny.  This is quite a departure from how programmes with different funding mechanisms have worked in the past and the level of detailed evidence required may come as a shock for Suppliers and Donors alike. In response, we suggest that future programmes should provide clear guidance on technical specifications relating to targets and guidelines for evidencing achievements.

3. Think carefully about enabling alignment with local government and other WASH Stakeholders
One concern that we discussed in the meeting was that the design of the WASH Results Programme does not sufficiently incentivise alignment with local government. We suspect that this was a result of the scale of the programme and the tight timelines, but also the demands of verification. The need to generate verifiable results can dis-incentivise both the pursuit of “soft” outcomes such as collaboration, and, working with government monitoring systems.

We suggest that PBR programmes need to think carefully about how to incentivise devolution of support services from progamme teams to local governments, and to other sector stakeholders during the life of the programme, for example by linking payments to these activities. Also, to think how programme design could encourage long-term strengthening of government monitoring systems.

4. Reconsider the 100% PBR mechanism to avoid verification inefficiencies
The merits or otherwise of the 100% PBR mechanism in the WASH Results Programme are subject to much discussion; we considered them from a verification perspective. We believe that, in response to the 100% PBR mechanism, some Suppliers included input- and process-related milestone targets to meet their internal cash flow requirements. In some cases, this led to verification processes that required high levels of effort (i.e. paperwork) with relatively few benefits.

We suggest that people designing future PBR programmes consider non-PBR upfront payments to Suppliers to avoid the need to set early input and process milestones, and run a substantial inception phase that includes paid-for outputs for Suppliers and Verifiers. In the implementation phase of the WASH Results Programme, payment milestones have been mainly quarterly, so requiring seemingly endless rounds of verification that put pressure on all involved, particularly Supplier programme staff. In response, we suggest that payments over the course of a programme should be less frequent (and so possibly larger), so requiring fewer verification rounds and allowing greater space between them. This may have implications for the design of the PBR mechanism.

5. Consider payments for over-achievement of outcomes, but not of outputs
The WASH Results Programme does not include payment for over-achievement. Over the course of the programme, some Suppliers have argued that over-achievement should be rewarded, just as under-achievement is penalised. As Verifiers, we agree that paying for over-achievement for outcomes would be a positive change in a future PBR design. However, there were concerns among our team that encouraging over-achievement of outputs could have unintended consequences such as inefficient investments or short-term efforts to achieve outputs without sufficient attention to sustainability and the quality of service delivery.

6. Include provision for a joint Supplier and Verifier inception phase that will streamline verification
It is broadly accepted that the WASH Results Programme would have benefited from a more substantial inception phase with the Verification Team in place at the start. Our recommendations about how an inception phase could help streamline and strengthen verification are as follows:

  • Key inception outputs should include a monitoring and results reporting framework agreed between the Supplier and the Verification Agent. Suppliers and Verifiers could be paid against these outputs to overcome cash flow issues.
  • The inception phase should include Verification Team visits to country programmes to establish an effective dialogue between the Verifiers and Suppliers early on.
  • If Suppliers evidence their achievements (as opposed to independent collection of evidence by the Verification Agent – see below), assessment of, and agreement on, what are adequate results reporting systems and processes need to be included in the inception phase.
  • Run a ‘dry’ verification round at the beginning of the verification phase where payments are guaranteed to Suppliers irrespective of target achievement so that early verification issues can be sorted out without escalating stress levels.

7. Consider pros and cons of relying more on Supplier-generated evidence as opposed to independent evidence generation
In the WASH Results Programme, Suppliers provide evidence against target achievements, which is subsequently verified by the Verification Team (we will be producing a paper soon that outlines how this process works in more detail). Is this reliance on Supplier-generated evidence the best way forward? What are the pros and cons of this approach as compared with independent (verification-led) evidence generation?

Indications are that the PBR mechanism has improved Suppliers’ internal monitoring systems, and has shifted the internal programming focus from the finance to the monitoring and evaluation department. However, relying on Suppliers’ internal reporting systems has required some Suppliers to introduce substantial changes to existing reporting systems and the MV team has faced challenges in ensuring standards of evidence, particularly in relation to surveys.

We have some ideas about pros and cons of Supplier-generated evidence as opposed to evidence generated independently, but feel this can only be fully assessed in conversation with the Suppliers. We plan to have this conversation at a WASH Results Programme Supplier learning event in March. So, this is not so much a suggestion as a request to watch this space!

Coming up…

WASH Results Programme Learning Event:  On March 7 2016 Suppliers, the e-Pact Monitoring & Verification and Evaluation teams, and DFID will convene to compare and reflect on learning so far. Key discussions at the event will be shared through this blog.

Verification framework paper: an overview of how the verification process works in the WASH Results Programme. This will present a behind-the-scenes look at verification in practice and provide background for future lessons and reflections that we intend to share through our blog and other outputs.

 

 


* About the MV Team: In the WASH Results Programme, the monitoring, verification and evaluation functions are combined into one contract with e-Pact. In practice, the ongoing monitoring and verification of Suppliers’ results is conducted by one team (the MV team) and the evaluation of the programme by another.  The lessons here are based on the experience of the MV team although members of the Evaluation team were also present at the workshop. Read more about the WASH Results Programme.

Comparative experiences of evaluating and verifying Payment by Results programmes

Highlights of the discussion about verification and evaluation of Payment By Results in our chat show at the UK Evaluation Society Conference.

A session at the UK Evaluation Society conference in May 2015 compared two Payment by Results (PbR) programmes in different sectors. The discussion revealed that although PbR programmes can be set up in very different ways, the management, evaluation and verification face similar tensions. These include balancing cash flows with the focus on “real outcomes” that PbR is designed to encourage.

Using an informal chat show approach, the session explored practical experiences of monitoring and verification of two DFID funded PbR programmes: the Water, Sanitation and Hygiene (WASH) Results Programme and the Girls Education Challenge (GEC). The chat show was attended by one of the WASH Results Programme service providers thus providing a different perspective on the issues.

Chat show participants were:
Dr Katharina Welle, Deputy Team Leader, WASH Results Monitoring, Verification & Evaluation (MVE) Team, (Itad)
Dr Lucrezia Tincani, Manager, WASH Results MVE Team, (Oxford Policy Management)
Jason Calvert, Global Monitoring & Evaluation (M&E) Lead, Girls’ Education Challenge
The host was me, Catherine Fisher, Learning Adviser, WASH MVE Team, (Independent Consultant).

The discussion focused on the practicalities of implementing programmes financed through a Payment by Results funding mechanism (specifically Results Based Financing, a term used by DFID, among others, when the payments from funders or government go to service providers or suppliers). This blog post shares a few of the areas that were discussed.

There are huge variations in the set-up of PbR programmes
Perhaps unsurprisingly for two programmes of different sizes in different sectors, the WASH Results Programme (£68.7 million over approximately four years) and GEC (£330 million over four years) are managed, monitored and evaluated in very different ways.

GEC is managed by Price Waterhouse Coopers (PwC) who manage the funding to the 37 GEC projects, set suppliers’ targets and create M&E frameworks, while suppliers themselves contract external evaluators to monitor and evaluate their progress. By contrast the three WASH Results Programme supplier contracts are managed in-house by DFID and the Monitoring, Verification and Evaluation is contracted out to a consortium: e-Pact. There are no doubt pros and cons of each approach which the chat show did not explore. But a key advantage of the GEC set-up (from the perspective of the e-Pact WASH MVE team) was the involvement of PwC from the start of the GEC programme. This meant PwC could shape the “rules of the game” for verification and were able to set standardised targets up front.

Differences in amount of funding subject to PbR
Another key difference between the two programmes was the amount of the supplier funding that was subject to PbR. For GEC suppliers it is on average 10%, whereas WASH Results Programme suppliers see 100% of their funding dependent on verified results. But this startling figure may mask some similarities….

GEC differentiates between payment against inputs (not called PbR) and payment against outcomes, the “real” PbR which constitutes 10% of the total funding. By contrast, the WASH programme has a broader definition of results that includes outputs, inputs and outcomes, and varies across suppliers. So in practice the programmes may be more similar than they appear.

Balancing cash flow and adaptive programming in a PbR context
An important driver for the broader interpretation of “results” within the WASH Results programme was the very real need for some suppliers to maintain a steady cash flow across the course of the programme in a 100% PbR context.   The temptation for suppliers can be to set lots of milestones that trigger payment. However the need to stick to and demonstrate achievement of these milestones may inhibit flexibility in delivery. This tension between maintaining cash flow and the adaptive programming that PbR is intended to foster has been experienced by both programmes.

Rigour in measurement
The ability to effectively measure results is at the heart of PbR. For PwC, this means that every project subject to PbR is monitored through use of a quasi-experimental Randomised Controlled Trial (RCT) that measures outcomes in the project site with a control group. One reason PwC insist on an RCT for each project is to protect themselves against the risk of being sued for incorrectly withholding payment.

Where an RCT is not possible, for example in Afghanistan where security risks for implementers and cultural considerations mean that control groups are not feasible, these projects are removed from PbR. A number of the 37 GEC projects have been taken off PbR due to cultural considerations and challenging environments.

The ability to measure results is also dependent on the existence of consensus and evidence about expected results and effective means of measurement. This is more the case in the education sector than the WASH sector and makes setting targets and assessing progress towards them more difficult for a WASH programme, particularly in hygiene promotion activities such as those promoting hand washing.

As a result of these discussions, participants suggested the use of a spectrum (see Figure 1) that matches the proportion of programme funding dependent on PbR to how to how easy it is to measure results in that particular sector.

A potential PbR - measurement spectrum

Does PbR generate perverse incentives?
One of the much talked about cons of PbR is that it will create perverse incentives among stakeholders that drive them to behave in the opposite way than intended. Participants shared stories that included examples of the PbR mechanism inhibiting innovation and encouraging suppliers to focus on “low hanging fruit” rather than greatest need. A review of PbR in Swiss cantons suggested it didn’t work at all in terms of generating efficiencies and effectiveness.

PbR is not a one-size-fits-all solution; there remains lots to learn about when and where it can work
There was consensus that PbR cannot be used effectively in all contexts. The risks and uncertainties of working in fragile states are one setting in which PbR can be difficult (but not impossible?) to implement, however even in more stable contexts, issues around organisational capacity and motivations can inhibit its effective implementation.

Participants agreed that there are probably sets of circumstances in which PbR can be effective. People working in countries and sectors in which PbR has been used for many years have spent time identifying what those contexts are – see for example this paper about PbR in UK community work. For those of us who are new to the challenges of commissioning, designing, implementing, monitoring, evaluating or verifying PbR projects and are doing it in contexts in which it has not been tried, there is a lot to learn.  This chat show illustrates that there is great value in coming together to do so.

What do you think?
What are you learning about managing, implementing or verifying Payment By Results programmes? Does this discussion reflect your experience? Would you like to learn with us?  Please use the comment box to share your thoughts.

Catherine Fisher, Learning Adviser, WASH Results MVE Team.

See also: Payment-by-results: the elixir or the poison?, 6 January 2015, Jason Calvert