What have we learned about Payment by Results (PBR) programmes from verifying one?

After 19 verification rounds, the WASH Results Monitoring and Verification team shares its suggestions for how to design future PBR programmes.

Martha Keega assesses a latrine in South Sudan

Verification in action: MV team member Martha Keega assesses a latrine in South Sudan

Verification is at the heart of the WASH Results Programme. Suppliers only get paid if we, the Monitoring and Verification (MV) team, can independently verify the results they are reporting. Usually we can: results are reported by Suppliers, verified by us and Suppliers are paid by DFID to an agreed schedule. However, all Suppliers have received deductions at least once, which, although painful for everyone, is testament to the rigour of the verification process. Overall, the system is working and the results of the programme are clear. But the demands of verification are also undeniable, leading to some aspects of verification being labelled “Payment by Paperwork” and like any process, it could be improved.

In January 2016 the team* came together to reflect on what we have learned so far from conducting 19 rounds of verification across the three Suppliers. Our discussions focused on verification but inevitably considered wider issues around design of a PBR programme. Here we share some suggestions for design of future PBR programmes, from a verification perspective.

  1. Ensure targets and milestones reflect high level programme objectives
  2. Be clear on targets and assumptions about their measurement
  3. Think carefully about enabling alignment with local government and other WASH stakeholders
  4. Reconsider the 100% PBR mechanism to avoid verification inefficiencies
  5. Consider payments for over-achievement of outcomes, but not of outputs
  6. Include provision for a joint Supplier and Verifier inception phase that will streamline verification
  7. Consider pros and cons of relying more on Supplier-generated evidence as opposed to independent evidence generation

1. Ensure targets and milestones reflect high level programme objectives
The WASH Results Programme has ambitions with regard to equity, gender and disability and overall health benefits that are not universally built into targets and payment milestones agreed between DFID and Suppliers. As a consequence, these ambitions are not explicitly incentivised. Any future programme should think carefully about how the design of the programme, especially the targets set in the tender and agreed with Suppliers, uphold objectives based on good practice within the sector.

2. Be clear on targets and assumptions about their measurement
We have found that when payment decisions are riding on whether targets have been met, the devil is in the detail. During implementation, some discrepancies have emerged over targets and how to achieve them. Discussions have taken place about minimum standards for latrines (DFID or JMP definition) and hygiene targets (what does ‘reach’ mean?). In addition, there was occasionally lack of clarity on how achievement of targets would be measured.

When working at scale, assumptions made about the average size of a household in a particular area, or the best way of measuring the number of pupils in a school become subject to intense scrutiny.  This is quite a departure from how programmes with different funding mechanisms have worked in the past and the level of detailed evidence required may come as a shock for Suppliers and Donors alike. In response, we suggest that future programmes should provide clear guidance on technical specifications relating to targets and guidelines for evidencing achievements.

3. Think carefully about enabling alignment with local government and other WASH Stakeholders
One concern that we discussed in the meeting was that the design of the WASH Results Programme does not sufficiently incentivise alignment with local government. We suspect that this was a result of the scale of the programme and the tight timelines, but also the demands of verification. The need to generate verifiable results can dis-incentivise both the pursuit of “soft” outcomes such as collaboration, and, working with government monitoring systems.

We suggest that PBR programmes need to think carefully about how to incentivise devolution of support services from progamme teams to local governments, and to other sector stakeholders during the life of the programme, for example by linking payments to these activities. Also, to think how programme design could encourage long-term strengthening of government monitoring systems.

4. Reconsider the 100% PBR mechanism to avoid verification inefficiencies
The merits or otherwise of the 100% PBR mechanism in the WASH Results Programme are subject to much discussion; we considered them from a verification perspective. We believe that, in response to the 100% PBR mechanism, some Suppliers included input- and process-related milestone targets to meet their internal cash flow requirements. In some cases, this led to verification processes that required high levels of effort (i.e. paperwork) with relatively few benefits.

We suggest that people designing future PBR programmes consider non-PBR upfront payments to Suppliers to avoid the need to set early input and process milestones, and run a substantial inception phase that includes paid-for outputs for Suppliers and Verifiers. In the implementation phase of the WASH Results Programme, payment milestones have been mainly quarterly, so requiring seemingly endless rounds of verification that put pressure on all involved, particularly Supplier programme staff. In response, we suggest that payments over the course of a programme should be less frequent (and so possibly larger), so requiring fewer verification rounds and allowing greater space between them. This may have implications for the design of the PBR mechanism.

5. Consider payments for over-achievement of outcomes, but not of outputs
The WASH Results Programme does not include payment for over-achievement. Over the course of the programme, some Suppliers have argued that over-achievement should be rewarded, just as under-achievement is penalised. As Verifiers, we agree that paying for over-achievement for outcomes would be a positive change in a future PBR design. However, there were concerns among our team that encouraging over-achievement of outputs could have unintended consequences such as inefficient investments or short-term efforts to achieve outputs without sufficient attention to sustainability and the quality of service delivery.

6. Include provision for a joint Supplier and Verifier inception phase that will streamline verification
It is broadly accepted that the WASH Results Programme would have benefited from a more substantial inception phase with the Verification Team in place at the start. Our recommendations about how an inception phase could help streamline and strengthen verification are as follows:

  • Key inception outputs should include a monitoring and results reporting framework agreed between the Supplier and the Verification Agent. Suppliers and Verifiers could be paid against these outputs to overcome cash flow issues.
  • The inception phase should include Verification Team visits to country programmes to establish an effective dialogue between the Verifiers and Suppliers early on.
  • If Suppliers evidence their achievements (as opposed to independent collection of evidence by the Verification Agent – see below), assessment of, and agreement on, what are adequate results reporting systems and processes need to be included in the inception phase.
  • Run a ‘dry’ verification round at the beginning of the verification phase where payments are guaranteed to Suppliers irrespective of target achievement so that early verification issues can be sorted out without escalating stress levels.

7. Consider pros and cons of relying more on Supplier-generated evidence as opposed to independent evidence generation
In the WASH Results Programme, Suppliers provide evidence against target achievements, which is subsequently verified by the Verification Team (we will be producing a paper soon that outlines how this process works in more detail). Is this reliance on Supplier-generated evidence the best way forward? What are the pros and cons of this approach as compared with independent (verification-led) evidence generation?

Indications are that the PBR mechanism has improved Suppliers’ internal monitoring systems, and has shifted the internal programming focus from the finance to the monitoring and evaluation department. However, relying on Suppliers’ internal reporting systems has required some Suppliers to introduce substantial changes to existing reporting systems and the MV team has faced challenges in ensuring standards of evidence, particularly in relation to surveys.

We have some ideas about pros and cons of Supplier-generated evidence as opposed to evidence generated independently, but feel this can only be fully assessed in conversation with the Suppliers. We plan to have this conversation at a WASH Results Programme Supplier learning event in March. So, this is not so much a suggestion as a request to watch this space!

Coming up…

WASH Results Programme Learning Event:  On March 7 2016 Suppliers, the e-Pact Monitoring & Verification and Evaluation teams, and DFID will convene to compare and reflect on learning so far. Key discussions at the event will be shared through this blog.

Verification framework paper: an overview of how the verification process works in the WASH Results Programme. This will present a behind-the-scenes look at verification in practice and provide background for future lessons and reflections that we intend to share through our blog and other outputs.

 

 


* About the MV Team: In the WASH Results Programme, the monitoring, verification and evaluation functions are combined into one contract with e-Pact. In practice, the ongoing monitoring and verification of Suppliers’ results is conducted by one team (the MV team) and the evaluation of the programme by another.  The lessons here are based on the experience of the MV team although members of the Evaluation team were also present at the workshop. Read more about the WASH Results Programme.

As the MDGs become SDGs, what progress has WASH Results made?

What has the DFID WASH Results Programme achieved so far and what lies ahead for the programme in 2016?

January 1st, 2016 is an important date for the DFID WASH Results Programme. The Sustainable Development Goals (SDGs) take the place of the Millennium Development Goals (MDGs) and ‘WASH Results’ moves into the second half of its funding period. At this stage the focus for the programme’s Suppliers (SAWRP, SNV and SWIFT) will start shifting from getting things done, to keeping things going and ensuring the sustainability of WASH outcomes. For e-Pact too, as the independent verifiers, it’s farewell to Outputs and hello to Outcomes.

Once the goals to provide access have been reached, attention turns to sustainability.

Before we look ahead to next year, let’s take a look at what’s been achieved so far. According to the UN, 2.6 billion people gained access to improved drinking water sources between 1990 and 2015 – a key part of MDG 7. Worldwide, 2.1 billion people have gained access to improved sanitation but 2.4 billion are still using unimproved sanitation facilities, including nearly 1 billion people who are still defecating in the open. This July, DFID was able to report it had exceeded its own target of supporting 60 million people to access clean water, better sanitation or improved hygiene conditions.

What contribution can we attribute to the WASH Results Programme? The full report on the programme’s 2015 Annual Review is available on the UKAid Development Tracker website. Here are some of the highlights:

  • The reviewers gave the programme an ‘A’ overall (for the second year running) and considered it to be “on track” to meeting its targets by the end of 2015.
  • By December 2014 the WASH Results Programme had reached 296,438 people with improved sanitation, 65,611 people with improved water supply, and over 1.25 million people with hygiene promotion.
  • The reviewers noted that “strong independent verification systems” have been established that also allow for adjustment and improvement based on learning from previous verification rounds.
  • WASH Results is generating significant policy knowledge around use of Payment by Results and programming for outcomes (sustainability) in the WASH sector.

We’ll come back to these results in early 2016 when the final numbers are in and compare them to the programme’s targets for December 2015 which are:

  • 968,505 people have access to clean drinking water;
  • 3,769,708 people have access to an improved sanitation facility;
  • 9,330,144 people reached through hygiene promotion activities through DFID support.

Large numbers, however impressive, don’t fully convey the effects that improvements in water, sanitation and hygiene are having on people’s lives. The Suppliers have been collecting stories of change from some of the people on the ground who are closely involved in delivering the WASH Results Programme or directly benefitting from its work. You can read about 70 year old subsistence farmer Maria and mother-of-five, Jacinta, in a recent DFID blog post. The SWIFT consortium’s website is also packed with news and images from their involvement in the WASH Results Programme.

So what happens next?

From 2016 onwards, e-Pact starts answering a critical question for DFID: how many poor people continue to use improved water and sanitation facilities and are practising improved hygiene because of the WASH Results Programme? Right now we’re exploring how best to monitor, report on and verify these outcomes and look forward to sharing what we learn, with you.

As always, if you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us.

Suggestions to donors commissioning Payment by Results programmes

Reflections by DFID WASH Results’ suppliers on the programme’s design and commissioning, summarised into six suggestions for commissioners.

Payment by Results (PbR) is a technically challenging form of contracting and one that is new for DFID and many of its partners (NAO 2015, p8, DFID, 2015 p18).   The WASH Results Programme was designed using the PbR financing mechanism, and therefore has great potential for learning about the impact of PbR on programming.

One year into the programme, the WASH Results suppliers met to share experiences so far, particularly how the programme was designed and commissioned and the effects PbR had. This discussion has been distilled into a set of suggestions for people commissioning PbR programmes in development.

Six suggestions for commissioners of Payment By Results programmes

The discussions reflected the participants’ experiences as people leading supplier consortia, rather than those directly implementing the programme or its beneficiaries and although DFID staff were present their role was largely that of observer. Discussions built on the early findings shared by the Oxford Policy Management team leading the formal evaluation of the programme for the e-Pact consortium. The suggestions that emerged are inevitably not as rigorous as the final findings that will come from this formal evaluation nor from others’ extensive reviews drawing on multiple programmes and experience. However, our suggestions add support to some of the National Audit Office (NAO) recommendations on PbR that were published a month after our workshop, in particular the “Payment by Results: analytical framework for decision-makers” (NAO, 2015).

1. Consider what type of PbR is the right mechanism for achieving the desired outcomes.

There are multiple ways in which PbR programmes can be designed and the selection made will influence who applies, how suppliers devise and manage their programmes and the subsequent results. An important difference between PbR programmes is the proportion of payment that is “pure PbR”. WASH Results is designed as a 100% PbR programme and this prompted some suppliers to take steps to minimise the risk of not achieving targets. Some set disbursement milestones early in the impact pathway to aid cashflow, rather than set exclusively outcome-based milestones. Some rethought the areas they were working in, shifting away from work in the most challenging areas, and there is some indication that suppliers relied on tried and trusted methods rather than trying new approaches.

The workshop participants suggested that pre-financing may be necessary to achieve genuine emphasis on outcomes, particularly in risky contexts or if innovation is important so that risks are not born solely by suppliers. Echoing other discussions on PbR (BOND p13) , participants discussed how the PbR mechanism may not be appropriate for programmes that are intended to target the most vulnerable and/or operate in fragile contexts, however, the success of the WASH Results Programme in the Democratic Republic of Congo demonstrates that it should not be ruled out. These issues need careful consideration when selecting the design of a PbR programme and the NAO report provides an analytical framework for doing so.

2. Create spaces for learning about PbR within the commissioning process.

As the PbR modality is new to many organisations (suppliers and commissioners) it is important for commissioners and suppliers to explore options and learn from experience in order to design the most effective model. Ideally this will happen in an interactive way that will help to build relationships between stakeholders. Workshop participants stressed the importance of learning from the private sector about approaches to PbR, for example in areas such as how to assess and price risk. They also advocated a longer inception phase in programmes using multiple suppliers, to enable suppliers to talk to each other about the approaches they are considering adopting.

The NAO’s analytical framework for decision-makers advocates a learning approach to commissioning PbR programmes and it encourages commissioners to engage in dialogue with potential providers from the outset. However it does not explicitly recommend creating spaces for learning and relationship-building between stakeholders within the commissioning process; we think this would be a useful addition to the framework.

3. The tender documents need to be clear about what is meant by PbR, from the outset.

This includes providing the rationale for using PbR and sufficient detail about matters such as disbursement, results, and the processes and conditions around force majeure, particularly but not exclusively in fragile contexts. There should be clear guidance on how to define minimum standards: in sanitation, for example, does the minimum criteria for improved latrines allow for shared use?  The NAO framework helps commissioners to think through high-level issues, however the detail of sector specific standards may need to be determined in pre-tendering discussions with potential suppliers and/or monitoring and verification providers.

4. Set up Monitoring, Verification and Evaluation (MVE) frameworks before implementation starts.

The MVE team for a PbR programme needs to be appointed before the suppliers and the MVE framework needs to be created before implementation starts. This would enable standardisation of approaches across suppliers and enable suppliers to create their monitoring and evaluation frameworks according to the requirements of verification processes. This is perhaps the clearest suggestion to emerge from the workshop and once again accords with the NAO framework which asks “Is there an agreed system for measuring provider performance? Will this system be in place at the start of the scheme?” The WRP suppliers’ experience suggests that it should be!

5. Consider what impact the contracting process will have on opportunities for collaboration and learning.

The workshop participants felt that the tendering process for the WASH Results Programme made international non-governmental organisations into competitors rather than collaborators and this can lead to lack of transparency and learning between organisations. Given the broader move within aid commissioning from grants towards more competitively tendered contracts, this is an example of where it is difficult to attribute effects purely to the PbR mechanism.

As an alternative to a competitive tender, the WASH Results suppliers suggested commissioners pursue a “negotiated” procurement procedure during which the commissioner enters into contract negotiations with one or more suppliers.

6. Allow extra time to commission PbR programmes than for more familiar contracting processes.

The complexity of PbR contracts and their relative unfamiliarity in the development sector imply the need to allow more time for commissioning than usual. How to use that extra time? The previous suggestions from our workshop indicate that this lengthier commissioning process should comprise the following phases:

  • Pre-bidding phase: in which the donor’s understanding of PbR is clearly laid out, and in which potential suppliers are able to access insights into the different models of PbR financing and the requirements and risks of implementing a PbR programme.
  • Bidding/contracting phase: recognising that this is a resource-intensive process for suppliers; possibly conducted as a negotiated process rather than a competitive tender.
  • Inception phase: with space for suppliers to share approaches and refine their programme design, in which a standardised MVE framework is shared and refined and means of verification agreed; milestones and payment schedules are agreed; programme expectations are agreed with partners; and consortia develop common understanding, language and approaches.
  • Implementation phase: implementation with appropriate verification cycles and disbursement points, ongoing learning and review.
  • Closing phase: for end-of-term evaluation, especially to draw out lessons learnt and find ways of furthering work beyond the project period of implementation.

Have you tendered for, or commissioned a PbR programme?

Do our suggestions and observations reflect your experience?

What suggestions would you add to our list?

Comparative experiences of evaluating and verifying Payment by Results programmes

Highlights of the discussion about verification and evaluation of Payment By Results in our chat show at the UK Evaluation Society Conference.

A session at the UK Evaluation Society conference in May 2015 compared two Payment by Results (PbR) programmes in different sectors. The discussion revealed that although PbR programmes can be set up in very different ways, the management, evaluation and verification face similar tensions. These include balancing cash flows with the focus on “real outcomes” that PbR is designed to encourage.

Using an informal chat show approach, the session explored practical experiences of monitoring and verification of two DFID funded PbR programmes: the Water, Sanitation and Hygiene (WASH) Results Programme and the Girls Education Challenge (GEC). The chat show was attended by one of the WASH Results Programme service providers thus providing a different perspective on the issues.

Chat show participants were:
Dr Katharina Welle, Deputy Team Leader, WASH Results Monitoring, Verification & Evaluation (MVE) Team, (Itad)
Dr Lucrezia Tincani, Manager, WASH Results MVE Team, (Oxford Policy Management)
Jason Calvert, Global Monitoring & Evaluation (M&E) Lead, Girls’ Education Challenge
The host was me, Catherine Fisher, Learning Adviser, WASH MVE Team, (Independent Consultant).

The discussion focused on the practicalities of implementing programmes financed through a Payment by Results funding mechanism (specifically Results Based Financing, a term used by DFID, among others, when the payments from funders or government go to service providers or suppliers). This blog post shares a few of the areas that were discussed.

There are huge variations in the set-up of PbR programmes
Perhaps unsurprisingly for two programmes of different sizes in different sectors, the WASH Results Programme (£68.7 million over approximately four years) and GEC (£330 million over four years) are managed, monitored and evaluated in very different ways.

GEC is managed by Price Waterhouse Coopers (PwC) who manage the funding to the 37 GEC projects, set suppliers’ targets and create M&E frameworks, while suppliers themselves contract external evaluators to monitor and evaluate their progress. By contrast the three WASH Results Programme supplier contracts are managed in-house by DFID and the Monitoring, Verification and Evaluation is contracted out to a consortium: e-Pact. There are no doubt pros and cons of each approach which the chat show did not explore. But a key advantage of the GEC set-up (from the perspective of the e-Pact WASH MVE team) was the involvement of PwC from the start of the GEC programme. This meant PwC could shape the “rules of the game” for verification and were able to set standardised targets up front.

Differences in amount of funding subject to PbR
Another key difference between the two programmes was the amount of the supplier funding that was subject to PbR. For GEC suppliers it is on average 10%, whereas WASH Results Programme suppliers see 100% of their funding dependent on verified results. But this startling figure may mask some similarities….

GEC differentiates between payment against inputs (not called PbR) and payment against outcomes, the “real” PbR which constitutes 10% of the total funding. By contrast, the WASH programme has a broader definition of results that includes outputs, inputs and outcomes, and varies across suppliers. So in practice the programmes may be more similar than they appear.

Balancing cash flow and adaptive programming in a PbR context
An important driver for the broader interpretation of “results” within the WASH Results programme was the very real need for some suppliers to maintain a steady cash flow across the course of the programme in a 100% PbR context.   The temptation for suppliers can be to set lots of milestones that trigger payment. However the need to stick to and demonstrate achievement of these milestones may inhibit flexibility in delivery. This tension between maintaining cash flow and the adaptive programming that PbR is intended to foster has been experienced by both programmes.

Rigour in measurement
The ability to effectively measure results is at the heart of PbR. For PwC, this means that every project subject to PbR is monitored through use of a quasi-experimental Randomised Controlled Trial (RCT) that measures outcomes in the project site with a control group. One reason PwC insist on an RCT for each project is to protect themselves against the risk of being sued for incorrectly withholding payment.

Where an RCT is not possible, for example in Afghanistan where security risks for implementers and cultural considerations mean that control groups are not feasible, these projects are removed from PbR. A number of the 37 GEC projects have been taken off PbR due to cultural considerations and challenging environments.

The ability to measure results is also dependent on the existence of consensus and evidence about expected results and effective means of measurement. This is more the case in the education sector than the WASH sector and makes setting targets and assessing progress towards them more difficult for a WASH programme, particularly in hygiene promotion activities such as those promoting hand washing.

As a result of these discussions, participants suggested the use of a spectrum (see Figure 1) that matches the proportion of programme funding dependent on PbR to how to how easy it is to measure results in that particular sector.

A potential PbR - measurement spectrum

Does PbR generate perverse incentives?
One of the much talked about cons of PbR is that it will create perverse incentives among stakeholders that drive them to behave in the opposite way than intended. Participants shared stories that included examples of the PbR mechanism inhibiting innovation and encouraging suppliers to focus on “low hanging fruit” rather than greatest need. A review of PbR in Swiss cantons suggested it didn’t work at all in terms of generating efficiencies and effectiveness.

PbR is not a one-size-fits-all solution; there remains lots to learn about when and where it can work
There was consensus that PbR cannot be used effectively in all contexts. The risks and uncertainties of working in fragile states are one setting in which PbR can be difficult (but not impossible?) to implement, however even in more stable contexts, issues around organisational capacity and motivations can inhibit its effective implementation.

Participants agreed that there are probably sets of circumstances in which PbR can be effective. People working in countries and sectors in which PbR has been used for many years have spent time identifying what those contexts are – see for example this paper about PbR in UK community work. For those of us who are new to the challenges of commissioning, designing, implementing, monitoring, evaluating or verifying PbR projects and are doing it in contexts in which it has not been tried, there is a lot to learn.  This chat show illustrates that there is great value in coming together to do so.

What do you think?
What are you learning about managing, implementing or verifying Payment By Results programmes? Does this discussion reflect your experience? Would you like to learn with us?  Please use the comment box to share your thoughts.

Catherine Fisher, Learning Adviser, WASH Results MVE Team.

See also: Payment-by-results: the elixir or the poison?, 6 January 2015, Jason Calvert

Introducing DFID WASH Results

By 2015 the proportion of the population without sustainable access to safe drinking water and basic sanitation will be halved (according to Millennium Development Goal 7c.) Yet, millions of people worldwide still lack access to improved water supply and the sanitation target is severely off track.
The DFID Water, Sanitation and Hygiene (WASH) Results Programme aims to support poor people to access improved water and sanitation, and to introduce improved hygiene practices, to support the UK Government’s targets of reaching 60 million people with improved water, sanitation and hygiene initiatives. Three Suppliers have been contracted by DFID to deliver results by December 2015 and strengthen local systems and build local capacity to help sustain these results up to March 2018:

WASH Results uses a funding relationship known as Results Based Financing (RBF) where the funding goes from DFID to a Supplier or Service Provider. RBF is a type of Payment By Results (PbR), a relatively new form of financing used by DFID that makes payments contingent on the independent verification of results – which is where e-Pact comes in.

The e-Pact consortium is the Monitoring, Verification & Evaluation (MVE) services provider for WASH Results with Itad as the lead agency, joined by OPM, IWEL and Ecorys.
The Monitoring & Verification team supports DFID by monitoring progress and verifying that the Suppliers’ reporting on results is accurate and realistic.
The Evaluation team is assessing if and why the programme delivers sustainable outcomes, how efficiently and effectively it is and how to effectively design and implement RBF programmes in the WASH sector.
Cutting across both teams is a Learning & Dissemination function, that supports knowledge-sharing and learning between the Suppliers, DFID and the MVE team and helps lessons learned about RBF in WASH reach a wide audience.