Tag Archives: commissioning

The paybacks and pains of Payment by Results (Part 2)

Our series of reflections on WASH Results’ learning continues by exploring value and costs in a Payment by Results (PBR) programme.

DFID has been clear from the outset about what it wants from the Water, Sanitation and Hygiene (WASH) Results Programme: WASH interventions delivered at scale within a short time-frame and confidence in the results being reported to the UK taxpayer. DFID got what it wanted, but at what cost? In this post we build on discussions at the WASH Results Programme’s learning event held earlier this year which looked beyond the numbers of people reached with interventions to explore some of the challenges faced in implementing the programme.

Can PBR frameworks be designed to incentivise suppliers to focus on the “harder to reach”

A central theme in the workshop was the ongoing puzzle of how to place value (both in commercial/financial and Value for Money terms) on intangible aspirations and benefits, such as reaching the most vulnerable and investing in the processes and social capital that underpin effective programming. So, does the importance placed by a donor on achieving clearly costed, verified results risk squeezing out other values and principles that are central to development programming? Which values might end up being pushed aside and could this be mitigated through better design?

1. High quality programming

Suppliers hit a major challenge during tendering when DFID asked them to provide a price per beneficiary that reflected the cost to the suppliers of reaching that beneficiary. But calculating this cost is complex. Potential suppliers have to think about what kind of costs they should fold into that price per beneficiary when bidding: the more the costs, the higher the bid. During the workshop one Supplier asked rhetorically “Should we say $20 with alignment and $23 without?”

There is some apprehensiveness within the NGO sector about competing with the private sector in this commercial context and they are often advised to be cautious. Will they be undercut by commercial organisations submitting more attractive (read cheaper) bids that lack the added benefits that NGOs can bring: the social capital and ways of working that are difficult to put a commercial value on but will affect the quality of the programming?

DFID has been clear that it does not equate Value for Money (VfM) with “cheap” and it is willing to pay for quality programming, whoever is best placed to deliver it. One improvement to the tendering process would be to articulate some of these added benefits (such as existing relationships and social capital in a programme area) as requirements for bidders. Potential suppliers would thus need to provide evidence within the bidding process.

2. Reaching the hardest to reach 

A criticism levelled at PBR is that by using a fixed “price per beneficiary” approach, it encourages suppliers to focus on people who are easier to reach, a practice sometimes described as “creaming” or “cherry picking”. Stakeholders in the WASH Results Programme are firmly committed to inclusion and during the workshop investigated how that could be incentivised better within a PBR framework. Options explored included multi-tiered price per beneficiary frameworks (as used in drug and alcohol recovery programmes in the UK) although these carry the risk of increasing complexity and reducing flexibility. Another suggestion for incentivising inclusion was careful selection and wording of the objectives and appropriate verification processes in the tender document, however this may risk compromising the flexibility to negotiate targets and verification approaches in response to different contexts.

3. Investing for the future

One related but different challenge that emerged during the workshop was that of placing commercial value on activities that invest for future work in the sector. This includes building the social capital to work with local stakeholders and investing in programmatic innovation (which some suppliers suggested had not been possible under the WASH Results Programme). Do the practical implications of PBR risk capitalising on previous investment made by suppliers, without contributing to it in turn? This is perhaps not an issue while PBR contracts constitute a small proportion of aid financing but would become more so if PBR contracts started to dominate. On the other hand, the benefits that suppliers report, particularly in terms of strengthening monitoring and reporting systems to enable more rigorous real-time results tracking may also spill over into other programmes, benefitting them in turn. It is too early to draw conclusions but it may be the case that a range of different aid mechanisms are required, with the benefits and limitations of each clearly identified.

4. Confidence in results

Finally, it is worth observing the possible trade-off between the value placed by DFID on confidence in results that is so important for communicating with taxpayers, and the effectiveness of aid spending that can be achieved through PBR and the nature of the results it produces. Verification is undoubtedly costly (someone paid you to come here just to look at that toilet?” a baffled resident of a beneficiary village is reported to have asked of a verification team member).

But there is another aspect of effectiveness: if PBR prompts suppliers to focus their efforts on what can be counted (i.e. what can be verified at scale without incurring prohibitive expense), this may shift their efforts away from development programming with longer-term and more uncertain outcomes. Put simply, this could equate to building toilets rather than working on sanitation behaviour change interventions, that are considered to generate more sustainable positive outcomes. Of course there is no guarantee other forms of aid financing will generate these results and as there is likely to be less focus on measuring those results, comparison would be difficult.

Advice for PBR commissioners

What might this mean for those considering PBR modalities and designing PBR programmes? The experience of WASH Results so far suggests that when designing a PBR programme, commissioners need to:

  • be clear on the value implied in “value for money” – consider all of the “values” that are important, including the value of donor confidence in results;
  • strike a balance between clearly specifying expected results (particularly for more vulnerable people) and being flexible to the contexts in which suppliers are operating;
  • think creatively and collaboratively about how long-term outcomes can be measured;
  • explore hybrid funding models but avoid creating the “worst of all worlds” that lacks the flexibility of PBR, increases complexity and imposes multiple reporting frameworks;
  • consider whether PBR is the right funding mechanism for the kinds of results you wish to achieve (tools are emerging that can help) ;
  • view the PBR component in the context of the broad spectrum of funding to the sector – seek to maximise linkages and mutual value across the sector.

Catherine Fisher, Learning Advisor, WASH Results MVE Team

The report from the WASH Results learning workshop is available to download from DFID’s Research for Development website. 

As always, if you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us.

Suggestions to donors commissioning Payment by Results programmes

Reflections by DFID WASH Results’ suppliers on the programme’s design and commissioning, summarised into six suggestions for commissioners.

Payment by Results (PbR) is a technically challenging form of contracting and one that is new for DFID and many of its partners (NAO 2015, p8, DFID, 2015 p18).   The WASH Results Programme was designed using the PbR financing mechanism, and therefore has great potential for learning about the impact of PbR on programming.

One year into the programme, the WASH Results suppliers met to share experiences so far, particularly how the programme was designed and commissioned and the effects PbR had. This discussion has been distilled into a set of suggestions for people commissioning PbR programmes in development.

Six suggestions for commissioners of Payment By Results programmes

The discussions reflected the participants’ experiences as people leading supplier consortia, rather than those directly implementing the programme or its beneficiaries and although DFID staff were present their role was largely that of observer. Discussions built on the early findings shared by the Oxford Policy Management team leading the formal evaluation of the programme for the e-Pact consortium. The suggestions that emerged are inevitably not as rigorous as the final findings that will come from this formal evaluation nor from others’ extensive reviews drawing on multiple programmes and experience. However, our suggestions add support to some of the National Audit Office (NAO) recommendations on PbR that were published a month after our workshop, in particular the “Payment by Results: analytical framework for decision-makers” (NAO, 2015).

1. Consider what type of PbR is the right mechanism for achieving the desired outcomes.

There are multiple ways in which PbR programmes can be designed and the selection made will influence who applies, how suppliers devise and manage their programmes and the subsequent results. An important difference between PbR programmes is the proportion of payment that is “pure PbR”. WASH Results is designed as a 100% PbR programme and this prompted some suppliers to take steps to minimise the risk of not achieving targets. Some set disbursement milestones early in the impact pathway to aid cashflow, rather than set exclusively outcome-based milestones. Some rethought the areas they were working in, shifting away from work in the most challenging areas, and there is some indication that suppliers relied on tried and trusted methods rather than trying new approaches.

The workshop participants suggested that pre-financing may be necessary to achieve genuine emphasis on outcomes, particularly in risky contexts or if innovation is important so that risks are not born solely by suppliers. Echoing other discussions on PbR (BOND p13) , participants discussed how the PbR mechanism may not be appropriate for programmes that are intended to target the most vulnerable and/or operate in fragile contexts, however, the success of the WASH Results Programme in the Democratic Republic of Congo demonstrates that it should not be ruled out. These issues need careful consideration when selecting the design of a PbR programme and the NAO report provides an analytical framework for doing so.

2. Create spaces for learning about PbR within the commissioning process.

As the PbR modality is new to many organisations (suppliers and commissioners) it is important for commissioners and suppliers to explore options and learn from experience in order to design the most effective model. Ideally this will happen in an interactive way that will help to build relationships between stakeholders. Workshop participants stressed the importance of learning from the private sector about approaches to PbR, for example in areas such as how to assess and price risk. They also advocated a longer inception phase in programmes using multiple suppliers, to enable suppliers to talk to each other about the approaches they are considering adopting.

The NAO’s analytical framework for decision-makers advocates a learning approach to commissioning PbR programmes and it encourages commissioners to engage in dialogue with potential providers from the outset. However it does not explicitly recommend creating spaces for learning and relationship-building between stakeholders within the commissioning process; we think this would be a useful addition to the framework.

3. The tender documents need to be clear about what is meant by PbR, from the outset.

This includes providing the rationale for using PbR and sufficient detail about matters such as disbursement, results, and the processes and conditions around force majeure, particularly but not exclusively in fragile contexts. There should be clear guidance on how to define minimum standards: in sanitation, for example, does the minimum criteria for improved latrines allow for shared use?  The NAO framework helps commissioners to think through high-level issues, however the detail of sector specific standards may need to be determined in pre-tendering discussions with potential suppliers and/or monitoring and verification providers.

4. Set up Monitoring, Verification and Evaluation (MVE) frameworks before implementation starts.

The MVE team for a PbR programme needs to be appointed before the suppliers and the MVE framework needs to be created before implementation starts. This would enable standardisation of approaches across suppliers and enable suppliers to create their monitoring and evaluation frameworks according to the requirements of verification processes. This is perhaps the clearest suggestion to emerge from the workshop and once again accords with the NAO framework which asks “Is there an agreed system for measuring provider performance? Will this system be in place at the start of the scheme?” The WRP suppliers’ experience suggests that it should be!

5. Consider what impact the contracting process will have on opportunities for collaboration and learning.

The workshop participants felt that the tendering process for the WASH Results Programme made international non-governmental organisations into competitors rather than collaborators and this can lead to lack of transparency and learning between organisations. Given the broader move within aid commissioning from grants towards more competitively tendered contracts, this is an example of where it is difficult to attribute effects purely to the PbR mechanism.

As an alternative to a competitive tender, the WASH Results suppliers suggested commissioners pursue a “negotiated” procurement procedure during which the commissioner enters into contract negotiations with one or more suppliers.

6. Allow extra time to commission PbR programmes than for more familiar contracting processes.

The complexity of PbR contracts and their relative unfamiliarity in the development sector imply the need to allow more time for commissioning than usual. How to use that extra time? The previous suggestions from our workshop indicate that this lengthier commissioning process should comprise the following phases:

  • Pre-bidding phase: in which the donor’s understanding of PbR is clearly laid out, and in which potential suppliers are able to access insights into the different models of PbR financing and the requirements and risks of implementing a PbR programme.
  • Bidding/contracting phase: recognising that this is a resource-intensive process for suppliers; possibly conducted as a negotiated process rather than a competitive tender.
  • Inception phase: with space for suppliers to share approaches and refine their programme design, in which a standardised MVE framework is shared and refined and means of verification agreed; milestones and payment schedules are agreed; programme expectations are agreed with partners; and consortia develop common understanding, language and approaches.
  • Implementation phase: implementation with appropriate verification cycles and disbursement points, ongoing learning and review.
  • Closing phase: for end-of-term evaluation, especially to draw out lessons learnt and find ways of furthering work beyond the project period of implementation.

Have you tendered for, or commissioned a PbR programme?

Do our suggestions and observations reflect your experience?

What suggestions would you add to our list?