Tag Archives: measurement

Measuring WASH sustainability in a payment by results context

Andy Robinson, Lead Verifier for SNV, reports back from a WEDC Conference panel session organised by the three WASH Results suppliers

The three suppliers in the DFID WASH Results programme (SAWRP consortium, SWIFT consortium and SNV) came together at a side event held during the WEDC Conference in Kumasi, Ghana (11-15 July 2016) to present their thoughts on “measuring WASH sustainability in a Payment by Results (PBR) context”.

As Lead Verifier on the SNV contract and WEDC conference participant, I was invited to join the panel with the three suppliers and make a short presentation on behalf of the e-Pact consortium – to explain e-Pact’s role in WASH Results and elaborate some of the initial learning from the perspective of our Monitoring and Verification (MV) team.

Kevin Sansom (WEDC, SAWRP) began by outlining the key differences between PBR and grant programmes. He noted that PBR programmes require significant pre-finance and engender higher risks (particularly when tight timelines are applied), but allow greater flexibility and encourage more rigorous monitoring and evaluation (both internally, within the implementing agencies and externally by the verification and evaluation teams).

SAWRP presentation

Mimi Coultas (Plan UK) detailed the sustainability monitoring system adopted by the SAWRP consortium, explaining that some of the elements (sustainability assessment frameworks, outcome implementation manuals and the learning framework) are not linked to payments, but are designed to meet DFID’s requirement for reporting against five different dimensions of sustainability (functional, institutional, financial, environmental and equity).

Mimi noted that there was a lack of clarity at the outset around the criteria for payment (and the criteria for disallowance of payments), which caused some problems and could have been avoided by agreeing these details during a longer inception phase. She also suggested that the sampling approach used by the MV team has the potential “to scale mistakes” by exaggerating the effect of any poor results included in the sample (to suggest problems larger than actuality). Another comment was that the commercial pressures on the suppliers, all of whom are interested in bidding for any follow-on programmes, might have reduced collaboration and sharing of lessons learned.

Nonetheless, the SAWRP consortium felt that the programme had produced “amazing results”, with a high level of confidence in the quality and reliability of the results due to the strong scrutiny provided by the MV team. Mimi also noted that the monitoring and evaluation (M&E) focus required by the programme was a positive outcome, leading to a strengthening of M&E systems and the development of better ways of measuring WASH outcomes and sustainability. However, a longer programme duration would have been better, including an inception period during which the results framework and verification approaches could be carefully designed and negotiated.

SNV presentation

Anne Mutta (SNV) talked about the critical importance of political engagement to WASH sustainability, with governance activities integrated into the SNV programme from the start to address this requirement. Where local government capacity for sanitation and hygiene is low, sustainable results will obviously be harder to achieve. She also noted that some practical sustainability problems arise, such as heavy rain and flooding (which can wash away sanitation facilities, and constrain implementation) and changes in capacity, knowledge and commitment due to issues like government transfers or elections. Anne also agreed that the PBR programme required stronger progress monitoring, to track results and allow course corrections before the household survey results are verified.

SWIFT presentation

Rachel Stevens (TEARFUND) explained that the SWIFT consortium is using household, water point and latrine surveys, as well as local government and local service provider data, to assess sustainability (with two sets of surveys planned – one in mid-2016 and the other at end-2017). The SWIFT sustainability assessments use a similar traffic light system to those described by the other two suppliers, reporting against DFID’s five dimensions of sustainability.

Common challenges

The three suppliers had agreed on a list of common challenges, which were presented by Mimi Coultas (Plan UK). One of the most interesting of these was the risk that PBR encourages implementation in easier contexts – through the selection of less vulnerable and more accessible communities and project areas – in order to reduce both cost and risk.

The suppliers also questioned whether verification was appropriate for all aspects of sustainability, particularly the intangible and more qualitative factors (such as community empowerment), which are often important elements associated with the sustainability of sanitation and hygiene practices and outcomes.

Another potential issue is that the reduced reporting burden, with the production of evidence of results generally replacing the need for the detailed progress reporting and evaluation required by conventional programmes, may mean that the lessons learned by the programmes are not well captured or adequately documented.

Common opportunities

The suppliers agreed that, while some aspects of sustainability may be missed, the inclusion of payments for specific sustainability outcomes led to more attention to sustainability than in conventional programmes. Furthermore, the MV team’s work had encouraged greater transparency and accountability.

MV presentation

I made a short presentation on the role of the MV team and the key challenges and opportunities. After describing the composition of the e-Pact team, and introducing Bertha Darteh (Ghana country verifier for the SNV programme, who was in the audience), I explained that we were using “systems-based verification” rather than fully independent verification, which means that we are reliant on the data and reports produced by the suppliers’ M&E systems. As a result, we have to understand these systems well, and identify any weaknesses and any potential for errors, misreporting or gaming of results. DFID’s decision to adopt a systems-based verification approach was based on the assumption it would be cheaper than statistically sampled independent surveys (across such a large population), but the MV experience suggests that there are a lot of unforeseen costs (often to the suppliers) related to this systems-based approach.

Key verification challenges include the large number of closely spaced results, with little time between each verification cycle for the design, review and improvement of the verification process. The SNV programme includes nine country projects, with significant variations in context across the projects, which requires considerable flexibility in the verification system; whereas the other two suppliers’ programmes include multiple implementation partners, each of which has slightly different monitoring and reporting systems, and different priorities and targets, which in turn require adaptation of the verification systems.

I concurred that not enough time had been provided up front for the planning and design of the programme, including the MV framework and activities, which increased the pressure on all stakeholders during the first year of the programme, when suppliers were developing systems, implementing and reporting, with little time to respond to the additional demands of the verification process.

One positive outcome of the need for verified results has been the use of smartphone survey applications, which have greatly sped up and reduced the cost of the survey process; improved data processing and quality control; and made it much easier to verify large-scale results quickly. A key learning from the PBR programme is that these household surveys appear to be a far quicker and more effective way of evaluating programme outcomes than conventional evaluations.

Overall, the PBR approach appears to be improving M&E approaches and systems, encouraging more thinking about how to measure and evidence outcomes and sustainability, and providing reliable feedback on progress and performance at regular intervals during the life of the programme. This feedback enables regular improvements to be made to programme policy, planning and practice (unlike conventional programmes, which often are not rigorously evaluated until the end of the programme duration).

Questions from the floor

When the panel was asked whether the PBR approach encourages efficiency, the suppliers noted that both the programme and the approach encourage scale, which in turn encourages efficiency; however, the additional costs of verification and the related reporting were thought to partially offset the efficiency gains.

A similar question was asked about whether PBR encouraged value-for-money: the suppliers suggested that they are very confident of their results (compared to conventional programmes, which may over-report results), thus the cost-per-result is clear. They also noted that there is an incentive to reduce costs, but that these reductions may not always be passed on (and, because there is no payment for over-achievement in this programme, any additional results appear to reduce the cost per outcome/result, but do not change the suppliers’ fixed costs).

Several Ghanaian participants expressed their confusion about the new terminology associated with PBR. Output based Aid (OBA) is common in Ghana, notably through a World Bank WASH programme (with payments linked to toilet construction), and it was suggested that there “was no need to introduce yet another acronym for the same thing”. Louise Medland (WEDC, SAWRP) responded that DFID differentiated between the OBA and PBR approaches by the PBR focus on outcomes (whereas OBA focuses on outputs).

The final question was around PBR’s effect on innovation: the suppliers noted that the design was supposed to encourage innovation, but that the time pressure (of the short implementation period) limited the chance of innovation. I added that we have seen different outcomes in different contexts – in low capacity settings, the programme management generally provide firm guidelines to the project team to minimise risk; but in high capacity settings, there was evidence of innovation driven by the need to achieve results, especially in more difficult contexts where standard approaches were not working.

The general tone of the PBR session was positive, with the suppliers agreeing that the PBR approach has led to reliable and large-scale results, and that the need to report and verify results has led to significant improvements in M&E systems. A lot of learning has taken place, and the suppliers hoped that this learning will inform the design of any future WASH PBR programmes.

Andy Robinson, Lead Verifier on the SNV Contract, WASH Results MVE Team

The paybacks and pains of Payment by Results (Part 2)

Our series of reflections on WASH Results’ learning continues by exploring value and costs in a Payment by Results (PBR) programme.

DFID has been clear from the outset about what it wants from the Water, Sanitation and Hygiene (WASH) Results Programme: WASH interventions delivered at scale within a short time-frame and confidence in the results being reported to the UK taxpayer. DFID got what it wanted, but at what cost? In this post we build on discussions at the WASH Results Programme’s learning event held earlier this year which looked beyond the numbers of people reached with interventions to explore some of the challenges faced in implementing the programme.

Can PBR frameworks be designed to incentivise suppliers to focus on the “harder to reach”

A central theme in the workshop was the ongoing puzzle of how to place value (both in commercial/financial and Value for Money terms) on intangible aspirations and benefits, such as reaching the most vulnerable and investing in the processes and social capital that underpin effective programming. So, does the importance placed by a donor on achieving clearly costed, verified results risk squeezing out other values and principles that are central to development programming? Which values might end up being pushed aside and could this be mitigated through better design?

1. High quality programming

Suppliers hit a major challenge during tendering when DFID asked them to provide a price per beneficiary that reflected the cost to the suppliers of reaching that beneficiary. But calculating this cost is complex. Potential suppliers have to think about what kind of costs they should fold into that price per beneficiary when bidding: the more the costs, the higher the bid. During the workshop one Supplier asked rhetorically “Should we say $20 with alignment and $23 without?”

There is some apprehensiveness within the NGO sector about competing with the private sector in this commercial context and they are often advised to be cautious. Will they be undercut by commercial organisations submitting more attractive (read cheaper) bids that lack the added benefits that NGOs can bring: the social capital and ways of working that are difficult to put a commercial value on but will affect the quality of the programming?

DFID has been clear that it does not equate Value for Money (VfM) with “cheap” and it is willing to pay for quality programming, whoever is best placed to deliver it. One improvement to the tendering process would be to articulate some of these added benefits (such as existing relationships and social capital in a programme area) as requirements for bidders. Potential suppliers would thus need to provide evidence within the bidding process.

2. Reaching the hardest to reach 

A criticism levelled at PBR is that by using a fixed “price per beneficiary” approach, it encourages suppliers to focus on people who are easier to reach, a practice sometimes described as “creaming” or “cherry picking”. Stakeholders in the WASH Results Programme are firmly committed to inclusion and during the workshop investigated how that could be incentivised better within a PBR framework. Options explored included multi-tiered price per beneficiary frameworks (as used in drug and alcohol recovery programmes in the UK) although these carry the risk of increasing complexity and reducing flexibility. Another suggestion for incentivising inclusion was careful selection and wording of the objectives and appropriate verification processes in the tender document, however this may risk compromising the flexibility to negotiate targets and verification approaches in response to different contexts.

3. Investing for the future

One related but different challenge that emerged during the workshop was that of placing commercial value on activities that invest for future work in the sector. This includes building the social capital to work with local stakeholders and investing in programmatic innovation (which some suppliers suggested had not been possible under the WASH Results Programme). Do the practical implications of PBR risk capitalising on previous investment made by suppliers, without contributing to it in turn? This is perhaps not an issue while PBR contracts constitute a small proportion of aid financing but would become more so if PBR contracts started to dominate. On the other hand, the benefits that suppliers report, particularly in terms of strengthening monitoring and reporting systems to enable more rigorous real-time results tracking may also spill over into other programmes, benefitting them in turn. It is too early to draw conclusions but it may be the case that a range of different aid mechanisms are required, with the benefits and limitations of each clearly identified.

4. Confidence in results

Finally, it is worth observing the possible trade-off between the value placed by DFID on confidence in results that is so important for communicating with taxpayers, and the effectiveness of aid spending that can be achieved through PBR and the nature of the results it produces. Verification is undoubtedly costly (someone paid you to come here just to look at that toilet?” a baffled resident of a beneficiary village is reported to have asked of a verification team member).

But there is another aspect of effectiveness: if PBR prompts suppliers to focus their efforts on what can be counted (i.e. what can be verified at scale without incurring prohibitive expense), this may shift their efforts away from development programming with longer-term and more uncertain outcomes. Put simply, this could equate to building toilets rather than working on sanitation behaviour change interventions, that are considered to generate more sustainable positive outcomes. Of course there is no guarantee other forms of aid financing will generate these results and as there is likely to be less focus on measuring those results, comparison would be difficult.

Advice for PBR commissioners

What might this mean for those considering PBR modalities and designing PBR programmes? The experience of WASH Results so far suggests that when designing a PBR programme, commissioners need to:

  • be clear on the value implied in “value for money” – consider all of the “values” that are important, including the value of donor confidence in results;
  • strike a balance between clearly specifying expected results (particularly for more vulnerable people) and being flexible to the contexts in which suppliers are operating;
  • think creatively and collaboratively about how long-term outcomes can be measured;
  • explore hybrid funding models but avoid creating the “worst of all worlds” that lacks the flexibility of PBR, increases complexity and imposes multiple reporting frameworks;
  • consider whether PBR is the right funding mechanism for the kinds of results you wish to achieve (tools are emerging that can help) ;
  • view the PBR component in the context of the broad spectrum of funding to the sector – seek to maximise linkages and mutual value across the sector.

Catherine Fisher, Learning Advisor, WASH Results MVE Team

The report from the WASH Results learning workshop is available to download from DFID’s Research for Development website. 

As always, if you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us.

Comparative experiences of evaluating and verifying Payment by Results programmes

Highlights of the discussion about verification and evaluation of Payment By Results in our chat show at the UK Evaluation Society Conference.

A session at the UK Evaluation Society conference in May 2015 compared two Payment by Results (PbR) programmes in different sectors. The discussion revealed that although PbR programmes can be set up in very different ways, the management, evaluation and verification face similar tensions. These include balancing cash flows with the focus on “real outcomes” that PbR is designed to encourage.

Using an informal chat show approach, the session explored practical experiences of monitoring and verification of two DFID funded PbR programmes: the Water, Sanitation and Hygiene (WASH) Results Programme and the Girls Education Challenge (GEC). The chat show was attended by one of the WASH Results Programme service providers thus providing a different perspective on the issues.

Chat show participants were:
Dr Katharina Welle, Deputy Team Leader, WASH Results Monitoring, Verification & Evaluation (MVE) Team, (Itad)
Dr Lucrezia Tincani, Manager, WASH Results MVE Team, (Oxford Policy Management)
Jason Calvert, Global Monitoring & Evaluation (M&E) Lead, Girls’ Education Challenge
The host was me, Catherine Fisher, Learning Adviser, WASH MVE Team, (Independent Consultant).

The discussion focused on the practicalities of implementing programmes financed through a Payment by Results funding mechanism (specifically Results Based Financing, a term used by DFID, among others, when the payments from funders or government go to service providers or suppliers). This blog post shares a few of the areas that were discussed.

There are huge variations in the set-up of PbR programmes
Perhaps unsurprisingly for two programmes of different sizes in different sectors, the WASH Results Programme (£68.7 million over approximately four years) and GEC (£330 million over four years) are managed, monitored and evaluated in very different ways.

GEC is managed by Price Waterhouse Coopers (PwC) who manage the funding to the 37 GEC projects, set suppliers’ targets and create M&E frameworks, while suppliers themselves contract external evaluators to monitor and evaluate their progress. By contrast the three WASH Results Programme supplier contracts are managed in-house by DFID and the Monitoring, Verification and Evaluation is contracted out to a consortium: e-Pact. There are no doubt pros and cons of each approach which the chat show did not explore. But a key advantage of the GEC set-up (from the perspective of the e-Pact WASH MVE team) was the involvement of PwC from the start of the GEC programme. This meant PwC could shape the “rules of the game” for verification and were able to set standardised targets up front.

Differences in amount of funding subject to PbR
Another key difference between the two programmes was the amount of the supplier funding that was subject to PbR. For GEC suppliers it is on average 10%, whereas WASH Results Programme suppliers see 100% of their funding dependent on verified results. But this startling figure may mask some similarities….

GEC differentiates between payment against inputs (not called PbR) and payment against outcomes, the “real” PbR which constitutes 10% of the total funding. By contrast, the WASH programme has a broader definition of results that includes outputs, inputs and outcomes, and varies across suppliers. So in practice the programmes may be more similar than they appear.

Balancing cash flow and adaptive programming in a PbR context
An important driver for the broader interpretation of “results” within the WASH Results programme was the very real need for some suppliers to maintain a steady cash flow across the course of the programme in a 100% PbR context.   The temptation for suppliers can be to set lots of milestones that trigger payment. However the need to stick to and demonstrate achievement of these milestones may inhibit flexibility in delivery. This tension between maintaining cash flow and the adaptive programming that PbR is intended to foster has been experienced by both programmes.

Rigour in measurement
The ability to effectively measure results is at the heart of PbR. For PwC, this means that every project subject to PbR is monitored through use of a quasi-experimental Randomised Controlled Trial (RCT) that measures outcomes in the project site with a control group. One reason PwC insist on an RCT for each project is to protect themselves against the risk of being sued for incorrectly withholding payment.

Where an RCT is not possible, for example in Afghanistan where security risks for implementers and cultural considerations mean that control groups are not feasible, these projects are removed from PbR. A number of the 37 GEC projects have been taken off PbR due to cultural considerations and challenging environments.

The ability to measure results is also dependent on the existence of consensus and evidence about expected results and effective means of measurement. This is more the case in the education sector than the WASH sector and makes setting targets and assessing progress towards them more difficult for a WASH programme, particularly in hygiene promotion activities such as those promoting hand washing.

As a result of these discussions, participants suggested the use of a spectrum (see Figure 1) that matches the proportion of programme funding dependent on PbR to how to how easy it is to measure results in that particular sector.

A potential PbR - measurement spectrum

Does PbR generate perverse incentives?
One of the much talked about cons of PbR is that it will create perverse incentives among stakeholders that drive them to behave in the opposite way than intended. Participants shared stories that included examples of the PbR mechanism inhibiting innovation and encouraging suppliers to focus on “low hanging fruit” rather than greatest need. A review of PbR in Swiss cantons suggested it didn’t work at all in terms of generating efficiencies and effectiveness.

PbR is not a one-size-fits-all solution; there remains lots to learn about when and where it can work
There was consensus that PbR cannot be used effectively in all contexts. The risks and uncertainties of working in fragile states are one setting in which PbR can be difficult (but not impossible?) to implement, however even in more stable contexts, issues around organisational capacity and motivations can inhibit its effective implementation.

Participants agreed that there are probably sets of circumstances in which PbR can be effective. People working in countries and sectors in which PbR has been used for many years have spent time identifying what those contexts are – see for example this paper about PbR in UK community work. For those of us who are new to the challenges of commissioning, designing, implementing, monitoring, evaluating or verifying PbR projects and are doing it in contexts in which it has not been tried, there is a lot to learn.  This chat show illustrates that there is great value in coming together to do so.

What do you think?
What are you learning about managing, implementing or verifying Payment By Results programmes? Does this discussion reflect your experience? Would you like to learn with us?  Please use the comment box to share your thoughts.

Catherine Fisher, Learning Adviser, WASH Results MVE Team.

See also: Payment-by-results: the elixir or the poison?, 6 January 2015, Jason Calvert