Truly exceptional? Handling misfortune within Payment by Results

An exceptional event or a predictable adversity? The difference matters more in a Payment by Results (PbR) setting, as this blog post explores.

Conflict, political upheaval, epidemic, drought, flooding and earthquake; the WASH Results Programme has been hit by a wide range of disasters across the 12 countries in which it operates. All these adversities had an impact on the populations involved: some hit the programme’s implementation, some the sustainability of its achievements and others have affected the ability to monitor and verify those achievements.

Discussions on how to deal with these events have involved considering what is within the reasonable expectation of a Supplier to anticipate and deal with, and what is a truly exceptional event for which there should be flexibility around what Suppliers are expected to deliver – whether in the quantity, scope or timing of results.

The challenge of responding to exceptional events is not new for development programmers, but like many challenges, it takes a different shape in the context of a PbR programme. In such programmes, payment is linked to achieving and verifying results, and an impact on results ultimately leads to a financial impact for one or more of the parties. This is particularly challenging in the second phase of WASH Results, where Suppliers are paid for sustaining results achieved in the first “Outputs” phase. The passage of time means that programme achievements (whether physical infrastructure or behaviour change) are more likely to be affected by exceptional events, and suppliers may not have the resources (such as programme field staff) in place to undertake substantial mitigating actions.

Members of our team (Monitoring and Verification for the WASH Results Programme) recently met to discuss their experience of exceptional events in the programme and how these were tackled. Here are three of the examples they discussed followed by the team’s reflections on the issues they raise:

1) The moving population In this example a conflict-affected community relocated out of the proposed project area. In response, the Supplier closed the project in that area but thanks to overachievement of results in other locations, the overall outputs of the programme (in terms of beneficiaries reached) were not reduced and the Supplier did not suffer financially. In this case, the flexibility of PbR meant the risk to Supplier, Donor and Verifier could be effectively mitigated, although some of the original intended beneficiaries were not reached.

2) Destroyed infrastructure, different decisions In one instance, WASH infrastructure built in the first (output) phase of the WASH Results Programme was destroyed when a river eroded part of a village.  Although there was a known risk of erosion, the timing of erosion could not be foreseen nor the risk mitigated. The people living in this area were some of the poorest and most vulnerable whom the Supplier did not want to exclude from the programme.  The erosion was considered extreme, as evidenced by newspaper reports and other local data and it was agreed the area with the destroyed infrastructure would not be included in the sample frame for outcome surveys and so would not affect outcome results.

Meanwhile, in the same country, infrastructure was damaged by flooding, but this was considered expected, not extreme. In contexts where flooding can be expected, the demand for sustained outcomes (in which payment is linked to the sustained use of infrastructure) requires that infrastructure is built in such a way that it can withstand expected levels of flooding, or that plans for reconstruction or repair in the case of damage should be integral to programming. Consequently, areas in which infrastructure was affected by flooding were to be included in the sample frame for the outcome survey, which was amended to include questions about flood damage and beneficiary priorities for reconstruction.

3) When verification is too risky  When conflict erupted in one project location, the programme was able to implement activities regardless and continued to achieve results. However, the security situation on the ground made it too risky (for programme staff and the verification team) for the results to be independently verified through a field visit. In this case, alternative and less representative forms of verification were accepted. In this example, there was no adverse impact on the results achieved, or reduction in payment to the Supplier, but there was increased risk around the confidence that could be placed in the results.

Making decisions about risk

In exceptional circumstances, decisions need to be made about who bears the risk (Donor, Supplier, Verifier, Beneficiaries) and what kind of risk (physical, financial, reputational). If financial risk falls exclusively on Suppliers, they need to factor that into their “price per beneficiary” across the programme. Alternatively, Suppliers may choose not to operate in riskier areas, with potential negative consequences for the equity of programme interventions. If donors accept all risk, there is little incentive for Suppliers to programme in ways that account for predictable risks, such as flooding, particularly over the longer term.

Reflections and suggestions emerging from the discussion of these cases included the following:

• There are different types of impact to consider: effect on population, effect on ability to deliver activities, effect on achievement of results, and effect on ability to verify results. Being clear on the type of impact might aid decisions about who bears the risk and the mitigation strategy.
• Discussions about risk need to happen during the design phase; one approach is to use a risk matrix that explores what level of risk each party is going to absorb (and so design into the programme) and what would be considered ‘exceptional’.
• Programmes need to include within their design plans for responding to anticipated events e.g. in areas at risk of flood, include help for villages to cope with normal levels of flooding.
• Suppliers can minimise their financial and operational risk by balancing their work across a range of fragile and more secure areas, so enabling them to pull out of more hazardous areas in extreme circumstances and achieve results elsewhere. However, if the Supplier commits to working with specific communities in conflict-affected areas, then incentives will need to be set up differently within the results framework.
• In fragile contexts, a compromise may need to be made on rigour of verification and plans made for reliance on remote verification from the start, e.g. analysis of systems, remote data collection through phone or satellite, and beneficiary monitoring.

Our conclusions about exceptional events in the context of a PbR programme in WASH echo those in many of our previous blog posts. PbR throws a spotlight on an issue, raises questions of how risk is shared between stakeholders, and highlights the importance of planning at design phase and of flexibility, should the unforeseen occur.

If you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us

Recent views and news on Payment by Results – February 2018

What are people saying about Payment by Results (PBR) and development in 2018?

“We can’t ignore the importance of the process”, David Archer of ActionAid is reported to have said during a debate on Results Based Financing at the Global Partnership for Education’s Financing Conference. As well as more views from ActionAid, the live blog of the session on February 1st, also shares Norad’s experiences of Results Based Financing.

A policy paper published by UNESCO the previous month, questions the sustainability of impact on education systems achieved via results-based payments and urges donors to use caution. The paper looks at why there are so few evaluations in this area and the efforts being made to collect more evidence.

Also in January, the IDB published a Working Paper that seeks to answer the question ‘Is Results-Based Aid More Effective than Conventional Aid?’ by reviewing a natural experiment in the health sector in El Salvador where municipalities were funded through different mechanisms, including Results-Based Aid. The authors found that compared to conventional funding, financing that involved results-based conditionality roughly doubled aid effectiveness and conclude that “The effects appear to have been driven by a more rapid expansion of health infrastructure and qualified personnel by motivated national authorities”.

Coming up: ‘Outcomes-based commissioning: learning from Payment by Results and Social Impact Bonds’. On March 27th, Professor Chris Fox, Director of Policy Evaluation and Research Unit, at Manchester Metropolitan University, will share lessons learned from the UK’s use of Payment by Results and Social Impact Bonds since 2010.

From PERU’s website, you can take a look at earlier work and presentations by Professor Fox and colleagues on this topic, including a 2017 paper on evaluating outcome-based payment programmes.

Cheryl Brown, Communications Manager, WASH Results MVE Team.

 

 

 

WASH Results’ evaluation highlights differences in design

The WASH Results Evaluation Team (led by Oxford Policy Management) shared their midline evaluation findings with the programme’s Suppliers through a webinar earlier this year and the slides are now available online together with the full report.

The WASH Results Midline Evaluation was carried out mid-2016 and explored:

• The relevance of programme design (theory of change).
• To what extent the verification systems were fit-for purpose (relevance and efficiency).
• In what ways the Payment by Results (PBR) modality affected implementation.
• Which outputs were achieved (those paid for, and those outside the PBR framework).

The evaluation showcases how the three Suppliers designed their PBR systems in different ways, to suit the different WASH activities being implemented across 12 countries between them. As a result, a tailored verification system was also implemented by its Third Party Verifier (led by Itad) to fit this variation. This diversity is summarised nicely in the slides that accompanied the webinar.

The planned output targets of the programme were achieved by December 2015 – with significant over-achievement (though over-achievement was not paid for under the PBR modality). They were achieved off the back of results-oriented problem-solving by the Suppliers who adjusted activities to achieve agreed results. The increased scrutiny of the third-party verification mechanism also helped strengthen monitoring systems of the Suppliers, resulting in more regular and higher-quality monitoring data. However, the evaluators also found that the pressure of achieving results put significant burden on programme staff and partners.

Suppliers managed the risks of PBR in a variety of ways, including front-loading payments and moving output targets between countries if one was under-achieving. With a few exceptions, the Suppliers also opted to pre-finance their local implementation partners and thus hold the PBR risks at the headquarters level, instead of passing them down to local implementation partners.

The midline evaluation webinar concluded with a suite of recommendations for PBR suppliers (NGOs), for DFID and for independent verifiers. The endline evaluation will be carried out in 2018 and will focus on outcome achievement, impact and sustainability aspects.

If you would like to be notified when the WASH Results Endline Evaluation Report is available, please email Cheryl Brown, WASH Results MVE Team Communications Manager.  You may also be interested in the Evaluation Team’s Short Reading List on PBR.

 

 

 

Reflections on our experience of DFID’s results agenda

As verifiers of a DFID Results Based Finance programme, ODI’s research on the UK’s results agenda prompted us to reflect on our experience.

 

Why context matters when you focus on results #1: Some latrine building projects have to allow for the impact of hungry goats. Photo credit: Chamia Mutuku

 

In their report ‘The Politics of the Results Agenda in DFID: 1997 to 2017’, Craig Valters and Brendan Whitty argue that 2007 saw a new explicit focus from DFID on aggressively implementing results-based management. 10 years later, we have WASH Results: a DFID-funded programme where financial risk has been completely transferred away from UK taxpayers to the international NGOs who deliver the work and who only get paid for results that have been checked by a third party – us. However, as its name promises, the programme is delivering results in water, sanitation and hygiene (WASH). DFID was able to read in the programme’s 2017 annual report (with great confidence in the figures), for example, that WASH Results had reached over 1.1 million people with improved water supply, more than 4.7 million people with improved sanitation, and over 14.9 million people with hygiene promotion.

In our role as Monitoring, Verification and Evaluation Team for the WASH Results programme, our attention is less focused on the politics of the results agenda, and more in how results are monitored and verified and the very real impact that this approach has on ongoing programme delivery. However, we read the report and blog post by Valters and Whitty with great interest.

After more than three years of supporting the programme, how does our experience compare with the conclusions and recommendations of the ODI report? One key finding from the research is that some DFID staff have found ways to adhere to the results agenda, while retaining flexibility. This theme of the ways in which both donors and programme implementors are working creatively around the “tyranny of results” was one that we heard during last year’s BOND Conference session ‘How to crack Results 2.0’.

How can PBR be adapted to address the imbalance in accountability?

We absolutely agree with Valters and Whitty about the importance of finding a balance between being accountable to UK citizens and to the beneficiaries (poor people abroad). This time last year, we shared our opinion that if verification was designed to include beneficiary feedback and this was linked to payment, Payment by Results (PBR) could actually generate more downward accountability than other funding modalities. However, our team of verifiers felt that the demands of verification for large scale, representative, quantitative information on which to base payment decisions may leave less time, money and inclination to undertake more qualitative work with beneficiaries. So, we suggested that a resource-effective solution to upholding downwards accountability through verification would be to include payment for the existence and effective functioning of a beneficiary feedback system (rather than the results of that system). Payment would be made on verification of the effectiveness of the system in promoting downwards accountability.

We welcome the authors’ call to “Create a results agenda fit for purpose”. Our first reflection would be that a results agenda, at least one hard-wired into a PBR modality, is not going to be appropriate in all contexts and for all intended outcomes, particularly those where outcomes are difficult to predict or challenging to measure. Our set of recommendations to commissioners of PBR programmes, complement several of those made by ODI, for example, their suggestion that DFID spend more time considering whether its aid spending has the right mix of risks and the view that regular testing (that leads to course-correction) is important.

The challenge of communicating about costs and value

The authors also call on ministers to be honest with the British public about aid. Part of this, we feel, is making it clearer that Value for Money (VFM) is not synonymous with “cheap”. We feel that the results agenda, particularly a PBR model, should require donors/commissioners to clearly articulate the “value” they expect to see in VFM. Otherwise the importance placed by a donor on achieving clearly costed, verified results could risk squeezing out other values and principles that are central to development programming. A central theme in last year’s WASH Results learning workshop was the ongoing puzzle of how to place value (both in commercial/financial and VFM terms) on intangible aspirations and benefits, such as reaching the most vulnerable and investing in the processes and social capital that underpin effective programming. This is particularly important in an increasingly commercialised aid context, where one supplier’s approach would be to parachute in and build many toilets very quickly and cheaply, whereas another proposes taking longer to work with local stakeholders, norms and materials. This articulation of value may not be as simple as it sounds, when every commitment in a PBR programme, such as reaching the poorest, gender equity, national ownership, sustainable outcomes, etc. needs to be reflected in meaningful and measurable indicators.

Payment By Results can aid course correction

Interestingly, one of the reforms that the authors call for may be an inherent feature of the results framework itself. They say that “interventions need to be based on the best available information, with regular testing to see if they are on the right track”. We have found that a product of the PBR modality is that much greater emphasis is placed on monitoring systems and the generation of reliable data about what is happening within programmes. In WASH Results we have seen cases where the rigorous (compulsive?) tracking of results has identified areas where programmes are failing to deliver and rapid action has then been taken to address that failure. As verification agents we argue that this is due not only to the link between results and payment but also the independent verification of data and systems that has led to better information on which to base decision-making.

Benefits of the results agenda

In this way we think that the focus on monitoring within the results agenda, can, in some cases, enable flexibility and innovation. In its reliance on high quality data, it contains within it a driver that could improve the way that development work happens. The results agenda brings benefits – some of which we did not see reflected in the article – but it comes with risks; both ideological about the ambitions for UK Aid and practical for those involved in its delivery. And so we welcome this debate.

Catherine Fisher, Learning Advisor, WASH Results MVE Team

If you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us

Why is it so hard to measure handwashing with soap?

As Global Handwashing Day approaches, the WASH Results MV team explores the challenges of verifying changes in handwashing behaviour.

Hygiene has tended to be the Cinderella of WASH (Water, Sanitation & Hygiene): relegated to the end of the acronym and with hygiene promotion often having been treated as an afterthought in larger water and sanitation programmes. But good hygiene is recognised as an essential component in addressing the burden of disease, and hygiene promotion can be a cost-effective way of improving health outcomes. In linking payment milestones to sustained handwashing behaviour the WASH Results Programme is breaking new ground, and pushing against some of the limits of understanding in the sector. The programme has faced key challenges in how to establish what reasonable handwashing outcomes might look like, and the most appropriate way to measure them. This post draws on discussions within the Monitoring and Verification (MV) team on some of these issues, and resulting internal briefings.

What is a good handwashing outcome?

Across the WASH sector there is little established knowledge on the sustainability of WASH outcomes. Whilst the well-known (and well-resourced) SuperAmma campaign saw a sustained increase in handwashing practice of 28% after 12 months, it’s still an open question as to what level of behaviour change it might be reasonable to expect across differing countries, contexts and projects. This matters, because in a payment by results (PBR) context, payment is linked to achieving targets. Set targets too high and suppliers risk being penalised for failing to reach unrealistic expectations. Set targets too low and donors risk overpaying.

How do we measure handwashing?

Compounding the uncertainty over what reasonable targets may be, is uncertainty on how to measure achievement against those targets. There are several commonly accepted methodologies, but they all involve compromises. At one end of the scale there is structured observation of household handwashing behaviour. Considered the gold standard in measuring behaviour change, this can provide a wealth of data on the handwashing practices of individuals. But it’s also prohibitive to undertake at scale – the most rigorous designs can involve hundreds of enumerators making simultaneous observations. The feasibility of conducting such measurements regularly for a PBR programme is questionable.

A much simpler (and on the face of it, more objective) indicator might be the presence of handwashing facilities (a water container and soap). This is used by the Joint Monitoring Programme (JMP) to measure progress against SDG 6.2 because it is the most feasible proxy indicator to include in large, nationally-representative household surveys which form the basis for the Sustainable Development Goal monitoring (and typically collect data on WASH as just one of several health topics). However, these observations tell us nothing about individuals’ handwashing behaviour, or if the facilities are even being used.

Setting indicators for handwashing outcomes

Within the WASH Results Programme, the suppliers have tended to define handwashing outputs in terms of the reach of handwashing promotion (though evidencing how a programme has reached hundreds of thousands or even millions of beneficiaries leads to its own challenges). They have defined outcomes in terms of:

• knowledge of handwashing behaviour,
• self-reported behaviour, and
• the presence of handwashing facilities.

The suppliers have taken different approaches to considering these indicators separately or as part of a composite indicator to attempt to reach a reasonable conclusion on what the sustained change in handwashing practice has been. Each of the indicators has drawbacks (self-reporting, for example, has been shown to consistently over-estimate handwashing behaviour), but by considering all of them, a more reliable estimate may be reached.

Where this process has become particularly challenging for the WASH Results programme, is in attempting to measure outcomes at scale and across highly diverse contexts. All too often the devil is in the detail and small details in how indicators are set and measured could lead to huge variations in results, in turn leading to major differences in what suppliers get paid for. For example, we may wish to define a standard for the presence of a handwashing facility as a water container and soap, but very quickly some of the following issues arise:

• In some countries, ash is commonly used instead of soap: national governments may actively promote the use of ash, and in some areas, it’s almost impossible to even find soap to buy. But at the same time there is evidence that using ash (or soil, or sand) is less effective than using soap: and for this reason, handwashing with soap is the indicator for basic handwashing facilities for SDG 6.2. Should payment targets follow national norms, or would this be a case of paying for lower-quality outcomes?
• In other areas households choose not to store soap outside because it can be eaten by livestock or taken by crows (a far more common problem than one might imagine). Do we adopt a strict definition and record that there is no soap (so hence no handwashing facility) or is it acceptable if a household can quickly access soap from within a building? Do we need to see evidence that this soap has been used?
• Local practice may mean that handwashing facilities do not have water in-situ, instead people carry a jug of water to the latrine from a kitchen. Recording a handwashing facility only if there is water present may risk a significant underestimate of handwashing practice, but how can we determine if people actually do carry water? And does this practice adversely impact on marginalised groups such as the elderly or people living with disabilities?

And this is just for one indicator. Across all three (knowledge of handwashing behaviour, self-reported behaviour, and the presence of handwashing facilities) the complexity of the methodology for measuring the results can quite quickly become unwieldy. There are tensions between wanting to adopt the most rigorous assessment of handwashing possible (to be more certain that the results do reflect changed behaviour), what data it is feasible to collect, and what it is reasonable for the suppliers to achieve. The answers to these will depend on what the programme is trying to achieve, the results of baseline surveys, and the costs of measurement and verification of results.

There may not be an easy answer to the question of how to measure handwashing outcomes, but the experience of the WASH Results Programme suppliers has provided useful learning on some of the aspects that need to be thought through. As the programme progresses, the suppliers are continuing to refine their understanding of how best to approach this issue in the countries and contexts they work in, and iterate the indicators they use. What’s next? Well, exploring how the various handwashing indicators relate to improved public health could be interesting…

What can we learn from the Government Outcomes Lab?

Our Learning Advisor, Catherine Fisher, investigates a new research centre on outcomes-based commissioning and finds plenty of interest for the WASH Results Programme.

You know an idea has traction when the University of Oxford sets up a research centre to investigate it. This is the case for outcomes-based commissioning (aka Payment by Results), which is the focus of the new Government Outcomes Lab (GO Lab) based at the University of Oxford’s Blavatnik School of Government.

The GO Lab focusses on the interests and experience of government departments in procuring services in an outcomes-based way, rather than those of contractors (or suppliers, in WASH Results terminology) in providing them. It is a collaboration between Blavatnik School of Government and Her Majesty’s Government. The focus of the research centre is on outcomes-based commissioning that uses Social Impact Bonds (SIBs), a model in which external investors provide the initial investment for programme implementation which is repaid on achievement of outcomes. However, the GO Lab will also look at other models, presumably including that used in the WASH Results Programme in which the suppliers themselves provide the upfront investment.

The rationale for the GO Lab is as follows: “While numbers of, and funding for, outcomes-based approaches have increased over recent years, research has not kept pace with this speed of growth. Much is still unknown about whether outcomes-based commissioning will meet its promise….Through rigorous academic research, the GO Lab will deepen the understanding of outcomes-based commissioning and provide independent support, data and evidence on what works, and what doesn’t.”  (GO Lab FAQ) .

So far, the GO Lab has organised three “Better Commissioning” events looking at outcomes-based commissioning in different sectors in a UK context, namely Children’s Services, Older People’s Services and promotion of Healthy Lives.

A quick skim of the interesting post-event reports suggests that outcomes-based commissioning is seen as a way of promoting a greater focus on outcomes by providers (who may not already think in this way), of prompting innovation in service provision and of transferring the risk of new approaches from commissioners to socially-minded private enterprises. Similar themes occur in Results Based Aid discussions, although I’d suggest that the international development sector places a slightly greater emphasis on incentivising delivery, value for money and accountability to commissioners.

One aspect of the GO Lab work that caught my eye is their interest in the creation of Outcomes Frameworks which were discussed at each event: “Developing an outcomes framework is a key part of any SIB or outcome based contract, but accessing data and articulating robust metrics that can be rigorously defined and measured is often seen as a challenge by commissioning authorities.”  (Better Commissioning for Healthy Lives: a Summary Report, p 13).

This process of articulating appropriate metrics and identifying indicators has been a key area of learning within the WASH Results Programme and continues to be discussed. It was reassuring to see others grappling with similar challenges in different sectors, such as challenges of creating indicators that:

• are both unambiguous enough to trigger payment and reflective of the outcomes the programme is intended to achieve (Children’s Services Workshop Report, p. 5);
• reflect the context and needs of beneficiaries and are also able be aggregated at a higher level (and internationally in the case of the WASH Results Programme) (Healthy Lives Symposium Report, p.14  and Services for Older People Symposium Report, p.14 );
• reflect the sustainability of outcomes beyond the term of the contract.

During this, the outcomes-focused period of the WASH Results Programme, we will be following the progress of the GO Lab with interest, and hope to find opportunities to exchange learning with them, and others researching innovative funding approaches. Our team is particularly interested in contributing to, and benefiting from, learning around:

• independent monitoring and verification of outcomes-based contracts;
• creating outcomes frameworks that reflect sustained outcomes in areas such as behaviour change (e.g. handwashing behaviours) and institutional change (e.g. ability of district stakeholders to manage water systems);
• streamlining metrics and indicators while balancing needs of all parties: beneficiaries, service providers, commissioners and, in the case of international development programmes, national stakeholders and global commitments such as Sustainable Development Goals.

Catherine Fisher, Learning Advisor, WASH Results MVE Team

If you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us

Measuring progress towards SDGs: a Payment by Results perspective

Attending the 2017 WEDC Conference prompted our team members to share their reflections on measuring progress towards SDGs from a Payment by Results (PBR) perspective.

Some of the e-Pact Monitoring and Verification (MV) team recently attended the WEDC Conference – an annual international event focused on water, sanitation and hygiene (WASH), organised by the Water, Engineering and Development Centre (WEDC) at Loughborough University. One of the key themes this year was the Sustainable Development Goals (SDGs): what they are, how close we are to achieving them, and how we are going to monitor them. The SDGs are important for PBR programmes because they influence what programmes aspire to achieve and how they measure their progress.

The recent publication of the first report (and effective baseline) on SDG 6, covering drinking water, sanitation and hygiene, marked a watershed. With the shift to understanding universal and equitable access, the inclusion of hygiene and a focus on ‘safely managed’ and ‘affordable access’, the breadth and depth of data we aspire to have on water and sanitation services is unprecedented. But the first SDG progress report also highlights a yawning data gap: for example, estimates for safely managed drinking water are only available for one third of the global population, and we are only starting to get to grips with how to measure affordable levels of service.

As part of the WASH Results Programme, the three consortia are constantly grappling with how to objectively measure complex outputs and outcomes linked to water, sanitation and hygiene. At the same time our MV team is trying to understand how we can verify such measures, and if they are sufficiently robust to make payments against. How do the SDGs influence this process? We have three reflections from our experience of verifying results under the WASH Results Programme:

Reflection 1: the relationship between the SDGs and PBR-programming can be mutually beneficial.

Embed from Getty Images

The SDGs help PBR programmes to set ambitious benchmarks

It’s clear that to track progress against the SDGs, the WASH sector is going to have to become a lot better at collecting, managing and analysing an awful lot of data. One of the learning points from the WASH Results Programme is that the verification process requires the consortia (and in-country partners) to take data far more seriously.

Compared to more conventional grant programmes, Monitoring and Verification functions take on the importance of financial reporting. One function of this, is that everyone has more confidence that reported results (whether access to water, number of latrines built or handwashing behaviour) accurately reflect reality. As such, PBR programmes can help focusing peoples’ attention on improving service levels.

Conversely, the SDGs help PBR programmes to set ambitious benchmarks and provide an orientation on how to measure them. This is proving important under the WASH Results Programme, which has, at times, struggled with aligning definitions, targets, indicators and how to measure them.

Reflection 2: some of the SDG targets are hard to incorporate into a PBR programme
Embed from Getty ImagesPhysical evidence of a handwashing facility doesn’t guarantee use at critical times

Measuring hygiene behaviour change illustrates this point neatly: the simplest way to understand if people are washing their hand with soap may appear to be just to go out and ask them. Yet self-reported behaviour indicators are notoriously unreliable. Looking for physical evidence of a handwashing facility (with water and soap) is the approach currently suggested by the WHO/UNICEF Joint Monitoring Program (JMP), but there is no guarantee that people use such facilities at the most critical times, for example, after defecation or before handling food.

Under a PBR programme (where implementers get paid against pre-defined results) the temptation to take the shortest route to success, namely focusing on getting the hardware in place, may be high. Therefore, it may be important to complement this indicator with a knowledge-related indicator to also capture behaviour change albeit in a crude way. This brings along another challenge: how to agree on appropriate, payment-related targets in a situation where experience on how to accurately measure behaviour change is still in its infancy?

Reflection 3: keeping indicators lean is challenging when faced with the breadth and depth of the SDGs

Hygiene behaviour change is just one indicator. Attempting to robustly measure changes across three consortia, eight result areas and two phases (output and outcome) has resulted in the MV team reviewing a large amount of surveys, databases, and supporting evidence since 2014.

Under the WASH Results programme, the sustainability of services is incentivised via payment against outcomes: people continuing to access water and sanitation facilities and handwashing stations for up to two years after they gained access to improved services. In the meantime, between the final MDG report, and the initial SDG report, the number of data sources used by JMP to produce estimates for the water, sanitation and hygiene estimates has more than doubled[1]. Instead of more traditional household surveys, increasingly, data is obtained from administrative sources such as utilities, regulators and governments.

How to marry these new data ambitions with the necessary goal to keep the number of indicators manageable under a PBR programme will be an interesting challenge going forward.

Katharina Welle and Ben Harris, MV Team, WASH Results

[1] Progress on drinking water, sanitation and hygiene: 2017 update and SDG baselines (p50) https://washdata.org/reports 

Recent views and news on Payment By Results

New funding for learning, a new framework and a free webinar. In this post we highlight what’s happening online about Payment By Results

 

One of the challenges that the World Bank acknowledges it and its clients face, is “Choosing the incentive mechanism that best fits an intervention in a particular context and obtains the greatest impact”.  To that end, the World Bank has issued a call for proposals to learn more about the impact results-based financing can have on education systems at a district/regional level. One criticism often levelled at Results Based Finance (RBF) is that it incentivises “cherry-picking”, leaving the harder-to-reach behind in pursuit of easier ways to reach beneficiary targets. The World Bank is therefore requiring those submitting proposals to be able to measure the equity impact of RBF.

Could an “Expert Body” help overcome some of the problems experienced by past Payment By Results (PbR) programmes? Dr Laura Johnstone has just shared a framework for commissioning and implementing PbR programmes which she describes as “dynamic and applicable locally, regionally and nationally to all PbR provision across the public sector wherever target achievement and the provision of value to stakeholders is required”.  A key element of the framework is the establishment of an Expert Body that comprises key stakeholders for the programme “including the commissioner, the principal, the agent, the service users and academics with experience of PbR and the specific need”. The Expert Body plays the roles of governing body at the commissioning stage, and data and learning gatherer during implementation, to inform adjustments to delivery. Dr Johnstone invites feedback on the framework via the blog post or direct email.

Finally, even as this post goes live, a World Bank webinar is scheduled to discuss the findings of a study of results-based climate finance activities. The recording of ‘Achieving transformational change in climate change mitigation through Results-based financing‘ can be accessed after June 6th, from the World Bank’s Open Learning Campus.

Cheryl Brown, Communications Manager for WASH Results MVE

A short reading list on Payment By Results, from our Evaluation Team

Lucrezia Tincani (Evaluation Team for WASH Results) shares a compilation of guides and best practice reviews on Payment By Results (PBR).

PBR within the Water, Sanitation & Hygiene (WASH) sector

1. This review for the Gates Foundation (2015) examines 30 PBR WASH programmes, predominantly implemented by the World Bank, all relatively small-scale. It outlines the minimum conditions that should be in place before choosing a PBR design. It suggests that PBR is better suited to private than to public providers, and works well in developing countries with low government capacity. However, it concludes that there is insufficient evidence to determine whether PBR WASH programmes are more efficient or more sustainable than non-PBR WASH programmes, also cautioning that there is limited evidence that PBR is effective at achieving behaviour change in community sanitation projects.
2. Output-Based Aid is a form of PBR where aid is disbursed to governments who in turn contract a public service provider. These World Bank reports – (2010) and (2014) – outline the World Bank’s experiences of the Global Partnership on Output-Based Aid (GPOBA) with sanitation and water, respectively. The reports outline the preconditions necessary before choosing PBR approaches and outline case studies of the World Bank’s work.

PBR more broadly

1. This paper by the Center for Global Development (2015) presents four major theories explaining the rationale for using PBR interventions and then explores which ones hold, through four case studies of Results-Based Aid in education and health, which were implemented alongside conventional aid programmes. It provides useful background on the different views regarding how and why PBR can act, and provides rich insights of how PBR has operated in the four case studies.
2. The World Bank’s 330-page toolkit on performance-based financing (2014) contains a wealth of knowledge on design of PBR schemes, including selection of indicators and design of the verification process. Their GPOBA team also produced a useful overview note on approaches to independent verification of PBR programmes (2012). Though aimed at PBR schemes for health facilities, the lessons in the toolkit are generalisable across other sectors. The World Bank also produced a guide specifically for PBR in the energy sector (2013).
3. This note (2014) by Clist & Dercon proposes 12 principles which should be considered when designing a PBR programme. It is relevant both for the implementers and evaluators of PBR programmes. It is a summary of their theoretical paper (2014) which outlines the likely costs and benefits of PBR programmes according to behavioural economics theory.
4. This Cochrane Review of PBR programmes in health (2012) reviewed nine health PBR programmes and concluded that they were not uniform enough to be able to determine whether PBR programmes are more effective at improving health care than non-PBR ones. The review calls for more rigorous evaluations of PBR programmes, particularly to assess whether they are cost-effective. The German Development Institute (2013) also reviewed Results Based Financing experiences in the health sector.
5. A DFID review (2013) synthesises 33 evaluations of PBR programmes: mostly health, with some education and one WASH programme (in Indonesia). It concludes that the added-value of PBR is, as of yet, unproven, highlighting that many evaluations have weak ‘external validity’, meaning that their conclusion are not generalisable to other contexts. This reinforces the premise that the success of PBR programmes is highly context specific. This report for NORAD (2008) came to a similar conclusion, as did this DFID review (2010) of Results Based Aid and Results Based Financing schemes, which also provides a useful overview of different types of PBR.
6. A 2012 review by the European Network on Debt and Development (a network of 47 European NGOs) synthesises the experiences across six PBR multi-country initiatives (including GPOBA) and analyses how the amount of PBR financing has increased over time.
7. MANGO (a UK NGO network) produced a report (2015) specifically highlighting the contractual risks of PBR contracts.
8. This much-cited blog by Robert Chambers (2014) also highlights some of the risks of implementing PBR programmes.
9. A SIDA review (2015) outlines the factors to be considered when designing a PBR programme, and outlines SIDA’s experience with different types of PBR so far (albeit limited). Similarly, this DFID ‘Smart Guide’ (2014) gives a brief overview of the factors to be considered when designing a PBR programme, and provides a list of all DFID PBR programmes operating at that time.
10. DFID’s PBR evaluation framework (2015) provides useful guidance for those designing an evaluation of a PBR programme, including a detailed set of possible evaluation questions. These are also important for implementers when considering whether or not to implement a PBR programme.
11. A review by the UK National Audit Office (2015) cautions that PBR schemes are “hard to get right, and are risky and costly for commissioners”. Thus, most UK PBR programmes only link 20% of payments to performance. The review covers work done by the whole of the British Government, not just by DFID.

The websites of the Suppliers implementing the WASH Results Programme (SWIFT, SNV and SAWRP) also provide useful insight into the reality of implementing a PBR programme.

Given the absence of clear evidence about what works under different contexts, this reading list can never truly be complete or offer simple answers to questions about how best to implement PBR programmes. A key challenge is isolating the success or failure of the PBR modality itself from the success or failure of the technical intervention, which can be affected by external factors unrelated to PBR. Thus, there is still little evidence of the added benefits of PBR, because an accurate counterfactual cannot easily be established (rarely is a non-PBR programme with almost identical activities implemented alongside the PBR programme).

Our Evaluation Team would be glad to receive your suggestions of resources to add to this list, either through the comment section below or you can tweet them to us via @WASHResultsMVE

Lucrezia Tincani (OPM) leads the evaluation of the WASH Results Programme.

‘Getting Results: Independent Verification in Output-Based Aid’

In this post we compare the Output-Based Aid (OBA) experience of independent verification with that of the WASH Results Programme.

Earlier this year the Global Partnership on Output-Based Aid (GPOBA) produced a short paper called ‘Getting Results: Independent Verification in Output-Based Aid‘. Although the paper focusses on Output-Based Aid, while our work as independent verification agents on the WASH Results Programme also encompasses verification of outcomes, many of the findings were relevant to our work.

We agree about the fundamental importance of getting the right indicators, that they match the intended purpose of the programme and are clear and unambiguous. And we too recognise the tricky balance between optimum frequency of verification rounds and the need for suppliers to have regular cash-flow. One of our previous blog posts shares some of our learning in this area.

The most thought-provoking aspect of the paper was the case studies describing innovative approaches to verification.

Building national government capacity for verification
The first case study explores the World Bank’s Local Government and Decentralization Project in Indonesia— a pilot approach in which government auditors have been supported to take on the independent verification role. Capacity building and financial support are provided through a grant, presumably outside the OBA funding mechanism.  This need for strengthening government capacity in monitoring results is something we touch on in a previous post about alignment under Payment by Results in which we mention DFID’s work in this area in Tanzania. It’s good to see more examples of it happening.

The verification agent’s role in baseline surveys is controversial in our programme
The second set of examples talk positively of the role of verification agents in undertaking baseline surveys and how this can help project design. Our experience of this differs. At a learning event, earlier this year, the WASH Results Suppliers argued strongly that they wanted to undertake data collection themselves. Some of the arguments given included:

• Collecting reliable and meaningful data requires a strong understanding of the realities and dynamics of the field.
• Data collection requires relationships with local implementers who are key to providing information needed for verification. This requires a lot of goodwill and coordination, which would be unlikely if an independent third party monitor was used.
• Keeping data collection in-house avoids the huge risk inherent in de-linking the supplier (responsible for achieving the results, and getting paid for them) from the monitoring of the results.

On the other hand, the WASH Results Monitoring and Verification (MV) team has sometimes raised concerns about whether baseline surveys conducted by suppliers are rigorous enough for verification purposes. One way forward seems to be for MV teams to approve the design of baseline surveys before they are carried out and ideally for this to be hard-wired into programme design. For more about this discussion, see the report from the learning event.

Technological innovation in monitoring is good but getting paid is vital
The final box in the GPOBA note talks about technological innovation in verification. It shares some interesting examples such as low cost sensors and combining use of images and GPS. We too have found that the demands of verification have prompted technological innovation in monitoring, in particular using mobile-based technology. However, because the WASH Results Programme relies primarily on supplier-generated data, most of the innovation in this area has been undertaken by the suppliers.

Again, what is striking about the innovation in monitoring described in the paper is that it appears to have been funded outside the OBA funding mechanism e.g. through an Innovation Grant. Presumably, these methods were tested before being implemented in an OBA context. This resonates with a comment made during a debate within WASH Results about creative ways of monitoring outcomes “Do we want to be creative, or do we want to get paid?”. At its heart, verification is about finding the most efficient way of providing confidence to donors that suppliers have done what they have said in order that they can be paid. Creativity is good, but in a 100% PBR context in which suppliers pre-finance all their activities, getting paid is better.

Finally, it is worth noting that the paper does not reflect on the impacts (intended or otherwise) and value of independent verification of OBA, that we have been exploring.  It would be a much longer paper if it did!

Catherine Fisher, Learning Advisor, WASH Results MVE Team

As always, if you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us.

Leave a comment

Posted in Learning from WASH Results

Tagged aid effectiveness, alignment, government, Payment By Results, Results Based Aid, Results Based Finance

Edit

Measuring WASH sustainability in a payment by results context

Posted on August 1, 2016 | 4 comments

Andy Robinson, Lead Verifier for SNV, reports back from a WEDC Conference panel session organised by the three WASH Results suppliers

The three suppliers in the DFID WASH Results programme (SAWRP consortium, SWIFTconsortium and SNV) came together at a side event held during the WEDC Conferencein Kumasi, Ghana (11-15 July 2016) to present their thoughts on “measuring WASH sustainability in a Payment by Results (PBR) context”.

As Lead Verifier on the SNV contract and WEDC conference participant, I was invited to join the panel with the three suppliers and make a short presentation on behalf of the e-Pact consortium – to explain e-Pact’s role in WASH Results and elaborate some of the initial learning from the perspective of our Monitoring and Verification (MV) team.

Kevin Sansom (WEDC, SAWRP) began by outlining the key differences between PBR and grant programmes. He noted that PBR programmes require significant pre-finance and engender higher risks (particularly when tight timelines are applied), but allow greater flexibility and encourage more rigorous monitoring and evaluation (both internally, within the implementing agencies and externally by the verification and evaluation teams).

SAWRP presentation

Mimi Coultas (Plan UK) detailed the sustainability monitoring system adopted by the SAWRP consortium, explaining that some of the elements (sustainability assessment frameworks, outcome implementation manuals and the learning framework) are not linked to payments, but are designed to meet DFID’s requirement for reporting against five different dimensions of sustainability (functional, institutional, financial, environmental and equity).

Mimi noted that there was a lack of clarity at the outset around the criteria for payment (and the criteria for disallowance of payments), which caused some problems and could have been avoided by agreeing these details during a longer inception phase. She also suggested that the sampling approach used by the MV team has the potential “to scale mistakes” by exaggerating the effect of any poor results included in the sample (to suggest problems larger than actuality). Another comment was that the commercial pressures on the suppliers, all of whom are interested in bidding for any follow-on programmes, might have reduced collaboration and sharing of lessons learned.

Nonetheless, the SAWRP consortium felt that the programme had produced “amazing results”, with a high level of confidence in the quality and reliability of the results due to the strong scrutiny provided by the MV team. Mimi also noted that the monitoring and evaluation (M&E) focus required by the programme was a positive outcome, leading to a strengthening of M&E systems and the development of better ways of measuring WASH outcomes and sustainability. However, a longer programme duration would have been better, including an inception period during which the results framework and verification approaches could be carefully designed and negotiated.

SNV presentation

Anne Mutta (SNV) talked about the critical importance of political engagement to WASH sustainability, with governance activities integrated into the SNV programme from the start to address this requirement. Where local government capacity for sanitation and hygiene is low, sustainable results will obviously be harder to achieve. She also noted that some practical sustainability problems arise, such as heavy rain and flooding (which can wash away sanitation facilities, and constrain implementation) and changes in capacity, knowledge and commitment due to issues like government transfers or elections. Anne also agreed that the PBR programme required stronger progress monitoring, to track results and allow course corrections before the household survey results are verified.

SWIFT presentation

Rachel Stevens (TEARFUND) explained that the SWIFT consortium is using household, water point and latrine surveys, as well as local government and local service provider data, to assess sustainability (with two sets of surveys planned – one in mid-2016 and the other at end-2017). The SWIFT sustainability assessments use a similar traffic light system to those described by the other two suppliers, reporting against DFID’s five dimensions of sustainability.

Common challenges

The three suppliers had agreed on a list of common challenges, which were presented by Mimi Coultas (Plan UK). One of the most interesting of these was the risk that PBR encourages implementation in easier contexts – through the selection of less vulnerable and more accessible communities and project areas – in order to reduce both cost and risk.

The suppliers also questioned whether verification was appropriate for all aspects of sustainability, particularly the intangible and more qualitative factors (such as community empowerment), which are often important elements associated with the sustainability of sanitation and hygiene practices and outcomes.

Another potential issue is that the reduced reporting burden, with the production of evidence of results generally replacing the need for the detailed progress reporting and evaluation required by conventional programmes, may mean that the lessons learned by the programmes are not well captured or adequately documented.

Common opportunities

The suppliers agreed that, while some aspects of sustainability may be missed, the inclusion of payments for specific sustainability outcomes led to more attention to sustainability than in conventional programmes. Furthermore, the MV team’s work had encouraged greater transparency and accountability.

MV presentation

I made a short presentation on the role of the MV team and the key challenges and opportunities. After describing the composition of the e-Pact team, and introducing Bertha Darteh (Ghana country verifier for the SNV programme, who was in the audience), I explained that we were using “systems-based verification” rather than fully independent verification, which means that we are reliant on the data and reports produced by the suppliers’ M&E systems. As a result, we have to understand these systems well, and identify any weaknesses and any potential for errors, misreporting or gaming of results. DFID’s decision to adopt a systems-based verification approach was based on the assumption it would be cheaper than statistically sampled independent surveys (across such a large population), but the MV experience suggests that there are a lot of unforeseen costs (often to the suppliers) related to this systems-based approach.

Key verification challenges include the large number of closely spaced results, with little time between each verification cycle for the design, review and improvement of the verification process. The SNV programme includes nine country projects, with significant variations in context across the projects, which requires considerable flexibility in the verification system; whereas the other two suppliers’ programmes include multiple implementation partners, each of which has slightly different monitoring and reporting systems, and different priorities and targets, which in turn require adaptation of the verification systems.

I concurred that not enough time had been provided up front for the planning and design of the programme, including the MV framework and activities, which increased the pressure on all stakeholders during the first year of the programme, when suppliers were developing systems, implementing and reporting, with little time to respond to the additional demands of the verification process.

One positive outcome of the need for verified results has been the use of smartphone survey applications, which have greatly sped up and reduced the cost of the survey process; improved data processing and quality control; and made it much easier to verify large-scale results quickly. A key learning from the PBR programme is that these household surveys appear to be a far quicker and more effective way of evaluating programme outcomes than conventional evaluations.

Overall, the PBR approach appears to be improving M&E approaches and systems, encouraging more thinking about how to measure and evidence outcomes and sustainability, and providing reliable feedback on progress and performance at regular intervals during the life of the programme. This feedback enables regular improvements to be made to programme policy, planning and practice (unlike conventional programmes, which often are not rigorously evaluated until the end of the programme duration).

Questions from the floor

When the panel was asked whether the PBR approach encourages efficiency, the suppliers noted that both the programme and the approach encourage scale, which in turn encourages efficiency; however, the additional costs of verification and the related reporting were thought to partially offset the efficiency gains.

A similar question was asked about whether PBR encouraged value-for-money: the suppliers suggested that they are very confident of their results (compared to conventional programmes, which may over-report results), thus the cost-per-result is clear. They also noted that there is an incentive to reduce costs, but that these reductions may not always be passed on (and, because there is no payment for over-achievement in this programme, any additional results appear to reduce the cost per outcome/result, but do not change the suppliers’ fixed costs).

Several Ghanaian participants expressed their confusion about the new terminology associated with PBR. Output based Aid (OBA) is common in Ghana, notably through a World Bank WASH programme (with payments linked to toilet construction), and it was suggested that there “was no need to introduce yet another acronym for the same thing”. Louise Medland (WEDC, SAWRP) responded that DFID differentiated between the OBA and PBR approaches by the PBR focus on outcomes (whereas OBA focuses on outputs).

The final question was around PBR’s effect on innovation: the suppliers noted that the design was supposed to encourage innovation, but that the time pressure (of the short implementation period) limited the chance of innovation. I added that we have seen different outcomes in different contexts – in low capacity settings, the programme management generally provide firm guidelines to the project team to minimise risk; but in high capacity settings, there was evidence of innovation driven by the need to achieve results, especially in more difficult contexts where standard approaches were not working.

The general tone of the PBR session was positive, with the suppliers agreeing that the PBR approach has led to reliable and large-scale results, and that the need to report and verify results has led to significant improvements in M&E systems. A lot of learning has taken place, and the suppliers hoped that this learning will inform the design of any future WASH PBR programmes.

Andy Robinson, Lead Verifier on the SNV Contract, WASH Results MVE Team

4 Comments

Posted in Meeting reports

Tagged challenges, measurement, Monitoring & Evaluation, outcomes, outputs, Payment By Results, sustainability

Edit

The paybacks and pains of Payment by Results (Part 2)

Posted on July 25, 2016 | 2 comments

Our series of reflections on WASH Results’ learning continues by exploring value and costs in a Payment by Results (PBR) programme.

DFID has been clear from the outset about what it wants from the Water, Sanitation and Hygiene (WASH) Results Programme: WASH interventions delivered at scale within a short time-frame and confidence in the results being reported to the UK taxpayer. DFID got what it wanted, but at what cost? In this post we build on discussions at the WASH Results Programme’s learning event held earlier this year which looked beyond the numbers of people reached with interventions to explore some of the challenges faced in implementing the programme.
Embed from Getty Images
Can PBR frameworks be designed to incentivise suppliers to focus on the “harder to reach”

A central theme in the workshop was the ongoing puzzle of how to place value (both in commercial/financial and Value for Money terms) on intangible aspirations and benefits, such as reaching the most vulnerable and investing in the processes and social capital that underpin effective programming. So, does the importance placed by a donor on achieving clearly costed, verified results risk squeezing out other values and principles that are central to development programming? Which values might end up being pushed aside and could this be mitigated through better design?

1. High quality programming

Suppliers hit a major challenge during tendering when DFID asked them to provide a price per beneficiary that reflected the cost to the suppliers of reaching that beneficiary. But calculating this cost is complex. Potential suppliers have to think about what kind of costs they should fold into that price per beneficiary when bidding: the more the costs, the higher the bid. During the workshop one Supplier asked rhetorically “Should we say $20 with alignment and $23 without?”

There is some apprehensiveness within the NGO sector about competing with the private sector in this commercial context and they are often advised to be cautious. Will they be undercut by commercial organisations submitting more attractive (read cheaper) bids that lack the added benefits that NGOs can bring: the social capital and ways of working that are difficult to put a commercial value on but will affect the quality of the programming?

DFID has been clear that it does not equate Value for Money (VfM) with “cheap” and it is willing to pay for quality programming, whoever is best placed to deliver it. One improvement to the tendering process would be to articulate some of these added benefits (such as existing relationships and social capital in a programme area) as requirements for bidders. Potential suppliers would thus need to provide evidence within the bidding process.

2. Reaching the hardest to reach 

A criticism levelled at PBR is that by using a fixed “price per beneficiary” approach, it encourages suppliers to focus on people who are easier to reach, a practice sometimes described as “creaming” or “cherry picking”. Stakeholders in the WASH Results Programme are firmly committed to inclusion and during the workshop investigated how that could be incentivised better within a PBR framework. Options explored included multi-tiered price per beneficiary frameworks (as used in drug and alcohol recovery programmes in the UK) although these carry the risk of increasing complexity and reducing flexibility. Another suggestion for incentivising inclusion was careful selection and wording of the objectives and appropriate verification processes in the tender document, however this may risk compromising the flexibility to negotiate targets and verification approaches in response to different contexts.

3. Investing for the future

One related but different challenge that emerged during the workshop was that of placing commercial value on activities that invest for future work in the sector. This includes building the social capital to work with local stakeholders and investing in programmatic innovation (which some suppliers suggested had not been possible under the WASH Results Programme). Do the practical implications of PBR risk capitalising on previous investment made by suppliers, without contributing to it in turn? This is perhaps not an issue while PBR contracts constitute a small proportion of aid financing but would become more so if PBR contracts started to dominate. On the other hand, the benefits that suppliers report, particularly in terms of strengthening monitoring and reporting systems to enable more rigorous real-time results tracking may also spill over into other programmes, benefitting them in turn. It is too early to draw conclusions but it may be the case that a range of different aid mechanisms are required, with the benefits and limitations of each clearly identified.

4. Confidence in results

Finally, it is worth observing the possible trade-off between the value placed by DFID on confidence in results that is so important for communicating with taxpayers, and the effectiveness of aid spending that can be achieved through PBR and the nature of the results it produces. Verification is undoubtedly costly (“someone paid you to come here just to look at that toilet?” a baffled resident of a beneficiary village is reported to have asked of a verification team member).

But there is another aspect of effectiveness: if PBR prompts suppliers to focus their efforts on what can be counted (i.e. what can be verified at scale without incurring prohibitive expense), this may shift their efforts away from development programming with longer-term and more uncertain outcomes. Put simply, this could equate to building toilets rather than working on sanitation behaviour change interventions, that are considered to generate more sustainable positive outcomes. Of course there is no guarantee other forms of aid financing will generate these results and as there is likely to be less focus on measuring those results, comparison would be difficult.

Advice for PBR commissioners

What might this mean for those considering PBR modalities and designing PBR programmes? The experience of WASH Results so far suggests that when designing a PBR programme, commissioners need to:

• be clear on the value implied in “value for money” – consider all of the “values” that are important, including the value of donor confidence in results;
• strike a balance between clearly specifying expected results (particularly for more vulnerable people) and being flexible to the contexts in which suppliers are operating;
• think creatively and collaboratively about how long-term outcomes can be measured;
• explore hybrid funding models but avoid creating the “worst of all worlds” that lacks the flexibility of PBR, increases complexity and imposes multiple reporting frameworks;
• consider whether PBR is the right funding mechanism for the kinds of results you wish to achieve (tools are emerging that can help) ;
• view the PBR component in the context of the broad spectrum of funding to the sector – seek to maximise linkages and mutual value across the sector.

Catherine Fisher, Learning Advisor, WASH Results MVE Team

The report from the WASH Results learning workshop is available to download from DFID’s Research for Development website. 

As always, if you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us.

2 Comments

Posted in Learning from WASH Results, Meeting reports

Tagged challenges, commissioning, contracting, measurement, value for money

Edit

The paybacks and pains of Payment by Results (part 1)

Posted on July 11, 2016 | Leave a comment

Our series of reflections on the WASH Results Programme’s learning starts by identifying where Payment by Results has added value.

Payment by Results (PBR) has been “a highly effective means of incentivising delivery at scale” according to the people that deliver the WASH Results Programme. This finding taken from the report of a recent WASH Results learning event may surprise some PBR naysayers. However, as this first post in a series of reflections on the report shows, when the donors, suppliers and verifiers of WASH Results came together to reflect on their experience of actually delivering and verifying the programme, they were able to agree on several positives alongside their concerns.

The pros and cons of PBR in development are hotly debated online, but the Center for Global Development reminds us that when discussing PBR, we should be clear about who is being paid, for what and how. The particular way in which WASH Results was designed has therefore influenced the experiences of its suppliers (SNV, and the SAWRP and SWIFT consortia). An important feature of the design (extrinsic to the PBR modality) is that delivery was tied to the water and sanitation target (Target 7.C.) of the Millennium Development Goals. The programme began with an extremely time-pressured initial ‘output phase’ to December 2015 (focussing on installation of WASH infrastructure), followed by an ‘outcomes phase’ that started this year. Another key design feature is that WASH Results is 100% PBR. The nature of the results, however, were agreed on a case-by-case basis with each supplier and include outputs, outcomes and in some cases, process-type activities.

Sharpening focus on results
It is certainly the case that the WASH Results Programme has delivered huge results within a very tight time-frame. Earlier this year, for example, SWIFT reportedhaving reached close to 850,000 people with two or more of water, sanitation or hygiene services. During the workshop participants broadly agreed with the statement that PBR was an important factor in incentivising delivery. Some questioned the extent of the contribution of the PBR mechanism, highlighting instead their core commitment to delivery. However, others were clear that the PBR mechanism has sharpened the focus on achieving results:

“Grants have never made it so clear that you ought to deliver. Country directors have to deliver in ways that they have not necessarily had to deliver before and this transpires through to partners, local governments and sub-contractors…Quite a number of these actors have started to applaud us for it.” (Jan Ubels, SNV).

Different consortia passed on the risk of PBR to partners in different ways and the SNV experience reflects their particular approach. But it is evident that the clarity of expectations and pressure to deliver across consortia has been effective in generating results. So, apart from the focus on delivery, what else did people value about the way that PBR has been implemented in the WASH Results Programme?

Flexibility in PBR design
In particular, participants valued the flexibility shown by DFID in setting targets and results milestones to reflect different programme approaches – including agreeing payments for process achievements in some cases. Flexibility in definitions also allowed alignment with local government definitions. The drawback of this flexibility was lack of clarity about expectations and lack of standardisation across different suppliers.

Flexibility in implementation
Suppliers have been able to reallocate resources in response to changing contexts and priorities, without negotiating with the donor. It has also been possible to spread risk across multiple sites of operation; overachieving in one location to offset lower results in another.

Clarity of reporting
The focus on results has driven investment and improvements in Monitoring and Evaluation which is broadly thought to have value beyond the programme. Although reporting requirements that are focused exclusively on results are demanding, people welcomed not having to do the activity reporting that is a feature of many other forms of aid.

Some positives were identified during the discussions at the WASH Results workshop and there is much to celebrate. However, a central theme in the workshop was the ongoing challenge of how to place value (in commercial/financial and value for money terms) on intangible aspirations and benefits, such as reaching the most vulnerable and investing in the processes and social capital that underpin effective programming. These challenges will be explored in the next post.

Catherine Fisher, Learning Advisor, WASH Results MVE Team

The report from the workshop is available to download from DFID’s Research for Development website. 

As always, if you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us.

Leave a comment

Posted in Learning from WASH Results

Tagged advantages, incentives, Monitoring & Evaluation, Payment By Results

Edit

Is it time for Results 2.0?

Posted on March 4, 2016 | Leave a comment

Catherine Fisher reports back on a Bond Conference panel discussion about the origins and future of the ‘results agenda’.

Aid should generate results. On the face of it, an indisputably good idea, but the ‘results agenda’ is anything but uncontroversial and can spark “epic” debates. In the WASH Results Programme, this agenda is manifest in the funding relationship – Results Based Financing (RBF) – a form of Payment By Results (PBR) through which DFID makes payments to Suppliers contingent on the independent verification of results. As the Monitoring, Verification and Evaluation (MVE) providers for WASH Results, we’re keen to exchange the programme’s insights with those of other people who have first-hand experience of the results agenda.

One such opportunity arose this week at the Bond Conference, in a session entitled ‘How to crack Results 2.0’ chaired by Michael O’Donnell, Head of Effectiveness and Learning at Bond and author of ‘Payment by Results: What it means for UK NGOs’. The session considered the origins and implications of the results agenda and looked ahead to the next version. Catherine Fisher, Learning Advisor for the WASH Results MVE team, reports on a lively discussion about how results agendas could be aligned with work on social transformation, enable learning and reflection within programmes, and provide value for money themselves.

Looking back to the origins of results approaches in DFID

Opening the session, Kevin Quinlan, Head of Finance, Performance and Impact at DFID explained how, in 2010, DFID encountered two opposing forces: increased funding in order to meet the UK’s legal commitment to spending 0.7 percent of national income on Overseas Development Aid alongside the introduction of austerity measures that required cuts in, and increased scrutiny on, public spending. Results and transparency agendas were DFID’s response to those competing demands and made a shift towards delivering results now in comparison with systems strengthening for results in future. This implied a corresponding shift to talking about the results DFID is going to support rather than the activities they would support to achieve results in future. Six years on, DFID is reassessing its approach.

Can results approaches be reconciled with the “art of transformation”?

Earlier that day, Dr Danny Sriskandarajah, Secretary General of CIVICUS, told conference delegates that INGOs had become too focused on the “science of delivery” (which he described as the achievement of impact by any means) as opposed to the “art of transformation” – the work of bringing about social change. This theme re-emerged during the ‘Results 2.0’ discussion: how could the focus on hard results, embedded in results frameworks, be reconciled with the messy business of social transformation that is at the heart of struggles for equity and rights?

Jessica Horn, Director of Programmes at the African Women’s Development Fund, noted that results frameworks do not acknowledge power or monitor how it is transformed. Consequently she and her colleagues resort to what she called “feminist martial arts” – twisting and turning, blocking and jabbing to defend the transformative work they do, from the “tyranny of results”.  Often, Jessica argued, the politics of the process are as important as the politics of the outcome and asked “how does the results framework capture that?” Yet as Irene Guijt, newly appointed Head of Research at Oxfam GB, argued, being forced to think about results even in the social transformation context helps to make things clearer. Between them, they had some suggestions about how it could be done.

Irene contended that there needed to be greater differentiation of what kind of data we need for different reasons, rather than a one-size-fits-all approach to accountability. She argued that “results” are too often about numbers and we need to bring humans back in and tell the story of change. Irene recommended using the tool SenseMaker to bring together multiple qualitative stories which, through their scale, become quantifiable. Jessica shared some frameworks for approaching monitoring and reporting on social transformation more systematically and in ways that consider power, such as Making the Case: the five social change shifts and the Gender at WorkFramework.

Does focusing on monitoring results for accountability squeeze out reflection and learning?

This criticism is often levelled at results-based approaches and their associated heavy reporting requirements. Irene commented that “learning and data are mates but compete for space”. To align learning and reflection with results monitoring, she advised focusing on collective sense-making of reporting data, a process that enables evidence-based reflection and learning. She also suggested streamlining indicators focussing on those with most potential for learning, a point echoed by Kevin from DFID who emphasised the need to select indicators that are most meaningful to the people implementing programmes (rather than donors).

Do results agendas themselves demonstrate value for money?

This question resonated with the participants, triggering musings on the value of randomised controlled trials and the cost of management agents from the private sector. One point emerging from this discussion was that often what is asked for in results monitoring is difficult to achieve. Indeed, this has at times, been the experience of the WASH Results Programme, particularly in fragile contexts (see for example, the SWIFT Consortium’s report [PDF]) . Both Irene and Jessica talked of the need to use a range of different tools for different purposes and Irene made reference to her recent work on balancing feasibility, inclusiveness and rigour in impact assessments.

What is the trajectory for DFID and the results agenda?

Kevin Quinlan took this question head on, agreeing that this is something DFID needs to decide in the next few months. He suggested that some of the areas for discussion were:

• Getting to a more appropriate place on the spectrum between communication (to tax-payers) and better programme design; results are part of communicating to tax-payers but not the only part;
• Reducing standard indicators in favour of flexible local indicators; each project would need at least one standard indicator to allow aggregation but there should be more local indicators to enable learning;
• Alleviating the torture of results – “rightsizing” the reporting burden and reducing the transaction costs of results reporting; thinking about what results can do alongside other tools.
• Adopting a principles-based approach rather than a set of rules.

Meanwhile the Evaluation Team for WASH Results is investigating some of the issues raised during the panel such as examining the effect of results verification on Suppliers’ learning and reflection, and seeking to explore the value for money of verification.

So it sounds like there will be more interesting discussions about the results agenda in the near future and we look forward to contributing insights from WASH Results*. Whether Results 2.0 is on the horizon remains to be seen.

* Please email the MVE Team if you would like us to let you know when our evaluation findings are available.

Leave a comment

Posted in Meeting reports

Tagged indicators, value for money, verification

Edit

Latest views and news on Payment By Results

Posted on February 29, 2016 | Leave a comment

Our last post shared our lessons on Payment By Results (PBR); this time we’re taking a look at what other people are saying about it.

An over-looked benefit of PBR is sharpening the minds of donors and recipients, observes Joseph Holden, deputy lead of Monitoring and Evaluation (M&E) for the Fund Manager of DFID’s £350 million Girls’ Education Challenge (GEC) Programme. Writing at the end of 2015, Holden highlights PBR’s advantage over other funding models: reducing “fuzzy thinking” and over-statement of results by replacing self-reporting with independently verified, reliable and robust evidence. Writing about the approach and experience of the GEC, Holden comments “This is not cheap, and complexities have meant the application is far from perfect, but the evidence produced is of a higher quality than for the vast majority of other development programmes”. You can read more about the GEC’s experience of verifying PBR in the report on our chat show at the UK Evaluation Society

A new literature review [PDF]  intends to help PBR stakeholders: clarify the purpose of an individual PBR scheme, identify critical success factors, identify common issues which cause problems and difficulties, and be aware of how they might address or mitigate them. The review’s author, Russell Webster, is currently exploring the main themes of the review through a series of blog posts – the most recent ones question if PBR can improve outcomes and if PBR can save money.

Finally, even as this post is being written, delegates at the BOND conference are discussing ‘How to crack Results 2.0‘. We look forward to finding out what they had to say…

Cheryl Brown, Communications Manager for WASH Results MVE

Leave a comment

Posted in PBR in the news, Uncategorized

Edit

What have we learned about Payment by Results (PBR) programmes from verifying one?

Posted on February 5, 2016 | Leave a comment

After 19 verification rounds, the WASH Results Monitoring and Verification team shares its suggestions for how to design future PBR programmes.

Verification is at the heart of the WASH Results Programme. Suppliers only get paid if we, the Monitoring and Verification (MV) team, can independently verify the results they are reporting. Usually we can: results are reported by Suppliers, verified by us and Suppliers are paid by DFID to an agreed schedule. However, all Suppliers have received deductions at least once, which, although painful for everyone, is testament to the rigour of the verification process. Overall, the system is working and the results of the programme are clear. But the demands of verification are also undeniable, leading to some aspects of verification being labelled “Payment by Paperwork” and like any process, it could be improved.

In January 2016 the team* came together to reflect on what we have learned so far from conducting 19 rounds of verification across the three Suppliers. Our discussions focused on verification but inevitably considered wider issues around design of a PBR programme. Here we share some suggestions for design of future PBR programmes, from a verification perspective.

1. Ensure targets and milestones reflect high level programme objectives
2. Be clear on targets and assumptions about their measurement
3. Think carefully about enabling alignment with local government and other WASH stakeholders
4. Reconsider the 100% PBR mechanism to avoid verification inefficiencies
5. Consider payments for over-achievement of outcomes, but not of outputs
6. Include provision for a joint Supplier and Verifier inception phase that will streamline verification
7. Consider pros and cons of relying more on Supplier-generated evidence as opposed to independent evidence generation

1. Ensure targets and milestones reflect high level programme objectives
The WASH Results Programme has ambitions with regard to equity, gender and disability and overall health benefits that are not universally built into targets and payment milestones agreed between DFID and Suppliers. As a consequence, these ambitions are not explicitly incentivised. Any future programme should think carefully about how the design of the programme, especially the targets set in the tender and agreed with Suppliers, uphold objectives based on good practice within the sector.

2. Be clear on targets and assumptions about their measurement
We have found that when payment decisions are riding on whether targets have been met, the devil is in the detail. During implementation, some discrepancies have emerged over targets and how to achieve them. Discussions have taken place about minimum standards for latrines (DFID or JMP definition) and hygiene targets (what does ‘reach’ mean?). In addition, there was occasionally lack of clarity on how achievement of targets would be measured.

When working at scale, assumptions made about the average size of a household in a particular area, or the best way of measuring the number of pupils in a school become subject to intense scrutiny.  This is quite a departure from how programmes with different funding mechanisms have worked in the past and the level of detailed evidence required may come as a shock for Suppliers and Donors alike. In response, we suggest that future programmes should provide clear guidance on technical specifications relating to targets and guidelines for evidencing achievements.

3. Think carefully about enabling alignment with local government and other WASH Stakeholders
One concern that we discussed in the meeting was that the design of the WASH Results Programme does not sufficiently incentivise alignment with local government. We suspect that this was a result of the scale of the programme and the tight timelines, but also the demands of verification. The need to generate verifiable results can dis-incentivise both the pursuit of “soft” outcomes such as collaboration, and, working with government monitoring systems.

We suggest that PBR programmes need to think carefully about how to incentivise devolution of support services from progamme teams to local governments, and to other sector stakeholders during the life of the programme, for example by linking payments to these activities. Also, to think how programme design could encourage long-term strengthening of government monitoring systems.

4. Reconsider the 100% PBR mechanism to avoid verification inefficiencies
The merits or otherwise of the 100% PBR mechanism in the WASH Results Programme are subject to much discussion; we considered them from a verification perspective. We believe that, in response to the 100% PBR mechanism, some Suppliers included input- and process-related milestone targets to meet their internal cash flow requirements. In some cases, this led to verification processes that required high levels of effort (i.e. paperwork) with relatively few benefits.

We suggest that people designing future PBR programmes consider non-PBR upfront payments to Suppliers to avoid the need to set early input and process milestones, and run a substantial inception phase that includes paid-for outputs for Suppliers and Verifiers. In the implementation phase of the WASH Results Programme, payment milestones have been mainly quarterly, so requiring seemingly endless rounds of verification that put pressure on all involved, particularly Supplier programme staff. In response, we suggest that payments over the course of a programme should be less frequent (and so possibly larger), so requiring fewer verification rounds and allowing greater space between them. This may have implications for the design of the PBR mechanism.

5. Consider payments for over-achievement of outcomes, but not of outputs
The WASH Results Programme does not include payment for over-achievement. Over the course of the programme, some Suppliers have argued that over-achievement should be rewarded, just as under-achievement is penalised. As Verifiers, we agree that paying for over-achievement for outcomes would be a positive change in a future PBR design. However, there were concerns among our team that encouraging over-achievement of outputs could have unintended consequences such as inefficient investments or short-term efforts to achieve outputs without sufficient attention to sustainability and the quality of service delivery.

6. Include provision for a joint Supplier and Verifier inception phase that will streamline verification
It is broadly accepted that the WASH Results Programme would have benefited from a more substantial inception phase with the Verification Team in place at the start. Our recommendations about how an inception phase could help streamline and strengthen verification are as follows:

• Key inception outputs should include a monitoring and results reporting framework agreed between the Supplier and the Verification Agent. Suppliers and Verifiers could be paid against these outputs to overcome cash flow issues.
• The inception phase should include Verification Team visits to country programmes to establish an effective dialogue between the Verifiers and Suppliers early on.
• If Suppliers evidence their achievements (as opposed to independent collection of evidence by the Verification Agent – see below), assessment of, and agreement on, what are adequate results reporting systems and processes need to be included in the inception phase.
• Run a ‘dry’ verification round at the beginning of the verification phase where payments are guaranteed to Suppliers irrespective of target achievement so that early verification issues can be sorted out without escalating stress levels.

7. Consider pros and cons of relying more on Supplier-generated evidence as opposed to independent evidence generation
In the WASH Results Programme, Suppliers provide evidence against target achievements, which is subsequently verified by the Verification Team (we will be producing a paper soon that outlines how this process works in more detail). Is this reliance on Supplier-generated evidence the best way forward? What are the pros and cons of this approach as compared with independent (verification-led) evidence generation?

Indications are that the PBR mechanism has improved Suppliers’ internal monitoring systems, and has shifted the internal programming focus from the finance to the monitoring and evaluation department. However, relying on Suppliers’ internal reporting systems has required some Suppliers to introduce substantial changes to existing reporting systems and the MV team has faced challenges in ensuring standards of evidence, particularly in relation to surveys.

We have some ideas about pros and cons of Supplier-generated evidence as opposed to evidence generated independently, but feel this can only be fully assessed in conversation with the Suppliers. We plan to have this conversation at a WASH Results Programme Supplier learning event in March. So, this is not so much a suggestion as a request to watch this space!

Coming up…

WASH Results Programme Learning Event:  On March 7 2016 Suppliers, the e-Pact Monitoring & Verification and Evaluation teams, and DFID will convene to compare and reflect on learning so far. Key discussions at the event will be shared through this blog.

Verification framework paper: an overview of how the verification process works in the WASH Results Programme. This will present a behind-the-scenes look at verification in practice and provide background for future lessons and reflections that we intend to share through our blog and other outputs.

 

 

---

* About the MV Team: In the WASH Results Programme, the monitoring, verification and evaluation functions are combined into one contract with e-Pact. In practice, the ongoing monitoring and verification of Suppliers’ results is conducted by one team (the MV team) and the evaluation of the programme by another.  The lessons here are based on the experience of the MV team although members of the Evaluation team were also present at the workshop. Read more about the WASH Results Programme.

Leave a comment

Posted in Learning from WASH Results, Uncategorized

Tagged contracting, outcomes, outputs, Payment By Results, verification

Edit

As the MDGs become SDGs, what progress has WASH Results made?

Posted on November 26, 2015 | 1 comment

What has the DFID WASH Results Programme achieved so far and what lies ahead for the programme in 2016?

January 1^st, 2016 is an important date for the DFID WASH Results Programme.The Sustainable Development Goals (SDGs) take the place of the Millennium Development Goals (MDGs) and ‘WASH Results’ moves into the second half of its funding period. At this stage the focus for the programme’s Suppliers (SAWRP, SNVand SWIFT) will start shifting from getting things done, to keeping things going and ensuring the sustainability of WASH outcomes. For e-Pact too, as the independent verifiers, it’s farewell to Outputs and hello to Outcomes.

 
Embed from Getty Images
Once the goals to provide access have been reached, attention turns to sustainability. 

Before we look ahead to next year, let’s take a look at what’s been achieved so far. According to the UN, 2.6 billion people gained access to improved drinking water sources between 1990 and 2015 – a key part of MDG 7. Worldwide, 2.1 billion people have gained access to improved sanitation but 2.4 billion are still using unimproved sanitation facilities, including nearly 1 billion people who are still defecating in the open. This July, DFID was able to report it had exceeded its own target of supporting 60 million people to access clean water, better sanitation or improved hygiene conditions.

What contribution can we attribute to the WASH Results Programme? The full report on the programme’s 2015 Annual Review is available on the UKAid Development Tracker website. Here are some of the highlights:

• The reviewers gave the programme an ‘A’ overall (for the second year running) and considered it to be “on track” to meeting its targets by the end of 2015.
• By December 2014 the WASH Results Programme had reached 296,438 people with improved sanitation, 65,611 people with improved water supply, and over 1.25 million people with hygiene promotion.
• The reviewers noted that “strong independent verification systems” have been established that also allow for adjustment and improvement based on learning from previous verification rounds.
• WASH Results is generating significant policy knowledge around use of Payment by Results and programming for outcomes (sustainability) in the WASH sector.

We’ll come back to these results in early 2016 when the final numbers are in and compare them to the programme’s targets for December 2015 which are:

• 968,505 people have access to clean drinking water;
• 3,769,708 people have access to an improved sanitation facility;
• 9,330,144 people reached through hygiene promotion activities through DFID support.

Large numbers, however impressive, don’t fully convey the effects that improvements in water, sanitation and hygiene are having on people’s lives. The Suppliers have been collecting stories of change from some of the people on the ground who are closely involved in delivering the WASH Results Programme or directly benefitting from its work. You can read about 70 year old subsistence farmer Maria and mother-of-five, Jacinta, in a recent DFID blog post. The SWIFT consortium’s website is also packed with news and images from their involvement in the WASH Results Programme.

So what happens next?

From 2016 onwards, e-Pact starts answering a critical question for DFID: how many poor people continue to use improved water and sanitation facilities and are practising improved hygiene because of the WASH Results Programme? Right now we’re exploring how best to monitor, report on and verify these outcomes and look forward to sharing what we learn, with you.

As always, if you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us.

1 Comment

Posted in About WASH Results

Tagged outcomes, outputs, sustainability, targets, verification

Edit

Suggestions to donors commissioning Payment by Results programmes

Posted on July 27, 2015 | Leave a comment

Reflections by DFID WASH Results’ suppliers on the programme’s design and commissioning, summarised into six suggestions for commissioners.

Payment by Results (PbR) is a technically challenging form of contracting and one that is new for DFID and many of its partners (NAO 2015, p8, DFID, 2015 p18).   The WASH Results Programme was designed using the PbR financing mechanism, and therefore has great potential for learning about the impact of PbR on programming.

One year into the programme, the WASH Results suppliers met to share experiences so far, particularly how the programme was designed and commissioned and the effects PbR had. This discussion has been distilled into a set of suggestions for people commissioning PbR programmes in development.

The discussions reflected the participants’ experiences as people leading supplier consortia, rather than those directly implementing the programme or its beneficiaries and although DFID staff were present their role was largely that of observer. Discussions built on the early findings shared by the Oxford Policy Management team leading the formal evaluation of the programme for the e-Pact consortium. The suggestions that emerged are inevitably not as rigorous as the final findings that will come from this formal evaluation nor from others’ extensive reviews drawing on multiple programmes and experience. However, our suggestions add support to some of the National Audit Office (NAO) recommendations on PbR that were published a month after our workshop, in particular the “Payment by Results: analytical framework for decision-makers” (NAO, 2015).

1. Consider what type of PbR is the right mechanism for achieving the desired outcomes.

There are multiple ways in which PbR programmes can be designed and the selection made will influence who applies, how suppliers devise and manage their programmes and the subsequent results. An important difference between PbR programmes is the proportion of payment that is “pure PbR”. WASH Results is designed as a 100% PbR programme and this prompted some suppliers to take steps to minimise the risk of not achieving targets. Some set disbursement milestones early in the impact pathway to aid cashflow, rather than set exclusively outcome-based milestones. Some rethought the areas they were working in, shifting away from work in the most challenging areas, and there is some indication that suppliers relied on tried and trusted methods rather than trying new approaches.

The workshop participants suggested that pre-financing may be necessary to achieve genuine emphasis on outcomes, particularly in risky contexts or if innovation is important so that risks are not born solely by suppliers. Echoing other discussions on PbR (BOND p13) , participants discussed how the PbR mechanism may not be appropriate for programmes that are intended to target the most vulnerable and/or operate in fragile contexts, however, the success of the WASH Results Programme in the Democratic Republic of Congo demonstrates that it should not be ruled out. These issues need careful consideration when selecting the design of a PbR programme and the NAO report provides an analytical framework for doing so.

2. Create spaces for learning about PbR within the commissioning process.

As the PbR modality is new to many organisations (suppliers and commissioners) it is important for commissioners and suppliers to explore options and learn from experience in order to design the most effective model. Ideally this will happen in an interactive way that will help to build relationships between stakeholders. Workshop participants stressed the importance of learning from the private sector about approaches to PbR, for example in areas such as how to assess and price risk. They also advocated a longer inception phase in programmes using multiple suppliers, to enable suppliers to talk to each other about the approaches they are considering adopting.

The NAO’s analytical framework for decision-makers advocates a learning approach to commissioning PbR programmes and it encourages commissioners to engage in dialogue with potential providers from the outset. However it does not explicitly recommend creating spaces for learning and relationship-building between stakeholders within the commissioning process; we think this would be a useful addition to the framework.

3. The tender documents need to be clear about what is meant by PbR, from the outset.

This includes providing the rationale for using PbR and sufficient detail about matters such as disbursement, results, and the processes and conditions around force majeure, particularly but not exclusively in fragile contexts. There should be clear guidance on how to define minimum standards: in sanitation, for example, does the minimum criteria for improved latrines allow for shared use?  The NAO framework helps commissioners to think through high-level issues, however the detail of sector specific standards may need to be determined in pre-tendering discussions with potential suppliers and/or monitoring and verification providers.

4. Set up Monitoring, Verification and Evaluation (MVE) frameworks before implementation starts.

The MVE team for a PbR programme needs to be appointed before the suppliers and the MVE framework needs to be created before implementation starts. This would enable standardisation of approaches across suppliers and enable suppliers to create their monitoring and evaluation frameworks according to the requirements of verification processes. This is perhaps the clearest suggestion to emerge from the workshop and once again accords with the NAO framework which asks “Is there an agreed system for measuring provider performance? Will this system be in place at the start of the scheme?” The WRP suppliers’ experience suggests that it should be!

5. Consider what impact the contracting process will have on opportunities for collaboration and learning.

The workshop participants felt that the tendering process for the WASH Results Programme made international non-governmental organisations into competitors rather than collaborators and this can lead to lack of transparency and learning between organisations. Given the broader move within aid commissioning from grants towards more competitively tendered contracts, this is an example of where it is difficult to attribute effects purely to the PbR mechanism.

As an alternative to a competitive tender, the WASH Results suppliers suggested commissioners pursue a “negotiated” procurement procedure during which the commissioner enters into contract negotiations with one or more suppliers.

6. Allow extra time to commission PbR programmes than for more familiar contracting processes.

The complexity of PbR contracts and their relative unfamiliarity in the development sector imply the need to allow more time for commissioning than usual. How to use that extra time? The previous suggestions from our workshop indicate that this lengthier commissioning process should comprise the following phases:

• Pre-bidding phase: in which the donor’s understanding of PbR is clearly laid out, and in which potential suppliers are able to access insights into the different models of PbR financing and the requirements and risks of implementing a PbR programme.
• Bidding/contracting phase: recognising that this is a resource-intensive process for suppliers; possibly conducted as a negotiated process rather than a competitive tender.
• Inception phase: with space for suppliers to share approaches and refine their programme design, in which a standardised MVE framework is shared and refined and means of verification agreed; milestones and payment schedules are agreed; programme expectations are agreed with partners; and consortia develop common understanding, language and approaches.
• Implementation phase: implementation with appropriate verification cycles and disbursement points, ongoing learning and review.
• Closing phase: for end-of-term evaluation, especially to draw out lessons learnt and find ways of furthering work beyond the project period of implementation.

Have you tendered for, or commissioned a PbR programme?

Do our suggestions and observations reflect your experience?

What suggestions would you add to our list?

Leave a comment

Posted in Learning from WASH Results

Tagged commissioning, contracting, learning, Payment By Results

Edit

Older posts

• 

• HOW DO YOU MAKE PAYMENT BY RESULTS WORK IN WATER, SANITATION & HYGIENE (WASH)?

  The e-Pact Consortium is finding out through Verifying and Evaluating DFID’s WASH Results Programme. But we’re not waiting until 2018, when the programme ends, to pass on what we’ve discovered to other suppliers, donors and verifiers about how this relatively new funding mechanism works in the WASH sector.

  Throughout the programme we’ll be writing about our experience of being Independent Verifiers, sharing emerging findings from our Evaluation team, highlighting lessons learned by the Suppliers, and taking a look at what other people are saying about Payment by Results.

  We encourage you to share your lessons and ideas on this topic by making a comment on one of our posts or sending us an email.

• FIND OUT MORE…

  • Email the WASH Results MVE Team
  • About WASH Results
  • The SAWRP Consortium
  • SNV
  • The SWIFT Consortium
  • WASH Results on Development Tracker

• YOU ARE FOLLOWING THIS BLOG

  You are following this blog (manage).

• RECENT POSTS

  • Can Payment by Results raise the bar for downward accountability?
  • Alignment, aid effectiveness and Payment by Results
  • Measuring WASH sustainability in a payment by results context
  • The paybacks and pains of Payment by Results (Part 2)
  • The paybacks and pains of Payment by Results (part 1)

• @WASHRESULTSMVE ON TWITTER

• THE WASH RESULTS PROGRAMME IS FUNDED BY:

  

• DISCLAIMER

  This blog is an output from a programme that has been funded by UK aid from the UK government; however the views expressed do not necessarily reflect the UK government’s official policies or those of the programme’s partners.