Designing effective verification systems for Payment by Results contracts

New series of publications shares practical guidance for third-party verification services based on learning from the WASH Results Programme.

What do you need to consider when commissioning and designing a verification system? How do you go about creating a verification cycle, appraising monitoring systems or choosing good indicators? E-Pact has published a set of guidance notes to help others to design and implement an effective verification system, drawing on our experience from 2014-2020 as the Monitoring and Verification team for the DFID WASH Results Programme.  

WASH Results is one of DFID’s flagship programmes in using Payment by Results (PbR) and with a £112 million budget is one of the first large-scale applications of PbR in the sector. The programme was conceived in 2013 during an era where DFID shifted programming towards longer-term results. An important rationale for the programme was to incentivise large-scale delivery of WASH under the Millennium Development Goals in 2015. This ambitious delivery goal was coupled with payment for outcomes to encourage the continued use of water supply, latrines and handwashing at critical times, in line with the SDG agenda.

Delivered by three consortia of non-governmental organisations (‘suppliers’), the programme has enabled over 1.6 million people to gain access to water, 7.4 million access to sanitation, and reached 16.1 million with hygiene promotion across 11 countries. The programme has also overwhelmingly achieved its outcome targets.

WASH Results operates under a ‘100 percent’ PbR modality, whereby suppliers receive payments only on the successful verification of their results by a third-party Monitoring and Verification (MV) team. The verification was systems-based, meaning that Itad in collaboration with Ecorys and Iwel under the e-Pact consortium, independently verified the data reported by the suppliers. In this process, supplier monitoring and reporting systems are appraised and data they produce are verified by the third-party MV team to confirm they are accurate and realistic. This approach has benefits including value for money and those derived from strengthening suppliers’ own monitoring systems. This ‘systems-based’ approach to verification was completely novel at the time and over the past seven years, the verification team tested and refined its methods and processes and learned lessons, some of which are captured in our short series of publications.

The publications are is intended to support organisations when commissioning, managing or implementing third-party verification services as part of PbR contracts and those interested in strengthening monitoring systems through third parties. The Guidance Note provides overarching lessons, useful at an early stage of thinking through the design of a PbR programme while the accompanying Verification in Practice Notes dive more deeply into key aspects of systems-based verification.

The DFID Payment by Results Guidance Note: Lessons from an effective verification system [PDF] provides 12 lessons to inform the creation of a verification system. It covers three aspects: 1) getting the foundations right at design and tender stage, 2) designing the verification system so it is effective and efficient and avoids surprises when payment decisions are being made; and 3) the role of commissioners in a PbR contract compared to traditional grants.

Martha Keega assesses a latrine in South Sudan
MV team member Martha Keega assesses a latrine in South Sudan

Verification in Practice #1: The Verification Cycle – Step by Step [PDF] In a PbR programme, suppliers only get paid when results have been independently verified and this requires a verification process that delivers good quality data in a timely and predictable manner. The MV team has put this into practice through a nine-step, three-month verification cycle, which is outlined in this note.

Text Box
The WASH Results Programme’s verification cycle is explored in Verification in Practice #1

Verification in Practice #2: Appraising Monitoring Systems [PDF] A key part of a “systems-approach” to verification is the appraisal of supplier monitoring systems, known within the WASH Results Programme as a “systems appraisal”. This note explains how the MV team defines a systems appraisal, what it involves and why one can be useful. This will be of relevance to all those seeking to strengthen monitoring systems, not just those in PbR programmes.

Verification in Practice #3: What makes a good indicator for a Payment by Results programme? [PDF] Good quality indicators are essential to effective PbR programmes. This practical note suggests seven essential characteristics of a good PbR indicator and five additional characteristics that may be desirable.

We hope that this guidance drawn from our experience will be useful to you if you are commissioning or designing a verification system or just want to gain confidence in the reporting of results regardless of whether or not you decide to attach payment to it. After seven years and more than 50 rounds of verification we feel that we’ve got a bit of a handle on this, so please do get in touch if you have any questions and we encourage you to share your thoughts in the comments.


 

Working at scale in the WASH sector: lessons from the WASH Results Programme

Extensive evaluation of a four-year DFID funded WASH programme reveals lessons for future large-scale programmes.

Child's hands catching water from spout

Delivering WASH at scale – lessons from the WASH Results Programme. Photo by Kelly Lacey from pexel.com

The WASH Results Programme was a large-scale programme undertaken as part of the UK Government pledge to ensure that 60 million people would gain access to water, sanitation and hygiene (WASH) services by December 2015 as a result of UK investments.

The Department for International Development (DFID) funded WASH Results Programme worked in 11 countries, and aimed to provide 1.2 million people with access to clean drinking water sources, 4.3 million people with access to an improved sanitation facility, and reach 9.3 million people with hygiene promotion activities. By most measures the programme was very successful in achieving its stated objectives. The vast majority of output-level targets were achieved by all suppliers, and in many areas with significant overachievement. The suppliers also overwhelmingly achieved the outcome-level targets, with significant overachievement in several areas and modest underachievement in relatively few others.

The WASH Results Programme had two defining characteristics: the fact that it was a Payment-by-Results (PbR) programme, and its size and structure. The design and effects of the PbR modality are explored in earlier blog posts.

This blog post, based on a paper by Richard Carter and Jeremy Colin, draws on the mid-term and end-point evaluations of the WASH Results Programme to reflect on a number of aspects of working at scale in the WASH sector. In particular it explores:

  • How the WASH Results Programme and other WASH sector actors have conceptualised scale.
  • The outcomes and impacts set out in the WASH Results Programme design, and the extent of their achievement.
  • What requirements were placed on suppliers, in the pursuit of programme outcomes and impacts at scale.
  • How inclusive and sustainable services and behaviour changes at scale can best be pursued in future WASH programmes.

This blog post is intended for governments, international agencies, NGOs and development funding partners which wish to bring about beneficial, inclusive and sustainable changes in water, sanitation and hygiene at scale. The evaluation that this post is based on is available from the DFID Research Portal as a Synthesis Report, Executive Summary and Annexes. 

Aspects of working at scale in WASH sector

The wider literature on scale, within the WASH (eg Taylor, 2013[i]), health (eg Mangham and Hanson, 2010[ii]; Gargani and McLean, 2017[iii]), education (eg Coburn, 2003[iv]; Burns, 2014[v]) and food security (Frake and Messina, 2018[vi]) sectors, understandably focuses more on “scaling-up” than on scale per se. The two ideas are linked, however, in the question “assuming we know how to deliver services at small scale, how can those services be extended or accelerated to reach large numbers?” Cynthia Coburn provides a helpful four-dimensional understanding of scale.

Understanding “scale” (Coburn, 2003)
Coburn (writing about the education sector) highlights four necessary dimensions of scale:

1.    Depth. The nature and quality of services, “change that goes beyond surface structures or procedures to alter … beliefs … norms … and principles (as enacted in the curriculum)”.

2.    Sustainability. “The distribution and adoption of an innovation are only significant if its use can be sustained … there is ample evidence that sustainability may be the central challenge of bringing reforms to scale”.

3.    Spread. “ … the importance of taking into consideration what is spread … but also … influencing district policies, procedures and professional development”.

4.    Shift in reform ownership. “ … ownership over the reform must shift so that it is no longer an external reform, controlled by a reformer, but rather becomes an internal reform …”.

The significance of ‘scale’ in WASH programming has different implications in each of the sub-sectors. For example water services are more likely to be financially sustainable if the customer base is large and diverse, so that wealthier market segments (enjoying high levels of service) can subsidise less wealthy segments with more basic services. In regard to sanitation, beneficial health impacts are only likely to emerge as high levels of actual use are realised – reducing open defecation and other forms of unsafe disposal of human excreta to the minimum. Hygiene practices too need to reach high levels of compliance in order to provide a basis for health impacts.

Linked to all of the individual sub-sectors is the significance of the programme’s overall scale with regards to the geographical and administrative boundaries in which it is operating. Administrative boundaries are particularly significant with regards to achieving sustainability and capacity building aims, and programme’s that align themselves with the administrative levels that have responsibility for maintaining services enhance the prospects for sustainability. In the rural context this often entails taking a ‘district-wide approach’ to delivering services, and in the urban context focusing on city-wide service delivery. This view of scale is also significant with regards to equity as by aiming to reach everyone with services in a given area, by definition, implies a focus on equitable service delivery.

Scale in the WASH Results Programme

Scale was defined in terms of beneficiary numbers

In the WASH Results Programme, DFID called for suppliers which could deliver results at scale. No explicit definition of “scale” was provided, although the programme aimed to reach 1.2 million people with access to clean drinking water sources, 4.3 million people with access to an improved sanitation facility, and 9.3 million people with hygiene promotion activities.  “Scale” was effectively defined therefore in terms of beneficiary numbers.

In the WASH Results Programme, the suppliers did not formally define scale, but they pursued it in different ways, some working across multiple countries and districts, partially filling the gaps left by other government and non-government agencies; and others working in more focused locations (e.g., in certain cities or geographical areas).

The WASH Results Programme aimed to achieve scale by bringing services to millions of people. Alternative interpretations of scale in WASH programming, such as focusing on improving the effectiveness and reach of government interventions; establishing national funds which could be accessed by local governments; or advancing sector reforms and fiscal decentralisation were not pursued.

The programme was phased: an initial focus on outputs was followed by an outcomes phase

The WASH Results Programme consisted of two ‘phases’, and this had an important influence on suppliers’ programming. The output phase ran from April 2014 until December 2015 (the MDG deadline). During this phase, services were brought to the target populations and approximately 80% of the budget was disbursed. In the outcome phase (January 2016 to early 2018) the reach and sustainability of the services provided in the first phase were to be consolidated. The main argument in favour of the WASH Results Programme’s phasing was the fact that suppliers would have a real opportunity, post-implementation, to focus entirely on equity and sustainability issues. One of the three suppliers in the WASH Results Programme effectively ignored the phasing, due to having a more comprehensive framework for addressing sustainability. Though for the other two suppliers the pressure in the output phase to deliver results resulted in comparatively less attention being paid to equity and sustainability issues earlier in the programme.

The issue of phasing is significant for future programmes that seek results at the output and outcome level and as it is possible that a similar arrangement could occur in future as the SDG deadline approaches or in response to the WASH challenges posed by the COVID-19 pandemic. In particular, that the pressure to deliver output-level results should not take precedence over sustainability or equity considerations.

Delivering at scale – key outcomes and impacts sought under the WASH Results Programme

Inclusion. The WASH Results Programme did not set inclusion-related targets; though many suppliers nonetheless intended that all members of target communities should benefit from their interventions. In view of the public good (and public health) aspects of WASH specifically, this was important and necessary, even beyond the human rights issues involved. Although DFID did not explicitly require suppliers to undertake “total sanitation” interventions (i.e. those in which entire communities access and use safe sanitation), this was in fact the strategy pursued by most suppliers, in line with their usual practices. Local-level “scale” (i.e. near-complete coverage of services in targeted communities and areas) is essential in WASH programmes of any size.

Sustainability. The WASH Results Programme intended to deliver sustainable services and hygiene practices. DFID required suppliers to address the various aspects (functionality of systems, institutional aspects, financial and environmental matters) of sustainability, and it drew attention of suppliers to the need to engage both with communities and with local and national governments in the pursuit of sustainability.

Impacts. DFID’s intended impact was that “poor people have improved health (as a consequence of using safe and sustainable WASH services)”. The indicators of that improved health were to be reduced incidence of diarrhoea and reduced child mortality. Health impacts were not measured, either by suppliers or by the programme evaluation team.

As a statement of impact, this was simultaneously both highly ambitious and very limited. It was ambitious in the sense that the direct impact of WASH programming on diarrhoeal disease is frequently less than hoped for; indeed rigorous studies taking place at the time of the WASH Results Programme were finding such impacts to be elusive. It was also limited however, in the sense that potential health impacts extend well beyond the reduction in diarrhoea (to include a number of neglected tropical diseases – or NTDs – and possible reductions in stunting), and the wider potential impacts extend well beyond health, as attested by DFID’s own Evidence Paper on WASH Impact (Cairncross et al, 2013). With the exception of interventions which were scaled to be truly community-wide or area-wide (i.e. in which all or nearly all inhabitants of communities or administrative areas practise safe sanitation and good hygiene), the impacts on diarrhoeal disease as a result of the WASH Results Programme were likely to have been modest.

In future WASH programmes, a more explicit focus on community-wide use and practice, and on sustainability of services into the indefinite future would bring about a greater likelihood of such impacts being achieved – although current research evidence suggests that even these measures may not necessarily lead to major health impacts.

In addition to the requirements set out in DFID’s Business Case and Invitation to Tender, a number of other requirements were placed on suppliers, especially in regard to sustainability, in-country capacity, and disaggregated monitoring of results.

However, a number of potential requirements were not explicitly made; this was in part due to the programme having a rather rudimentary theory of change, in which key assumptions were neither made explicit nor tested. Had they been, then there is reason to believe that the WASH Results Programme’s outcomes and impact may have been even greater.

Delivering WASH results at scale: recommendations for future programmes

DFID’s emphasis on outcomes (inclusion and sustainability), and its requirement for robust and disaggregated monitoring, are important for WASH programmes at any scale.

A number of additional recommendations can be made for future programmes:

  • Make explicit that all interventions should be undertaken community-wide and not simply in pursuit of numbers of beneficiaries. In the event, the suppliers (all reputable INGOs) carried out their programming in this way, but this cannot be assumed.
  • Where possible, work to achieve full coverage across administrative units such as districts. This would further enhance the likelihood of health impacts, as well as facilitating further scaling-up within the country.
  • Sanitation and hygiene interventions are likely to be more effective where water services are already adequate. In cases where these services are not adequate; improvements in water services are necessary too.
  • Carry out hygiene promotion programmes of sufficient reach and duration to effect significant behaviour change. In the WASH Results Programme, suppliers were expected to increase hand-washing by 15% compared to baseline through their hygiene promotion efforts. The level of this target would appear limited in ambition and certainly insufficient to bring about significant health impacts. Furthermore, the assumption made in parts of the WASH Results Programme that Community Led Total Sanitation triggering included sufficient hygiene promotion, is unsafe.
  • Use the influence of large-scale budgets to put pressure on national governments to implement fiscal decentralisation more fully. The inability of local governments to provide effective post-implementation support to community-level services is due in no small measure to inadequate budgets.

Defining and delivering scale in WASH programming requires attention not only to the numbers of people served, but to the quality and sustainability of services; providing services which are used by very high proportions of the populations of served communities; and achieving a real transfer of ownership from externally-delivered programming to government-ownership of scaling-up processes.

A general lesson from the WASH Results Programme is that outcomes around sustainability, equity and inclusion need to be designed in and pursued from the outset. It is too late to address matters of equity and sustainability if they have been neglected in the pursuit of scale defined solely or mainly in terms of numbers of people reached.

Delivering WASH outcomes at scale is challenging. While it is relatively easy to provide many people with access to services, the more difficult task of ensuring changed use and behaviour takes different skills, more time, and sustained engagement. The transfer of ownership of that engagement with communities, from external suppliers to local government – as envisaged in a full conceptualisation of scale – is particularly difficult when those local governments lack capacity, and crucially, resources.

Given the present uncertainties in predicting health impacts consequent upon WASH interventions, those funding WASH programmes should exercise caution in articulating such intended impacts. However, an important prerequisite for such impacts is community-wide and area-wide use of safe sanitation and of good hygiene (especially handwashing) practices. An explicit and thorough Theory of Change can help in the specification of future programme designs, and of full sets of requirements for implementing partners.

This blog post is adapted from an original paper by Richard Carter and Jeremy Colin,  both members of the OPM evaluation team.

The evaluation on which this post is based is available from the DFID Research Portal and is summarised here:   WASH Results Programme Evaluation: findings and recommendations for the WASH sector


[i] Taylor B (2013) Effectiveness, scale and sustainability in WASH programmes – a review.  Springfield Working Paper Series (2), The Springfield Centre, Durham.  Available at: http://www.springfieldcentre.com/wp-content/uploads/2013/10/2013-10-Effectiveness-Scale-and-Sustainability-in-WASH.pdf

[ii] Mangham L J and Hanson K (2010) Scaling up in international health: what are the key issues?  Health Policy and Planning, Vol 25 pp85-96.  Available at: https://www.ncbi.nlm.nih.gov/pubmed/20071454

[iii] Gargani J and McLean R (2017), Scaling science.  Stanford Social Innovation Review, Fall 2017.  Available at: https://ssir.org/articles/entry/scaling_science

[iv] Coburn C E (2003) Rethinking scale: moving beyond numbers to deep and lasting change.  Educational Researcher, Vol 32, No. 6, pp 3-12.  Available at: https://www.sesp.northwestern.edu/docs/publications/139042460457c9a8422623f.pdf

[v] Burns M (2014) the myths of scaling-up.  Centre for Education Innovations.  Blog available at: https://educationinnovations.org/blog/myths-scaling

[vi] Frake A N and Messina J P (2018) Toward a common ontology of scaling-up in development.  Sustainability Vol 10, pp 835-845.  Available at: https://www.mdpi.com/2071-1050/10/3/835

Answering questions on the WASH Results Evaluation

Over 180 people from NGOs, funding agencies and research organisations around the world with an interest in WASH and Payment by Results participated in a panel discussion last week on the evaluation findings of the WASH Results Programme.

Participants grilled the panellists on issues of sustainability, sector strengthening and the role of monitoring in WASH programmes.   If you missed the webinar, you can watch it online here.

webinar grab

After a short presentation of the key findings, the panelists shared their experience of the programme and responded to questions from the audience which focused largely on:

  • The nature of the design of the Payment by Results modality and how the implementing organisations responded.
  • Whether the WASH Results Programme was good value for money.
  • The sustainability of the programme’s results.
  • How the WASH Results Programme contributed to systems strengthening in the countries in which it operated.
  • The role of monitoring in the programme.

The questions prompted lively debate during the Q&A session and continued in the webinar chat. 

panelFor those who were unable to attend the live webinar, it can be viewed here with a record of the Chat. The synthesis report and other publications from the evaluation are available to download from the DFID website.

If you would like to comment on the evaluation findings or share your experience of comparable programmes, you can do so in the series of blog posts summarising the findings  :  Using Payment by Results to deliver WASH services at scale: lessons from the WASH Results Programme . 

 

 

WASH Results Programme Evaluation: findings and recommendations for the WASH sector

Evaluation of large-scale WASH programme reveals Payment by Results (PbR) is a viable option for funding WASH at scale, if used carefully.

water pouring on pipe

Photo by Nitin Sharma on Pexels.com

This blog summarises findings of the evaluation by the e-Pact Consortium of the UK Department for International Development’s (DFID’s) Water, Sanitation, and Hygiene (WASH) Results Programme: a four-year programme (May 2014 to March 2018), which aimed to bring equitable and sustainable water and sanitation services and hygiene practices to millions of people in 11 countries. The programme was implemented by three suppliers contracted to DFID under a PbR financing modality (SAWRP, SWIFT and SSH4A).

As one of the first large-scale applications of PbR in the WASH sector and within DFID, the WASH Results Programme provides a rich source of learning about how PbR works at scale and how to use it in WASH in the future. Based on the evaluation, this blog explores the following areas:

  1. Whether the programme successfully achieved its stated objectives;
  2. The influence of programme design, including the PbR modality, on this achievement;
  3. Lessons for applying PbR in WASH programming going forward;
  4. Recommendations for future WASH PbR programmes.

This blog is an abridged version of the Evaluation Synthesis, which is available to download from DFID’s Research for Development online repository along with reports from all of the component studies of the evaluation.

Continue reading

Lessons from implementing the WASH Results Programme #3: Implications for funders

Our final blog post in a series, sharing lessons from implementing Payment by Results (PbR) to deliver WASH services at scale draws out implications for funders when designing future programmes.

business-4241792_1920The evaluation found that PbR appeared to be a viable option for funding WASH programmes but with some important design considerations that we explore in this blog post.

Continue reading

Lessons from implementing the WASH Results Programme #2: The influence of PbR

Our second blog post sharing lessons from implementing Payment by Results (PbR) to deliver WASH services at scale explores the influence of the PbR modality on suppliers’ behaviour. 

board-978179_1920The shape of implementing consortia, the countries in which programmes were implemented, and the targets that were set were all influenced by the PbR modality. In this post we explore how the PbR modality of the WASH Results Programme shaped the behaviour of the organisations who were implementing the programme, known as “suppliers”.

Continue reading

Our second blog post sharing lessons from implementing Payment by Results (PbR) to deliver WASH services at scale explores the influence of the PbR modality on suppliers’ behaviour. 

board-978179_1920The shape of implementing consortia, the countries in which programmes were implemented, and the targets that were set were all influenced by the PbR modality. In this post we explore how the PbR modality of the WASH Results Programme shaped the behaviour of the organisations who were implementing the programme, known as “suppliers”.

Continue reading

Lessons from implementing the WASH Results Programme #1: Intentions and key features

In this first post of a series, we look at what Payment by Results (PbR) in the WASH Results Programme was intended to achieve and key features of the programme.

arrow-2275729_1920

“All happy families are alike; each unhappy family is unhappy in its own way.”  Leo Tolstoy memorably declared in the opening lines of Anna Karenina. Without wanting to suggest that PbR contracts are unhappy, they are certainly all different. An understanding of what the programme was trying to achieve and the characteristics of the way in which PbR was applied are important when thinking about the effects of the PbR modality.

In this first in a short series of blogs exploring the effects of the PbR mechanism on a large scale WASH programme,  we explore what PbR was intended to achieve in this instance and key features of the programme.

Continue reading

Using Payment by Results to deliver WASH services at scale: lessons from the WASH Results Programme

Introducing our series of blog posts sharing lessons from implementing Payment by Results (PbR) to deliver WASH services at scale. 

shield-229112_1920Payment by Results is a form of Results Based Financing increasingly used by aid donors but opinions remain divided over its merits and limitations. Based on the evaluation of a large-scale water, sanitation and hygiene programme, this series of blogs explores one example of how this much-discussed funding mechanism has worked in practice.

The WASH Results Programme was one of the first large-scale applications of PbR in the WASH sector and within the Department for International Development (DFID). As such, the programme provides a rich source of learning about how PbR works at scale and how to use it in WASH in the future.

Blog posts in the series explore:

This series of posts draws on the findings of the evaluation of the first round of the programme (May 2014 to March 2018). Undertaken by the e-Pact consortium, the evaluation included document analysis, literature reviews, verification data, key informant interviews and supplier case studies.

For a more detailed exploration about how the PbR mechanism functioned within the WASH Results Programme, how it shaped supplier behaviour and implications for donors, please see the Executive Summary of the programme evaluation, available along with other publications from the evaluation, at the DFID Research for Development website.

 

Rewards and realities of Payment by Results in WASH

If you missed the WASH Results Programme’s panel discussion at World Water Week 2018, then this webpage on the conference organiser’s website is for you.  You can read details of the presenters, find out where the group discussions took us and the types of questions the panel was asked about PbR in WASH.

The conclusions from the session were:

  • There was some consensus that the data demands of PbR have served to strengthen suppliers’ data and monitoring systems, supported by an external verification system. This has led to better data on which to base evolving programming decisions.
  • Those designing WASH Programmes using other funding modalities could usefully consider how they too could incentivise investment in and use of monitoring data whilst managing risks.
  • The relatively limited experience of PbR in the WASH sector to date suggests a need for continued learning in this area if the funding modality is more widely adopted in future.

We’ll be sharing our reflections on the session, as the MV team for the WASH Results Programme, in a forthcoming blog post, but in the meantime please take a look at these reports by the programme’s Suppliers:

Selected key messages of our joint session (video by SNV)

Blog post on World Water Week, by SAWRP 

Related blog posts:

Oxfam’s experience with the SWIFT Consortium for Sustainable WASH in Fragile Contexts

Hard work pays off: access and service delivery results under a Payment by Results programme

WASH verification: in vino veritas, in aqua sanitas

 

 

WASH verification: in vino veritas, in aqua sanitas

Ahead of the WASH Results Programme’s session at World Water Week, Andy Robinson (Lead Verifier in the Monitoring and Verification Team for the DFID WASH Results Programme) looks at the importance of third party and internal verification in improving the quality and reliability of programme monitoring and evaluation.

‘In wine there is truth; in water there is health’. Sanitation is derived from sanitas, the Latin word for health; and veritas means truth, the root of the word verification – the process of establishing that something is true. Verification may be the missing ingredient in large-scale monitoring and evaluation systems – the wine that enables WASH programmes to hear the truth.

Despite the investment of considerable programme time, capacity and resources in the development and implementation of rural sanitation monitoring and evaluation systems, the reality is that many large-scale monitoring systems in developing countries do not work well. Large-scale sanitation monitoring systems rarely provide the detail that we need; and it can be hard to distinguish reliable data from unreliable. This is a huge sector problem: when you don’t know what is happening early enough, or where things are going wrong, it is hard to respond quickly or effectively.

Lessons learnt from the DFID WASH Payment by Results (PbR) programme suggest that verification may be an important part of the solution: both external (third party) and internal (within programme) verification appear to be important factors in improving the quality and reliability of programme monitoring and evaluation [1].

DFID contracted an independent monitoring and verification (MV) team to verify the results from the WASH PbR programme. PbR financing means that implementing agencies receive funding on the basis of verified results, which generates strong incentives for agencies to achieve results, and to provide solid evidence of these results to the MV team. As a result, agencies invest in more comprehensive and responsive programme monitoring and evaluation systems, which generate rapid and detailed information on programme performance and progress.

The MV team verifies the results based on the data and reports from the monitoring and evaluation systems set up by the implementing agencies. This arrangement requires the MV team to appraise how these systems work; to identify potential risks and weaknesses; and to review whether systems and processes are working as intended. In the multi-country SNV Sustainable Sanitation and Hygiene for All WASH Results Programme (SSH4A WRP), the verification role appears to have had a noticeable impact (see box).


Box 1     Impact of verification: both internal and external

Firstly, it has encouraged SNV to strengthen its monitoring and reporting systems. Household surveys are the main instrument used by SNV to evidence sanitation and hygiene outcomes. Third party verification confirmed that some survey enumerators did not do their work properly – perhaps by taking short cuts to save time, or failing to follow protocols correctly – and that some processes could be improved. The mobile-to-web survey approach used by SNV greatly facilitated the verification process, with GPS and time data available for household interviews, and toilet photographs where appropriate. The rich survey data facilitated a cost-efficient and largely remote verification process, with only a small sample of villages and households visited in person to check for systemic errors.

Many of the survey problems were identified at the start of the programme, and SNV worked to strengthen and professionalise its monitoring and reporting processes so that later surveys ran smoothly and efficiently. This brings us to the second impact of the verified PbR – the development of the verification process was transparent, with all checks discussed and agreed with SNV before use. In order to avoid any surprises during verification, SNV began to run similar checks on the surveys before verification. As a result, SNV spotted several monitoring issues early, and was able to take prompt action to remedy these problems.


The combined effect of the PbR imperative to achieve results, and the detailed verification of these results, have led to more professional and systematic approaches to monitoring and reporting. All of these have important effects on implementation – as more reliable data and more rapid feedback mechanisms encourage adaptive management.

The use and analysis of the monitoring data also strengthens the systems – data collectors know that someone will be analysing the data, someone else will be verifying the data, and that these data will form the basis of programme payments. The shift caused by the greater scrutiny and use of the monitoring data has been a game-changer. The value of reliable monitoring and evaluation systems is increasingly apparent, with programme managers and project staff now able to focus on the systems and processes that achieve sustained outcomes, rather than on the inputs and expenditures that drive conventional WASH programmes [2].

Verification has proved a useful tool to encourage more professional and systematic approaches to monitoring and evaluation, but it is only one small part of the wider challenge of achieving large-scale and sustained development outcomes – wine may bring out the truth, but turning truth into health is a much harder trick!

Suggestions for further reading: Visit e-Pact’s blog series, Learning about Payment by Results in Water, Sanitation and Hygiene (WASH) A blog by the team verifying and evaluating DFID’s Results Programme:  

[1] “Beyond a burden: what value does verification offer

[2] “The payback and pains of Payment by Results (part 1)“. 

About the Author: Andy Robinson is an independent water and sanitation specialist based in the French Alps. Since 1987, Andy has worked on the design, implementation, and evaluation of WASH programmes in Asia and Africa for a diverse range of clients. In the last fifteen years, Andy has been heavily involved in the promotion of improved sanitation and hygiene, working with governments, development partners, and communities in more than 20 countries in Africa and Asia.     

This post was originally published on the SNV website and is one of a series leading up to World Water Week 2018.

Going to #WWWeek? Come to the WASH Results session on Sunday morning and find out more about the realities of Payment by Results in WASH.