How regular outcome monitoring makes WASH programmes more effective

Measuring progress towards WASH outcomes is more difficult and more expensive than reporting outputs, but it is valuable tool for quality programming

Data on outcomes can support effective programming: MV team member Martha Keega assesses a latrine in South Sudan

Understanding outcomes – for example, the number of people using improved water and sanitation facilities, or practising desired hygiene behaviour – is the first step in understanding whether programming for water, sanitation and hygiene is effective, and whether the WASH sector as a whole is making progress towards achieving the SDGs. The experience of the WASH Results Programme has illustrated that regular monitoring can be effective in understanding progress, or lack thereof, towards outcomes and hence represents a valuable tool for quality programming

Because it typically relies on representative survey data, measuring progress towards outcomes is more difficult and more expensive than reporting outputs. While reporting of outcomes has not yet taken hold across programming in the WASH sector, there is a growing interest and ambition to do so. The DFID-funded WASH Results Programme, where assessing outcomes was incentivised by payment-linked outcome targets, is one example amongst several initiatives introducing sustainability checks, clauses and compacts in recent years.

The outcomes focus of the WASH Results Programme

The £112 million WASH Results Programme aimed to support poor people in 11 countries to access improved water and sanitation, and to practice improved hygiene. Three consortia (‘suppliers’) of non-governmental organisations (NGOs) were contracted by DFID in 2014 to undertake large-scale delivery of WASH in advance of the conclusion of Millennium Development Goals. This ambitious delivery goal was coupled with payment for outcomes – measured up to two years later – to encourage the continued use of water supply, latrines and handwashing at critical times.

In the WASH Results Programme, outcomes were measured through surveys that assessed access of water services and latrines, and handwashing at critical times. One supplier carried out five survey while the other two suppliers implemented two surveys, after one and two years of their implementation phases. Payment for outcome targets was contingent on an independent verification of the survey design, implementation and data analysis.

In our learning brief on outcome achievements in the WASH Results Programme, we shared data and related lessons around measuring sanitation and hygiene outcomes. Here, we focus on two aspects of outcome monitoring in WASH Results: 1) how regular and verified monitoring of outcomes contributed to more reliable data on programme achievements and 2) how this data helped making programming more effective.

Regular (and verified) outcome monitoring produces more reliable data

With payment contingent on achievement in the WASH Results Programme, it was vital that outcome data was reliable and verifiable. This meant suppliers had to really think through methods for measuring outcomes, and the evidence required to demonstrate results. Providing this wasn’t always straightforward; handwashing with soap is notoriously difficult to measure, for example, as discussed in this blog. The data had to be reliable, or suppliers risked losing out on payment.

All three suppliers reported that the programme’s emphasis on and scrutiny of outcome data led them to scrutinise their outcome monitoring systems and processes. For example, both the SWIFT and SSH4A consortia reviewed their survey design, sampling approaches, and enumerator trainings and closer supervision based on verification feed-back. Specific problems identified in verification reports were further discussed in joint supplier-verifier after action reviews and improvements checked through systems appraisals. One supplier also repeated household surveys in two cases when their results were not sufficiently robust.

How more frequent outcome monitoring makes programmes more effective

The evaluation of the WASH Results Programme  examined the link between outcome-level monitoring and the programme’s performance and found that regular outcome monitoring had a positive effect on the outcomes achieved. The biggest benefits were observed where suppliers introduced periodic outcome monitoring right at the beginning of the programme.

When outcomes are monitored regularly and data is reliable, suppliers can use it to understand what approaches are (not) working and where. This gives them the tools to use resources and capacity effectively. If a challenge arises, the programme’s course can be quickly corrected.

• All three suppliers found that up-to-date outcome monitoring data was vital for adapting programme delivery, and deciding what approaches should be scaled up.In the Democratic Republic of Congo, for example, the SWIFT consortium ran an outcome survey to identify villages with a lower performance so that they could dedicate more resources and consider their approach. In Uganda, the SSH4A programme re-prioritised sanitation programming activities based on survey data. For example, when upgrading of latrines was slow, the programme expanded and intensified messaging in these areas. In Mozambique, where survey data revealed that behaviour change messages had not translated into an increased presence of handwashing facilities, the programme overhauled its behaviour change communication approach.
  
• Regular monitoring at outcome-level can reveal important lessons and patterns that would not be captured otherwise. Surveys conducted by SNV and SAWRP found that use of handwashing facilities would often drop after 1-2 years. Both suppliers learned that households in some communities did not find tippy taps to be durable and easy to use. Based on this information, they were able to re-examine what handwashing solutions they promoted in those contexts. In addition SNV found that in areas where sanitation coverage was low to start with, there was often a period of rapid progress that then stagnated quickly without further input.  This trend had not been identified before.

• Disaggregated outcome monitoring for specific groups (e.g. women, people living with disabilities), helps implementers understand who is / is not being effectively reached by the programme. There are challenges in collecting disaggregated data but suppliers report that insights strengthened their programming. Further discussion on this can be found in the WASH Results Programme Learning Brief #3  : Reaching the vulnerable and those in fragile contexts with WASH services.

Fully understanding sustainability requires assessment 2-3 years after implementation

Ongoing outcome monitoring is extremely valuable. However, it may not be possible to identify within the life-time of a programme whether the outcomes achieved will be sustained into the longer term.

In the case of the WASH Results Programme, which generally measured outcomes only one and two years after implementation, we cannot fully assess from the data whether the efforts to put the necessary support systems in place have been successful. For example, rule of thumb suggests that most toilets last around two years before collapsing when not properly maintained, so slippage due to insufficient maintenance may not be captured by outcome monitoring in the lifetime of the programme. 

To fully understand sustainability and to understand how effective proxy indicators for sustainability are at predicting longer term outcomes, it would be valuable to revisit programme locations 2-3 years after programmes have finished to undertake post-implementation studies. SNV have undertaken a study of this type and plan to undertake more:

Assessing the Impact and Equity of an Integrated Rural Sanitation Approach: A Longitudinal Evaluation in 11 Sub-Saharan Africa and Asian Countries

Post-implementation impact studies could provide valuable additional learning that could help the sector progress towards its goal of achieving sustainable access for all.  As such they represent a valuable investment for donors committed to achieving the SDGs. 

This blog post is based on discussions at the WASH Results Learning event held virtually in May 2020 and attended by representatives of programme suppliers, DFID (now FCDO) and the Monitoring and Verification Team. It was co-written by members of the Monitoring and Verification team for the WASH Results Programme: Alena Cierna, Joseph Thompson, Kathi Welle and Catherine Fisher.

---

Strengthening government monitoring systems: what role for NGOs and PbR?

NGOs strengthened their monitoring systems during the WASH Results programme, but did this lead to benefits for government monitoring systems?

Photo by cottonbro on Pexels.com

In the WASH Results Programme, the data demands of the Payment by Results mechanism and the scrutiny of third-party verification, drove implementing NGOs to strengthen their monitoring systems.  The resulting monitoring data enabled implementing organisations undertake adaptive management, helped to identify trends in the sector and has contributed to the evidence base about what levels of outcomes can be expected in WASH programming. 

As the WASH sector increasingly turns its attention to systems strengthening and government responsibility for providing WASH services,  some observers have questioned the value of developing NGO monitoring systems in parallel to government monitoring systems. 

 This prompted us to reflect on the relationship between NGO monitoring systems and government monitoring systems in the WASH Results programme.  Strengthening government monitoring systems was not the focus of the WASH Results Programme,  however suppliers report that some government monitoring systems did benefit from engagement over the course of the programme. 

This blog outlines how.

Suppliers aligned their monitoring with government systems and shared their data. 
SNV’s SSH4A programme used existing government monitoring systems for their routine monitoring of project progress in Kenya, Nepal, Tanzania, and Zambia. They ensured core elements of the systems existed across all countries, and the routine monitoring data was then either entered into national Management Information Systems by government partners or by SNV’s Local Capacity Builders.    SWIFT also aligned their monitoring systems with those of government and transferred their independently collected data to the government systems.  The data will be kept and managed by the Ministry of Health, however without significant resources follow up and effective use of this data is not guaranteed.  

SAWRP found it challenging to incorporate the significant amount of data they collected into government systems. In Bangladesh the government monitoring system, under the Department of Public Health Engineering (DPHE), was limited, and had no mechanism to accept programme data. As such, no central institution now has the full database that was generated by the programme.  However, data was shared with and used at a sub-national level by Union Information and Service Centres, for example, data identifying poorer households and households with disabled members, was used by some local Union Chairmen to help direct social protection resources for the most vulnerable.

The  programme’s emphasis on accurate monitoring data prompted discussion with government on data quality, and improved understanding of approaches to data collection.
SNV found that despite their usage and improvement of government monitoring systems for routine monitoring, these systems could not be improved to the point where they were reliable enough to be used for payment by results (nor was this a key aim of the programme). The routine monitoring data were therefore complemented by more rigorous and reliable household surveys completed or commissioned by SNV to report against the payable results. These data were seen as particularly reliable because they were subject to independent verification, and were validated by government partners. In some cases, this opened up discussions with government on the robustness of their own data.     In Ethiopia, government partners initially thought the SNV data were incorrect as they were so different to their data. It took around three years before the government agreed their routine monitoring data had weaknesses, and they then made efforts to improve the government monitoring systems.

SAWRP also worked towards improving government understanding of approaches to data collection, including by undertaking quarterly joint monitoring visits with local government personnel. However, while these efforts were useful to encourage and support the achievement of sustainable outcomes, they were not closely linked to long-term government management and improvement of monitoring systems.

Suppliers developed, tested and modelled new approaches to monitoring, in some cases supporting their adoption by government 
SNV’s household surveys helped to generate interest in additional monitoring indicators, show how this data collection was possible, and in some cases (for example data on vulnerable groups) these indicators were added to government data. SWIFT worked with the DRC government to develop a separate section of the national databases to include the new classification of semi-urban communities. This was taken up by government partly due to SWIFT’s positive relationship with government partners cultivated over years.  

Insights

Weaknesses in monitoring systems is a chance for dialogue
Weaknesses in monitoring of WASH services does not only have to be a challenge to sustainability but can be an opportunity to engage with government institutions and decision makers to both contribute and coordinate better as well as support improvements.  Relationships and trust between government and NGOs are important in enabling these conversations.

Existing capacity of government monitoring systems is a key factor
Unsurprisingly, where government monitoring systems were well developed, it tended to be relatively simple for the programme to align their monitoring with government systems at programme outset, hand that data over to government, and improve the reliability of government data.  It was more challenging where systems were weaker for example there was lack of clarity over which Ministry was responsible.  However, even where systems are stronger it is often not clear whether data will be updated after the programme ends.

Collection and use of data by sub-national and non-state actors is also important
All suppliers supported sub-national actors to collect and use data. In Kenya, SWIFT worked with water utilities to adjust reporting, increase management capability and support expansion. Accurate data and knowing how to use that data was key to unlocking some of the blockages previously experienced. Being able to respond to breakages, improve meter readings and support management decisions was partly due to an improvement in the use of data. 

NGOs can play a useful role in developing, testing and demonstrating progressive monitoring systems .
A useful role for implementing partners is the development and demonstration of more progressive monitoring systems, which allow new indicators and data collection methods to be tested and refined in the programme area. Government officials often need practical demonstration of the benefits and ease-of-use of new indicators before they can be adopted by government. Understanding and communicating what the data is useful for, both within programmes and with government entities to support decision making is essential to bringing people on board.   

What role for PbR in strengthening government monitoring systems? 

Although strengthening government monitoring systems was not an objective of the WASH Results Programme,  all suppliers collaborated with government agencies in this area.  However, it is difficult to unpick the extent to which this was prompted by the PbR mechanism of the programme and how much was driven by supplier commitments to strengthen WASH governance. Lessons from other programmes are needed to further strengthen understanding of the role of PbR, either for NGO or government contracts can play in strengthening government monitoring systems .  

One such programme is an  DFID funded, PbR programme for improving rural water supply sustainability in Tanzania. The programme is innovative in that it is the first WASH PbR programme that DFID (globally) is implementing directly through a host government. 

Learning from that programme is emerging.  It  confirms that PbR programmes require “robust data to prove the attainment of results, hence there has been a strong focus on strengthening government functionality monitoring systems.” The programme has seen a focus on indicators, strengthening of data collection  and chances for discussion about the use of data. However, it warns that  strengthened monitoring systems alone may not lead to desired outcomes.    You can access learning from that programme here:

Using Payment by Results to Improve the Sustainability of Rural Water Supply Services in Tanzania

As WASH Results Programme concludes, this very different application of PbR in the WASH Sector will generate further learning about the potential of PbR to strengthen government monitoring systems.   We look forward to following its progress.  

This blog post was based on discussions at the WASH Results Learning event held virtually in May 2020 and attended by representatives of programme suppliers, DFID (now FCDO) and the Monitoring and Verification Team. It was co-written by members of the Monitoring and Verification team for the WASH Results Programme, Amy Weaving Joseph Thompson and Catherine Fisher.

---

Achieving sustainable WASH at scale: Lessons from an NGO-led Payment by Results Programme Webinar highlights

Does Payment by Results encourage or limit innovation? Why is adaptive management so important for  sustaining outcomes in WASH programmes?  Why is it so difficult to set targets for handwashing behaviour?  Just  few of themes explored when  over 200 WASH specialists from around the world joined WASH Results Programme representatives for at a RWSN webinar and summarised here.

The webinar recording can be accessed from the RWSN website

A lively session chaired by Stef Smits from IRC, featured representatives from the WASH Results Programme implementing organisations, donor and third-party verifier:
John Dean,  Plan International, UK, SAWRP consortium
Hebdavi Muhindo,  Tearfund DRC, SWIFT Consortium
Antoinette Kome,   SNV, Sustainable Sanitation Hygiene for All
Leonard Tedd, from Foreign and Commonwealth and Development (FCDO) office, formerly DFID.
Katharina Welle,  Itad, Monitoring and Verification provider

If you missed the webinar, the recording is available online. Here, we summarise key points made during the discussion and provide links to further information from the programme shared during the webinar.   We’d like to thank all participants for attending and for sharing such interesting comments and questions

Summary of discussion

Outcomes achievements of the WASH Results Programmes
The initial presentation shared details of baseline and endline levels of access to sanitation and hygiene indicators from comparable data across the programme.   Insights shared were:
Sanitation: High levels of access to sanitation were maintained in many contexts, low baseline is not a barrier to high achievement but contextual factors were the main determinant.
Hygiene: High levels of knowledge about handwashing at critical times were maintained, low presence of soap / as was the limiting factor in sustaining handwashing with soap in practice.

Defining, monitoring and achieving hygiene outcomes:  
Unlike outcomes targets for sanitation and water, hygiene outcomes targets were set low, in keeping with the fairly limited evidence base at the time. Those targets were almost universally overachieved. In addition the focus on hygiene targets led to the adoption of approaches to measurement of hygiene outcomes, comprising combinations of indicators around knowledge, presence of handwashing facility and soap/ash and reported or observed behaviour.

“In the second phase of the programme, we changed our hygiene indicator to have an ambitious target and also a more meaningful one as it looked  not only at knowledge but at presence of handwashing facility and soap….Setting clear, measurable outcomes targets were key.”   

John Dean, SAWRP

For further info see Learning Brief:   #1 Outcome achievements in the WASH Results Programme: data and insights 
Further data on outcomes achievements within each country in the the SSH4A programme can be found on the SNV website

How the PbR worked in practice : Risks and incentives

Linking payment to outcomes incentivised implementers to focus on sustained use and behaviour change:  this resulted in changes to how programmes were implemented with a greater focus on sustained outcomes, as explored below.   How incentives were shared between partners is complex but usually the PbR mechanism was not passed on to local partners in this programme.  Lessons from a PbR mechanism with government are emerging from a different PbR-led programme in Tanzania’s rural water supply sector, shared by one of the participants – see

PbR used in the WASH Results Programme  puts the financial risk in the hands of whoever implements the programme:  if implementing organisations did not achieve targets, they did not get paid.  This deterred some organisations from tendering and meant that implementing organisations had to manage risk carefully, particularly in setting realistic, evidence-based and achievable targets. 

The budget flexibility in PbR enables local level innovation through adaptive management;  not having to negotiate budget and activity changes with donors gives implementors the flexibility to adapt and respond to what is working or not working in practice and innovate locally.  However, roll out of radical innovation at scale is too risky under PbR.

Its not re-inventing toilets or doing something new at-scale that you have never done before;  the risks are too great if you are not sure they are going to work….  This is about all the innovations that come from the field, adjusting strategies at a local level. Everyone was focused on how to achieve and sustain the results…continuously measuring what is working or not.” 

Antoinette Kome, SSH4A

Further info see Learning Brief #2  Setting and monitoring outcome targets in WASH programmes 
Evaluation findings about the influence of PbR on supplier behaviour

Ensuring WASH outcomes and sustainability

Staying beyond the implementation phase was important  for sustainability and resilience: 
In many WASH programme, implementers leave after initial programme implementation;  in this programme,  payments linked to outcomes incentivised ongoing engagement with communities after the initial phase of the programme.  For example in Eastern DRC,  a water scheme serving 30,000 people was handed over to the community, ongoing engagement by the SWIFT programme showed that there was need for ongoing accompaniment and support to the community to help take charge of the system that has been handed over to them.  

Timeframes were important for sustainability of outcomes:  in addition to enabling sustained engagement,  the six/seven-year programme timeframe also enabled several cycles of learning and adaptation and the collection of data over long periods of time.   SNV has undertaken a retrospective survey after having exited areas that will illustrate levels of slippage.   

Flexibility, learning and adaptation were important for sustainability:   local level adaptation of approaches and trying new things, from behaviour change communication to testing new technologies  were important in ensuring sustained outcomes. The flexibility of the PbR enabled this.   Baseline data, regular monitoring of outcome data and learning cycles helped identify where there were problems.

“the budget flexibility of this programme meant we could be learning continuously, adapting and trying new things as we go along… we are not afraid to say hey this isn’t working any more lets, try something different,  or ‘ I got this feedback from beneficiaries and therefore we can adapt and do this differently.”…   “Defined budget lines in typical grant projects mean the activities are pre-defined is a blockage to innovation”

Hebdavi Muhindo, SWIFT

NGOs as third-party actors can contribute to systems strengthening:   working with stakeholders (government, private sector and community)  to strengthen systems was a key feature of implementers’ approaches to ensuring sustainability.  However, NGOs can only contribute to sustainability of outcomes; there are many other factors at play, you cannot attribute the sustainability or non-sustainability of outcomes to the NGO that has been working there.

Further info, see Learning brief #4: Experiences in WASH systems strengthening

Equity and inclusion

To ensure inclusion, PbR programmes need to avoid perverse incentives that encourage “cherry picking” the easiest to reach:  in this programme, this risk was mitigated by NGOs committing to work in specific geographic areas and by the implementing organisations’ commitment to serve the poorest communities and embeddedness in the communities in which they work.  However, payments were not linked to area-wide achievements but to the number of beneficiaries reached.

Building equity and inclusion into PbR programmes needs to be balanced with simplicity:  one approach to hard-wiring equity into PbR frameworks is  pre-defining specific target groups and setting disaggregated targets.  This requires information about numbers of people within specific potentially vulnerable groups and expected success rate for different groups and requires considerable research which has cost and resource implications. An alternative approach is setting area-wide targets that outline not just how many people are reached but what percentage of people are reached within programmes.   Disaggregated monitoring is important for monitoring progress and reach, even if not linked to payment.

For further info, see Learning brief #3 Reaching the vulnerable and those in fragile contexts with WASH services

Final words from the UK Foreign and Commonwealth and Development Office

Leonard Tedd from the UK Foreign and Commonwealth and Development Office said a few words about the new organisation. He reiterated the UK strong commitment to ODA, acknowledging there will be resource constraints linked to contraction of the economy.  Regarding WASH funding, FCDO is currently supporting large COVID response programmes, many of which have a WASH element.   A FCDO priority is to strategically shift to working more on systems, both in hygiene and in WASH.     

As the funder of the WASH Results Programme, Leonard thanked all those involved for their hard work in achieving such huge results,  particularly the staff from implementing partners working in the 11 countries in which the programme was implemented.  

Everyone on the WASH Results Programme, hope the lessons from this programme will be useful for others.   Please explore learning from the WASH Results programme using the links below.  

Further resources from the WASH Results Programme

#1 Outcome achievements in the WASH Results Programme: data and insights 
What levels of access and behaviour change were maintained two years after initial engagement with communities? This brief shares compiled sanitation and hygiene outcomes achievement data from 11 countries and identifies insights for the sector

#2  Setting and monitoring outcome targets in WASH programmes 
This brief focusses on lessons for design of large-scale WASH programmes, exploring both using Payment by Results mechanisms and approaches to promoting sustained WASH outcomes. 

#3 Reaching the vulnerable and those in fragile contexts with WASH services

 This brief shares reflections from the three suppliers on the WASH Results Programme on their work towards equity and inclusion in WASH.

#4: Experiences in WASH systems strengthening
This brief explores some of the approaches to strengthen systems for sustainable WASH services: working with local sanitation entrepreneurs; with local government to support inclusive, district-wide approaches to service delivery; and professionalising water user committees  

Previous blog posts

Designing effective verification systems for Payment by Results contracts
Series of publications sharing practical guidance for third-party verification services

WASH verification: in vino veritas, in aqua sanitas  – reflections from Lead Verifier Andy Robinson on the role of verification in strengthening large scale M&E systems.

Hard work pays off: access and service delivery results under a Payment by Results programme – reflections from Anne Mutta, from implementing organisation SNV

WASH Results Programme Evaluation: findings and recommendations for the WASH sector  summary of the programme evaluation undertaken by e-Pact

“Truly exceptional? Handling misfortune within Payment by Results” this blog post from 2018 explores risk sharing in the context of exceptional events: 

Working at scale in the WASH sector: lessons from the WASH Results Programme Blog from Richard Carter and Jeremy Colin, based on the programme evaluation

Selected publications from implementing organisations

For more info on SNV’s work on leaving no one behind, see:   Sanitation and hygiene for all: a comparative study of approaches to leaving no one behind across five countries

For details of SNV approach to monitoring outcomes see SNV (2019) Sustainable Sanitation and Hygiene for All (SSH4A) Performance Monitoring Framework Part 2. Outcome indicators 

For details of the ASUREP model, see The ASUREP: a promising water management model in the DRC and Research on the ASUREP model: interview with Ian Langdown from ODI

For more information on SAWRP approach to behaviour change communication see https://plan-uk.org/about/our-work/healthcare-and-clean-water/clean-water-and-sanitation/sawrp-behaviour-change

---

Delivering sustainable WASH outcomes: insights from the WASH Results Programme

New series of learning briefs from this large-scale programme share data on outcome achievements, insights on designing outcome focussed programmes and strategies adopted toward inclusivity and sector strengthening.

A safe latrine under construction in Kenya (SNV Kenya/Admedia)

As the WASH Results Programme drew to a close after seven years, representatives of suppliers, DFID/FCDO and the third-party monitoring and verification supplier reflected on what has been learned from the final, outcomes-focused years of the programme.    

WASH Results Programme key facts and achievements

The following learning briefs explore the WASH Results Programme’s achievements in delivering WASH outcomes (#1) and lessons about programme design (#2)  and share suppliers efforts in working towards sustainable WASH particularly the dimensions of equity and reaching the potentially vulnerable (#3) and institution strengthening (#4)

#1 Outcome achievements in the WASH Results Programme: data and insights 
What levels of access and behaviour change were maintained two years after initial engagement with communities? This brief shares compiled sanitation and hygiene outcome achievement data from 11 countries and identifies insights for the sector.

#2  Setting and monitoring outcome targets in WASH programmes 
This brief focusses on lessons for design of large-scale WASH programmes, both in using PbR mechanisms and promoting sustained WASH outcomes.

#3 Reaching the vulnerable and those in fragile contexts with WASH services
This brief shares reflections from the three suppliers on the WASH Results Programme on their work towards equity and inclusion in WASH.

#4: Experiences in WASH systems strengthening
This brief explores some of the approaches to strengthening systems for sustainable WASH services: working with local sanitation entrepreneurs; with local government to support inclusive, district-wide approaches to service delivery; and professionalising water user committees.

Join us to discuss these lessons at a forthcoming RSWN webinar

Achieving sustainable WASH at scale: lessons from an NGO-led Payment by Results Programme Tuesday 10 November 14.30 CET
Register here:  https://bit.ly/2Iq16Ff

Hear directly from implementers, donors and verifiers at a webinar on Tues 10 Nov
register here

You can hear directly from those involved in the programme: Hebdavi Muhindo, SWIFT, Antoinette Kome, SNV,  John Dean, SWARP, Katharina Welle:  Monitoring and Verification team and Leonard Tedd FCDO.

If you have questions on the papers – please do share in the comments box and we hope to see you on Thursday 10 November. Why not register now?

Designing effective verification systems for Payment by Results contracts

New series of publications shares practical guidance for third-party verification services based on learning from the WASH Results Programme.

What do you need to consider when commissioning and designing a verification system? How do you go about creating a verification cycle, appraising monitoring systems or choosing good indicators? E-Pact has published a set of guidance notes to help others to design and implement an effective verification system, drawing on our experience from 2014-2020 as the Monitoring and Verification team for the DFID WASH Results Programme.  

WASH Results is one of DFID’s flagship programmes in using Payment by Results (PbR) and with a £112 million budget is one of the first large-scale applications of PbR in the sector. The programme was conceived in 2013 during an era where DFID shifted programming towards longer-term results. An important rationale for the programme was to incentivise large-scale delivery of WASH under the Millennium Development Goals in 2015. This ambitious delivery goal was coupled with payment for outcomes to encourage the continued use of water supply, latrines and handwashing at critical times, in line with the SDG agenda.

Delivered by three consortia of non-governmental organisations (‘suppliers’), the programme has enabled over 1.6 million people to gain access to water, 7.4 million access to sanitation, and reached 16.1 million with hygiene promotion across 11 countries. The programme has also overwhelmingly achieved its outcome targets.

WASH Results operates under a ‘100 percent’ PbR modality, whereby suppliers receive payments only on the successful verification of their results by a third-party Monitoring and Verification (MV) team. The verification was systems-based, meaning that Itad in collaboration with Ecorys and Iwel under the e-Pact consortium, independently verified the data reported by the suppliers. In this process, supplier monitoring and reporting systems are appraised and data they produce are verified by the third-party MV team to confirm they are accurate and realistic. This approach has benefits including value for money and those derived from strengthening suppliers’ own monitoring systems. This ‘systems-based’ approach to verification was completely novel at the time and over the past seven years, the verification team tested and refined its methods and processes and learned lessons, some of which are captured in our short series of publications.

The publications are is intended to support organisations when commissioning, managing or implementing third-party verification services as part of PbR contracts and those interested in strengthening monitoring systems through third parties. The Guidance Note provides overarching lessons, useful at an early stage of thinking through the design of a PbR programme while the accompanying Verification in Practice Notes dive more deeply into key aspects of systems-based verification.

The DFID Payment by Results Guidance Note: Lessons from an effective verification system [PDF] provides 12 lessons to inform the creation of a verification system. It covers three aspects: 1) getting the foundations right at design and tender stage, 2) designing the verification system so it is effective and efficient and avoids surprises when payment decisions are being made; and 3) the role of commissioners in a PbR contract compared to traditional grants.

MV team member Martha Keega assesses a latrine in South Sudan

Verification in Practice #1: The Verification Cycle – Step by Step [PDF] In a PbR programme, suppliers only get paid when results have been independently verified and this requires a verification process that delivers good quality data in a timely and predictable manner. The MV team has put this into practice through a nine-step, three-month verification cycle, which is outlined in this note.

The WASH Results Programme’s verification cycle is explored in Verification in Practice #1

Verification in Practice #2: Appraising Monitoring Systems [PDF] A key part of a “systems-approach” to verification is the appraisal of supplier monitoring systems, known within the WASH Results Programme as a “systems appraisal”. This note explains how the MV team defines a systems appraisal, what it involves and why one can be useful. This will be of relevance to all those seeking to strengthen monitoring systems, not just those in PbR programmes.

Verification in Practice #3: What makes a good indicator for a Payment by Results programme? [PDF] Good quality indicators are essential to effective PbR programmes. This practical note suggests seven essential characteristics of a good PbR indicator and five additional characteristics that may be desirable.

We hope that this guidance drawn from our experience will be useful to you if you are commissioning or designing a verification system or just want to gain confidence in the reporting of results regardless of whether or not you decide to attach payment to it. After seven years and more than 50 rounds of verification we feel that we’ve got a bit of a handle on this, so please do get in touch if you have any questions and we encourage you to share your thoughts in the comments.

---

 

Working at scale in the WASH sector: lessons from the WASH Results Programme

Extensive evaluation of a four-year DFID funded WASH programme reveals lessons for future large-scale programmes.

Delivering WASH at scale – lessons from the WASH Results Programme. Photo by Kelly Lacey from pexel.com

The WASH Results Programme was a large-scale programme undertaken as part of the UK Government pledge to ensure that 60 million people would gain access to water, sanitation and hygiene (WASH) services by December 2015 as a result of UK investments.

The Department for International Development (DFID) funded WASH Results Programme worked in 11 countries, and aimed to provide 1.2 million people with access to clean drinking water sources, 4.3 million people with access to an improved sanitation facility, and reach 9.3 million people with hygiene promotion activities. By most measures the programme was very successful in achieving its stated objectives. The vast majority of output-level targets were achieved by all suppliers, and in many areas with significant overachievement. The suppliers also overwhelmingly achieved the outcome-level targets, with significant overachievement in several areas and modest underachievement in relatively few others.

The WASH Results Programme had two defining characteristics: the fact that it was a Payment-by-Results (PbR) programme, and its size and structure. The design and effects of the PbR modality are explored in earlier blog posts.

This blog post, based on a paper by Richard Carter and Jeremy Colin, draws on the mid-term and end-point evaluations of the WASH Results Programme to reflect on a number of aspects of working at scale in the WASH sector. In particular it explores:

• How the WASH Results Programme and other WASH sector actors have conceptualised scale.
• The outcomes and impacts set out in the WASH Results Programme design, and the extent of their achievement.
• What requirements were placed on suppliers, in the pursuit of programme outcomes and impacts at scale.
• How inclusive and sustainable services and behaviour changes at scale can best be pursued in future WASH programmes.

This blog post is intended for governments, international agencies, NGOs and development funding partners which wish to bring about beneficial, inclusive and sustainable changes in water, sanitation and hygiene at scale. The evaluation that this post is based on is available from the DFID Research Portal as a Synthesis Report, Executive Summary and Annexes. 

Aspects of working at scale in WASH sector

The wider literature on scale, within the WASH (eg Taylor, 2013[i]), health (eg Mangham and Hanson, 2010[ii]; Gargani and McLean, 2017[iii]), education (eg Coburn, 2003[iv]; Burns, 2014[v]) and food security (Frake and Messina, 2018[vi]) sectors, understandably focuses more on “scaling-up” than on scale per se. The two ideas are linked, however, in the question “assuming we know how to deliver services at small scale, how can those services be extended or accelerated to reach large numbers?” Cynthia Coburn provides a helpful four-dimensional understanding of scale.

Understanding “scale” (Coburn, 2003)

Coburn (writing about the education sector) highlights four necessary dimensions of scale: 1.    Depth. The nature and quality of services, “change that goes beyond surface structures or procedures to alter … beliefs … norms … and principles (as enacted in the curriculum)”. 2.    Sustainability. “The distribution and adoption of an innovation are only significant if its use can be sustained … there is ample evidence that sustainability may be the central challenge of bringing reforms to scale”. 3.    Spread. “ … the importance of taking into consideration what is spread … but also … influencing district policies, procedures and professional development”. 4.    Shift in reform ownership. “ … ownership over the reform must shift so that it is no longer an external reform, controlled by a reformer, but rather becomes an internal reform …”.

The significance of ‘scale’ in WASH programming has different implications in each of the sub-sectors. For example water services are more likely to be financially sustainable if the customer base is large and diverse, so that wealthier market segments (enjoying high levels of service) can subsidise less wealthy segments with more basic services. In regard to sanitation, beneficial health impacts are only likely to emerge as high levels of actual use are realised – reducing open defecation and other forms of unsafe disposal of human excreta to the minimum. Hygiene practices too need to reach high levels of compliance in order to provide a basis for health impacts.

Linked to all of the individual sub-sectors is the significance of the programme’s overall scale with regards to the geographical and administrative boundaries in which it is operating. Administrative boundaries are particularly significant with regards to achieving sustainability and capacity building aims, and programme’s that align themselves with the administrative levels that have responsibility for maintaining services enhance the prospects for sustainability. In the rural context this often entails taking a ‘district-wide approach’ to delivering services, and in the urban context focusing on city-wide service delivery. This view of scale is also significant with regards to equity as by aiming to reach everyone with services in a given area, by definition, implies a focus on equitable service delivery.

Scale in the WASH Results Programme

Scale was defined in terms of beneficiary numbers

In the WASH Results Programme, DFID called for suppliers which could deliver results at scale. No explicit definition of “scale” was provided, although the programme aimed to reach 1.2 million people with access to clean drinking water sources, 4.3 million people with access to an improved sanitation facility, and 9.3 million people with hygiene promotion activities.  “Scale” was effectively defined therefore in terms of beneficiary numbers.

In the WASH Results Programme, the suppliers did not formally define scale, but they pursued it in different ways, some working across multiple countries and districts, partially filling the gaps left by other government and non-government agencies; and others working in more focused locations (e.g., in certain cities or geographical areas).

The WASH Results Programme aimed to achieve scale by bringing services to millions of people. Alternative interpretations of scale in WASH programming, such as focusing on improving the effectiveness and reach of government interventions; establishing national funds which could be accessed by local governments; or advancing sector reforms and fiscal decentralisation were not pursued.

The programme was phased: an initial focus on outputs was followed by an outcomes phase

The WASH Results Programme consisted of two ‘phases’, and this had an important influence on suppliers’ programming. The output phase ran from April 2014 until December 2015 (the MDG deadline). During this phase, services were brought to the target populations and approximately 80% of the budget was disbursed. In the outcome phase (January 2016 to early 2018) the reach and sustainability of the services provided in the first phase were to be consolidated. The main argument in favour of the WASH Results Programme’s phasing was the fact that suppliers would have a real opportunity, post-implementation, to focus entirely on equity and sustainability issues. One of the three suppliers in the WASH Results Programme effectively ignored the phasing, due to having a more comprehensive framework for addressing sustainability. Though for the other two suppliers the pressure in the output phase to deliver results resulted in comparatively less attention being paid to equity and sustainability issues earlier in the programme.

The issue of phasing is significant for future programmes that seek results at the output and outcome level and as it is possible that a similar arrangement could occur in future as the SDG deadline approaches or in response to the WASH challenges posed by the COVID-19 pandemic. In particular, that the pressure to deliver output-level results should not take precedence over sustainability or equity considerations.

Delivering at scale – key outcomes and impacts sought under the WASH Results Programme

Inclusion. The WASH Results Programme did not set inclusion-related targets; though many suppliers nonetheless intended that all members of target communities should benefit from their interventions. In view of the public good (and public health) aspects of WASH specifically, this was important and necessary, even beyond the human rights issues involved. Although DFID did not explicitly require suppliers to undertake “total sanitation” interventions (i.e. those in which entire communities access and use safe sanitation), this was in fact the strategy pursued by most suppliers, in line with their usual practices. Local-level “scale” (i.e. near-complete coverage of services in targeted communities and areas) is essential in WASH programmes of any size.

Sustainability. The WASH Results Programme intended to deliver sustainable services and hygiene practices. DFID required suppliers to address the various aspects (functionality of systems, institutional aspects, financial and environmental matters) of sustainability, and it drew attention of suppliers to the need to engage both with communities and with local and national governments in the pursuit of sustainability.

Impacts. DFID’s intended impact was that “poor people have improved health (as a consequence of using safe and sustainable WASH services)”. The indicators of that improved health were to be reduced incidence of diarrhoea and reduced child mortality. Health impacts were not measured, either by suppliers or by the programme evaluation team.

As a statement of impact, this was simultaneously both highly ambitious and very limited. It was ambitious in the sense that the direct impact of WASH programming on diarrhoeal disease is frequently less than hoped for; indeed rigorous studies taking place at the time of the WASH Results Programme were finding such impacts to be elusive. It was also limited however, in the sense that potential health impacts extend well beyond the reduction in diarrhoea (to include a number of neglected tropical diseases – or NTDs – and possible reductions in stunting), and the wider potential impacts extend well beyond health, as attested by DFID’s own Evidence Paper on WASH Impact (Cairncross et al, 2013). With the exception of interventions which were scaled to be truly community-wide or area-wide (i.e. in which all or nearly all inhabitants of communities or administrative areas practise safe sanitation and good hygiene), the impacts on diarrhoeal disease as a result of the WASH Results Programme were likely to have been modest.

In future WASH programmes, a more explicit focus on community-wide use and practice, and on sustainability of services into the indefinite future would bring about a greater likelihood of such impacts being achieved – although current research evidence suggests that even these measures may not necessarily lead to major health impacts.

In addition to the requirements set out in DFID’s Business Case and Invitation to Tender, a number of other requirements were placed on suppliers, especially in regard to sustainability, in-country capacity, and disaggregated monitoring of results.

However, a number of potential requirements were not explicitly made; this was in part due to the programme having a rather rudimentary theory of change, in which key assumptions were neither made explicit nor tested. Had they been, then there is reason to believe that the WASH Results Programme’s outcomes and impact may have been even greater.

Delivering WASH results at scale: recommendations for future programmes

DFID’s emphasis on outcomes (inclusion and sustainability), and its requirement for robust and disaggregated monitoring, are important for WASH programmes at any scale.

A number of additional recommendations can be made for future programmes:

• Make explicit that all interventions should be undertaken community-wide and not simply in pursuit of numbers of beneficiaries. In the event, the suppliers (all reputable INGOs) carried out their programming in this way, but this cannot be assumed.
• Where possible, work to achieve full coverage across administrative units such as districts. This would further enhance the likelihood of health impacts, as well as facilitating further scaling-up within the country.
• Sanitation and hygiene interventions are likely to be more effective where water services are already adequate. In cases where these services are not adequate; improvements in water services are necessary too.
• Carry out hygiene promotion programmes of sufficient reach and duration to effect significant behaviour change. In the WASH Results Programme, suppliers were expected to increase hand-washing by 15% compared to baseline through their hygiene promotion efforts. The level of this target would appear limited in ambition and certainly insufficient to bring about significant health impacts. Furthermore, the assumption made in parts of the WASH Results Programme that Community Led Total Sanitation triggering included sufficient hygiene promotion, is unsafe.
• Use the influence of large-scale budgets to put pressure on national governments to implement fiscal decentralisation more fully. The inability of local governments to provide effective post-implementation support to community-level services is due in no small measure to inadequate budgets.

Defining and delivering scale in WASH programming requires attention not only to the numbers of people served, but to the quality and sustainability of services; providing services which are used by very high proportions of the populations of served communities; and achieving a real transfer of ownership from externally-delivered programming to government-ownership of scaling-up processes.

A general lesson from the WASH Results Programme is that outcomes around sustainability, equity and inclusion need to be designed in and pursued from the outset. It is too late to address matters of equity and sustainability if they have been neglected in the pursuit of scale defined solely or mainly in terms of numbers of people reached.

Delivering WASH outcomes at scale is challenging. While it is relatively easy to provide many people with access to services, the more difficult task of ensuring changed use and behaviour takes different skills, more time, and sustained engagement. The transfer of ownership of that engagement with communities, from external suppliers to local government – as envisaged in a full conceptualisation of scale – is particularly difficult when those local governments lack capacity, and crucially, resources.

Given the present uncertainties in predicting health impacts consequent upon WASH interventions, those funding WASH programmes should exercise caution in articulating such intended impacts. However, an important prerequisite for such impacts is community-wide and area-wide use of safe sanitation and of good hygiene (especially handwashing) practices. An explicit and thorough Theory of Change can help in the specification of future programme designs, and of full sets of requirements for implementing partners.

This blog post is adapted from an original paper by Richard Carter and Jeremy Colin,  both members of the OPM evaluation team.

The evaluation on which this post is based is available from the DFID Research Portal and is summarised here:   WASH Results Programme Evaluation: findings and recommendations for the WASH sector

---

[i] Taylor B (2013) Effectiveness, scale and sustainability in WASH programmes – a review.  Springfield Working Paper Series (2), The Springfield Centre, Durham.  Available at: http://www.springfieldcentre.com/wp-content/uploads/2013/10/2013-10-Effectiveness-Scale-and-Sustainability-in-WASH.pdf

[ii] Mangham L J and Hanson K (2010) Scaling up in international health: what are the key issues?  Health Policy and Planning, Vol 25 pp85-96.  Available at: https://www.ncbi.nlm.nih.gov/pubmed/20071454

[iii] Gargani J and McLean R (2017), Scaling science.  Stanford Social Innovation Review, Fall 2017.  Available at: https://ssir.org/articles/entry/scaling_science

[iv] Coburn C E (2003) Rethinking scale: moving beyond numbers to deep and lasting change.  Educational Researcher, Vol 32, No. 6, pp 3-12.  Available at: https://www.sesp.northwestern.edu/docs/publications/139042460457c9a8422623f.pdf

[v] Burns M (2014) the myths of scaling-up.  Centre for Education Innovations.  Blog available at: https://educationinnovations.org/blog/myths-scaling

[vi] Frake A N and Messina J P (2018) Toward a common ontology of scaling-up in development.  Sustainability Vol 10, pp 835-845.  Available at: https://www.mdpi.com/2071-1050/10/3/835

Answering questions on the WASH Results Evaluation

Over 180 people from NGOs, funding agencies and research organisations around the world with an interest in WASH and Payment by Results participated in a panel discussion last week on the evaluation findings of the WASH Results Programme.

Participants grilled the panellists on issues of sustainability, sector strengthening and the role of monitoring in WASH programmes.   If you missed the webinar, you can watch it online here.

After a short presentation of the key findings, the panelists shared their experience of the programme and responded to questions from the audience which focused largely on:

• The nature of the design of the Payment by Results modality and how the implementing organisations responded.
• Whether the WASH Results Programme was good value for money.
• The sustainability of the programme’s results.
• How the WASH Results Programme contributed to systems strengthening in the countries in which it operated.
• The role of monitoring in the programme.

The questions prompted lively debate during the Q&A session and continued in the webinar chat. 

For those who were unable to attend the live webinar, it can be viewed here with a record of the Chat. The synthesis report and other publications from the evaluation are available to download from the DFID website.

If you would like to comment on the evaluation findings or share your experience of comparable programmes, you can do so in the series of blog posts summarising the findings  :  Using Payment by Results to deliver WASH services at scale: lessons from the WASH Results Programme . 

 

 

WASH Results Programme Evaluation: findings and recommendations for the WASH sector

Evaluation of large-scale WASH programme reveals Payment by Results (PbR) is a viable option for funding WASH at scale, if used carefully.

Photo by Nitin Sharma on Pexels.com

This blog summarises findings of the evaluation by the e-Pact Consortium of the UK Department for International Development’s (DFID’s) Water, Sanitation, and Hygiene (WASH) Results Programme: a four-year programme (May 2014 to March 2018), which aimed to bring equitable and sustainable water and sanitation services and hygiene practices to millions of people in 11 countries. The programme was implemented by three suppliers contracted to DFID under a PbR financing modality (SAWRP, SWIFT and SSH4A).

As one of the first large-scale applications of PbR in the WASH sector and within DFID, the WASH Results Programme provides a rich source of learning about how PbR works at scale and how to use it in WASH in the future. Based on the evaluation, this blog explores the following areas:

1. Whether the programme successfully achieved its stated objectives;
2. The influence of programme design, including the PbR modality, on this achievement;
3. Lessons for applying PbR in WASH programming going forward;
4. Recommendations for future WASH PbR programmes.

This blog is an abridged version of the Evaluation Synthesis, which is available to download from DFID’s Research for Development online repository along with reports from all of the component studies of the evaluation.

Lessons from implementing the WASH Results Programme #3: Implications for funders

Our final blog post in a series, sharing lessons from implementing Payment by Results (PbR) to deliver WASH services at scale draws out implications for funders when designing future programmes.

The evaluation found that PbR appeared to be a viable option for funding WASH programmes but with some important design considerations that we explore in this blog post.

Lessons from implementing the WASH Results Programme #2: The influence of PbR

Our second blog post sharing lessons from implementing Payment by Results (PbR) to deliver WASH services at scale explores the influence of the PbR modality on suppliers’ behaviour. 

The shape of implementing consortia, the countries in which programmes were implemented, and the targets that were set were all influenced by the PbR modality. In this post we explore how the PbR modality of the WASH Results Programme shaped the behaviour of the organisations who were implementing the programme, known as “suppliers”.

Our second blog post sharing lessons from implementing Payment by Results (PbR) to deliver WASH services at scale explores the influence of the PbR modality on suppliers’ behaviour. 

The shape of implementing consortia, the countries in which programmes were implemented, and the targets that were set were all influenced by the PbR modality. In this post we explore how the PbR modality of the WASH Results Programme shaped the behaviour of the organisations who were implementing the programme, known as “suppliers”.