WASH verification: in vino veritas, in aqua sanitas

Ahead of the WASH Results Programme’s session at World Water Week, Andy Robinson (Lead Verifier in the Monitoring and Verification Team for the DFID WASH Results Programme) looks at the importance of third party and internal verification in improving the quality and reliability of programme monitoring and evaluation.

‘In wine there is truth; in water there is health’. Sanitation is derived from sanitas, the Latin word for health; and veritas means truth, the root of the word verification – the process of establishing that something is true. Verification may be the missing ingredient in large-scale monitoring and evaluation systems – the wine that enables WASH programmes to hear the truth.

Despite the investment of considerable programme time, capacity and resources in the development and implementation of rural sanitation monitoring and evaluation systems, the reality is that many large-scale monitoring systems in developing countries do not work well. Large-scale sanitation monitoring systems rarely provide the detail that we need; and it can be hard to distinguish reliable data from unreliable. This is a huge sector problem: when you don’t know what is happening early enough, or where things are going wrong, it is hard to respond quickly or effectively.

Lessons learnt from the DFID WASH Payment by Results (PbR) programme suggest that verification may be an important part of the solution: both external (third party) and internal (within programme) verification appear to be important factors in improving the quality and reliability of programme monitoring and evaluation [1].

DFID contracted an independent monitoring and verification (MV) team to verify the results from the WASH PbR programme. PbR financing means that implementing agencies receive funding on the basis of verified results, which generates strong incentives for agencies to achieve results, and to provide solid evidence of these results to the MV team. As a result, agencies invest in more comprehensive and responsive programme monitoring and evaluation systems, which generate rapid and detailed information on programme performance and progress.

The MV team verifies the results based on the data and reports from the monitoring and evaluation systems set up by the implementing agencies. This arrangement requires the MV team to appraise how these systems work; to identify potential risks and weaknesses; and to review whether systems and processes are working as intended. In the multi-country SNV Sustainable Sanitation and Hygiene for All WASH Results Programme (SSH4A WRP), the verification role appears to have had a noticeable impact (see box).

---

Box 1     Impact of verification: both internal and external

Firstly, it has encouraged SNV to strengthen its monitoring and reporting systems. Household surveys are the main instrument used by SNV to evidence sanitation and hygiene outcomes. Third party verification confirmed that some survey enumerators did not do their work properly – perhaps by taking short cuts to save time, or failing to follow protocols correctly – and that some processes could be improved. The mobile-to-web survey approach used by SNV greatly facilitated the verification process, with GPS and time data available for household interviews, and toilet photographs where appropriate. The rich survey data facilitated a cost-efficient and largely remote verification process, with only a small sample of villages and households visited in person to check for systemic errors.

Many of the survey problems were identified at the start of the programme, and SNV worked to strengthen and professionalise its monitoring and reporting processes so that later surveys ran smoothly and efficiently. This brings us to the second impact of the verified PbR – the development of the verification process was transparent, with all checks discussed and agreed with SNV before use. In order to avoid any surprises during verification, SNV began to run similar checks on the surveys before verification. As a result, SNV spotted several monitoring issues early, and was able to take prompt action to remedy these problems.

---

The combined effect of the PbR imperative to achieve results, and the detailed verification of these results, have led to more professional and systematic approaches to monitoring and reporting. All of these have important effects on implementation – as more reliable data and more rapid feedback mechanisms encourage adaptive management.

The use and analysis of the monitoring data also strengthens the systems – data collectors know that someone will be analysing the data, someone else will be verifying the data, and that these data will form the basis of programme payments. The shift caused by the greater scrutiny and use of the monitoring data has been a game-changer. The value of reliable monitoring and evaluation systems is increasingly apparent, with programme managers and project staff now able to focus on the systems and processes that achieve sustained outcomes, rather than on the inputs and expenditures that drive conventional WASH programmes [2].

Verification has proved a useful tool to encourage more professional and systematic approaches to monitoring and evaluation, but it is only one small part of the wider challenge of achieving large-scale and sustained development outcomes – wine may bring out the truth, but turning truth into health is a much harder trick!

Suggestions for further reading: Visit e-Pact’s blog series, Learning about Payment by Results in Water, Sanitation and Hygiene (WASH) A blog by the team verifying and evaluating DFID’s Results Programme:  

[1] “Beyond a burden: what value does verification offer“

[2] “The payback and pains of Payment by Results (part 1)“. 

About the Author: Andy Robinson is an independent water and sanitation specialist based in the French Alps. Since 1987, Andy has worked on the design, implementation, and evaluation of WASH programmes in Asia and Africa for a diverse range of clients. In the last fifteen years, Andy has been heavily involved in the promotion of improved sanitation and hygiene, working with governments, development partners, and communities in more than 20 countries in Africa and Asia.     

This post was originally published on the SNV website and is one of a series leading up to World Water Week 2018.

Going to #WWWeek? Come to the WASH Results session on Sunday morning and find out more about the realities of Payment by Results in WASH.

Hard work pays off: access and service delivery results under a Payment by Results programme

In 2014, the ‘new kid on the block’ for WASH sector funding was introduced through DFID’s WASH Results Programme’s Payment by Results. The fund was designed to give impetus to meet the MDG goals on WASH, found to be lagging behind in many countries. The targets were ambitious, the model risky. SNV’s Anne Mutta shares SSH4A RP’s experience in implementing PbR.

The Sustainable Sanitation and Hygiene for All Results Programme (SSH4A RP) of SNV, financed by UKAID, was among the first organisations to engage in PBR programming in 2014. By end of December 2017, our four-year experience in implementing SSH4A RP in 62 districts across nine countries in Africa and Asia successfully engaged 7 million people. Independently verified results of the programme found that at least 2.7 million people have gained access to and are using new and improved sanitation facilities. Beyond meeting our pre-defined results at household level, SNV’s payment was tied to so-called ‘Sustainability Results’, measuring progress towards sustainable service delivery systems in each district.

Stephen Covey says ‘begin with the end in mind’, under a PBR system this translates to having a definite goal to work towards (the results) and constantly reviewing what you are doing (process) to see if it is getting you where you want to go!

SNV staff and government officials dialogue with kebele female sanitation group

Kebele residents count days before flag turns green signifying ODF achievement

Under a PBR contract, programme targets are fixed and underachievement is ‘penalised’. Because overly-optimistic targets are considered financial risks, organisations are challenged to be more realistic at the start of programme design. Building on and ‘growing’ our years of SSH4A work (developed in 2008) had helped us offer ambitious but realistic targets that can be delivered within a given time frame. Similarly, because we’ve been physically present in our SSH4A RP countries for years (for example, we’ve been present in Kenya since 1969) our knowledge base and partnership with local government and CSOs allowed us to prepare for potential risks and unforeseen events. The number of countries and districts, helped to balance out the effect of those unforeseen events, as there are many factors beyond our control, in particular affecting sustainability indicators.

Although targets are fixed under DFID’s WASH Results Programme (WRP), there is flexibility in implementation: programme activities may be changed to respond to evolving contexts. A lot of time savings were realised because there was no need for us to engage in a protracted consultation process with UKAID when funds had to be shifted. The WRP’s PbR gave us more control on how things can be done and allowed us to swiftly respond to shifting needs or priorities. Our country teams –  who had a better handle of the contexts they were working in – were able to adjust their interventions and search for those most suitable in supporting households to gain access to sanitation facilities. Experience and engagements on the ground guided our interventions, and facilitated actual ‘learning and doing’.

PbR increased attention on the need for sound monitoring systems within project designs. A defining characteristic of the WRP is the integration of an independent third party verification process, which is conducted prior to ‘pay out’. With quality monitoring and evaluation (M&E) becoming an integral part of programme conversations, we introduced robust M&E tools, our country teams became involved in conducting household surveys, and they organised regular stakeholder review/ reflection meetings on sustainability indicators. Constantly questioning whether progress had been made, why and how things can be done better and sustainably have helped us reach optimal results. This reflection process served as the main driver behind our achievements, and the results that we all take pride in.

By embracing this new working modality, and with hard work, we managed to surpass at least 95% of our SSH4A RP pre-defined targets. Without a reflective, evidence-based process of target setting and progress monitoring, it is foolhardy to imagine that we are able to get the same desired results.

About the Author: Anne Mutta is multi-country programme manager of SNV’s largest Payment by Results (PbR) programme to date – a multi-million programme that is being implemented across eight countries in Africa and Nepal. Anne, based in Kenya, has 20 solid years of experience leading action-oriented and evidence-based WASH programmes.

Photo credit: Anjani Abella/SNV

This post was originally published on the SNV website and is one of a series leading up to World Water Week 2018.

Going to #WWWeek? Come to the WASH Results session on Sunday morning and find out more about the realities of Payment by Results.

Oxfam’s experience with the SWIFT Consortium for Sustainable WASH in Fragile Contexts

What is Payment by Results like for Suppliers? In this post (first published by SNV) the SWIFT Consortium’s Global Programme Manager reflects on Oxfam’s experience.

In 2014 Oxfam made the decision to bid for a big commercial tender. It was not just any commercial tender but one that was structured along a Payment by Results (PbR) process. This meant that all programme funding had to be pre-financed by Oxfam and its partners. Once results have been proven, Oxfam gets paid by DFID (Department for International Development). This process introduced a new way of working for a lot of teams: we had to provide concrete evidence to prove our success.

Oxfam took a lead in this PbR-funded project as member of the SWIFT Consortium for Sustainable WASH in Fragile Contexts: a consortium with partners from Kenya, DRC and Liberia. Our ambition was to reach over 1 million people across these three countries over a four-year period. Unfortunately, the Ebola crisis in Liberia halted programming there, but we continued to work in DRC and Kenya.

Between 2014 and 2016 the programme had reached over 850,000 people in DRC and Kenya, providing safe water or undertaking sanitation and hygiene promotion activities. During this period, Oxfam achieved its first set of goals. Despite the immense pressure on our teams in countries, we received 100% payment for the initial phase.

Unlike many traditional implementation programmes, DFID included a sustainability phase in its programming. During this phase – between 2016 and 2018 – teams supported local capacity for the management of systems. Through working with water utility companies, we sought to improve services and introduce an e-billing service to improve efficiency and decrease corruption. Both water utility companies that Oxfam worked with now have improved revenue collection systems; staff of one company now have their own uniforms.

In Nairobi, Oxfam partner WSUP (Water and Sanitation for the Urban Poor) worked with Nairobi Water Company to bring water to individual compounds in some of the poorest neighbourhoods of the city. Oxfam also expanded Water ATMs, which allowed for people to access water whenever it is convenient, and ensured that payments are clear and transparent.

Arron Gesar holds token to collect water from a Water ATM in Hadado Kenya

In DRC Oxfam piloted ‘ASUREPs’, a professional water management model, which has made significant improvements to the maintenance of water systems, increased the proportion of people paying for the service and, in some places, raised money to extend the system. This model was implemented in semi-urban localities: small towns or high growth areas. It has been praised by the government, leading to an opportunity to influence WASH policies at national level.

Ms Kahindo Byamungu, Sake ASUREP worker, opens water point at scheduled times

The benefits of this programme, beyond its achievements, have been for all of us – from head offices to country teams and partners – to really think about sustainability, about WASH, and look at how we are measuring success.

All these activities took place during a drought in Kenya, insecurity in DRC, and heightened political tensions during the election period for both countries. The teams, throughout the programmes, went above and beyond to deliver a successful programme, which has been called a ‘high performing programme’ by DFID.

Finally, this risk has been paid off with DFID extending our DRC programme for another three years.

About the Author: Joanna Trevor has over 15 years of experience working in fragile states, holding a range of advocacy and programme management roles across East and Central Africa. She is currently Global Programme Manager for the SWIFT Consortium, which aims to deliver sustainable access to safe water and sanitation, and to encourage the adoption of basic hygiene practices in the Democratic Republic of Congo (DRC) and Kenya. Having joined Oxfam in 2013, Joanna has previously worked for World Vision, Mercy Corps and as a consultant for the British Government’s Stabilisation Unit. Most recently, she was Oxfam’s Programme Director in DRC. She holds a degree in Social Policy from Swansea University.

Photo credits: Banner and 1st photo by Katie G. Nelson/Oxfam | 2nd photo by Anne Cossutta/Oxfam

This post was originally published on the SNV website and is one of a series leading up to World Water Week 2018.

Going to #WWWeek? Come to the WASH Results session on Sunday morning and find out more about the realities of Payment by Results.

Beyond a burden: what value does verification offer?

Police officers, auditors, teachers marking homework and giving out detentions – just some of the unfavourable analogies we have heard applied to the role of the independent verification team in the WASH Results Programme. Catherine Fisher highlights the positive roles missing from these comparisons.

Our job is to verify that the achievements reported by Suppliers delivering the programme are accurate and reliable in order that DFID can make payment. It’s easy to see why the relationship between Verifier and Supplier can be an uncomfortable one, but in this post we look at the value of verification and what, if any, benefits it brings to Suppliers.

Why does the WASH Results Programme have a verification team?

Payment by Results (PbR) guru, Russell Webster, concluded from his review of PbR literature :

“When commissioners devise a contract where payment is mainly contingent on providers meeting outcome measures, they need to be confident in the data relating to whether these measures are achieved. There are two main issues:

• Is the provider working with the right people (i.e. not cherry picking those who will achieve the specified outcomes most easily)?
• Are the data reliable?”

Let’s take each of these in turn.

All the Suppliers in the WASH Results Programme are international NGOS who have continued to pursue their commitment to values such as equity and inclusiveness even if it has not been incentivised by the PbR mechanism. A central theme in our peer learning workshops has been the ongoing puzzle of how to place value (both in commercial/financial and Value for Money terms) on intangible aspirations and benefits, such as reaching the most vulnerable and investing in the processes and social capital that underpin effective programming. Suppliers and the Verification Team have been exploring how PbR can enable alignment with national systems and promote downward, as well as upward accountability.

There has been no evidence of gaming in the WASH Results Programme. That is not to say that it might never be an issue for other PbR contracts and the higher the risk of gaming, the greater emphasis there needs to be on verification. So if verification has not identified any gaming, what value has it brought?

Are the data reliable?
Because the WASH Results Programme relies largely on Supplier’s own monitoring data, the benefits of verification stem from the question of whether Suppliers’ data about their achievements are reliable. This has been a matter of great debate.

We have found that in some cases it is right not to rely unquestioningly on data that comes from Suppliers’ monitoring systems – those systems are not always as robust as Suppliers themselves thought. Verification has identified several situations where Suppliers could have gone on to inadvertently over-report results, which would have led to DFID paying for results that had not been achieved. Verification ensured DFID only paid for genuine results and helped Suppliers improve their monitoring. We explore the value to Suppliers of improved monitoring, later.

One of our Country Verifiers (members of the Verification Team based where the implementation is taking place) recently observed: “From my experience, the WASH Results programme is quite different from the traditional way of doing implementation – having someone who is independent, who checks the Suppliers’ results before they are paid for, makes it quite a good tool to hold Suppliers to account.”

So far, the obvious value that verification in the WASH Results Programme has brought to DFID is confidence in results, through third party information about those results, and a reduced risk of paying for results that were not achieved. But there are more, less apparent, benefits and we describe towards the end of this post.

Can verification bring value to Suppliers?

Having explored the value of verification to the donor, we now turn to the value for Suppliers.

The same Country Verifier commented that while he felt some Suppliers were initially scared that the verifier was there to spot their mistakes, “I think with time they realise that the role of independent verification is just to check that what they’re reporting is what the reality is when the verifier goes out to sites where they’ve been working. You’re only checking.”

Although Suppliers often view verification as a “burden”, our team identified a set of potential returns for the Suppliers on the effort and investment they put into participating in the process (effects, we suspect, that donors would appreciate). We acknowledge that it can be hard to unpick the value of verification from the value of investing in better monitoring per se, but without overstating our role, we feel we have contributed to:

• Identifying areas for improvement – verification has revealed flaws in a system thought by the Supplier to be strong and introduced tests that were not previously used. In one example, verification revealed problems with third party enumerators’ work and this prompted greater scrutiny of their data by the Supplier and changes to training processes.
• Strengthening Quality Assurance – We have seen how the expectation of verifiers checking data can prompt Suppliers to improve their own Quality Assurance (QA) processes, for example, carrying out internal checks prior to submitting data for verification and introducing QA protocols.
• Increasing the value of data – the process of verification counters the commonly-held belief that “no-one looks at this data anyway”, which, unchecked, can reduce the effort put into data collection and the usability of the data systems.
• Reducing risk of failure (and withholding of payment) – The requirement to have more and better data can pre-empt some problems. For example, knowing that they would need to demonstrate to verifiers that they had met their water systems targets, prompted one Supplier to check in advance if the declared yield of sources would be enough to reach the population they were planning to reach.
• Forcing deeper reflection – linking PbR to the achievement of WASH outcomes has forced Suppliers to think about WASH outcomes and how they can be measured and be clearer on definitions to a greater degree than in other, non-PbR, programmes. Verification has by no means driven that process but has contributed to it.

We acknowledge that these may not always have felt like benefits to the Suppliers! In particular, some Suppliers have pointed out the trade-off between data collection and learning, and suggested that the burden of verification has stifled innovation and inhibited adaptive programming. Others, however claim the opposite, which implies there may be other factors at play.

In spite of concerns, there is broad consensus that the PbR modality, of which verification is a part, has driven higher investment in and attention to programme M&E systems. PbR requires Suppliers to be clear about what they are trying to achieve, to collect good quality data to monitor their progress and to use that data to report on their progress regularly. Verification has helped to build confidence in the strength of systems and data on which those processes are based. There is an emerging sense that effective use of reliable M&E data by Suppliers has enabled rapid course correction and so contributed to high achievements across the WASH Results Programme.

And if that is not enough, we think there are benefits for other stakeholders in countries in which WASH Results is operating. We have seen some benefits from capacity spillover– skills and knowledge acquired through working in or observing the data collection, analysis and verification in the WASH Results Programme are available to other programmes e.g. among enumerators, Country Verifiers, programme staff, even Government agencies.  Again, this is by no means all attributable to verification but verification has contributed.

Value and the limits of verification

It can be hard to unpick the benefits of verification from benefits that stem from the greater emphasis on data collection inherent to PbR. In some contexts PbR is being used without third party verification. But, in contexts where reassurance is needed about the reliability of the data on outputs and outcomes, we believe verification offers value to the donor, to the Suppliers and, potentially to others in the country in which the programme is operating.

While we have argued for the benefits of verification, there are weaknesses in PbR that verification cannot solve. Verifiers, like police officers, don’t make the rules, they just enforce them. They verify results that have been agreed between the donor and the supplier. As one of our team observed recently “Payment by Results makes sure you do what you said you would. It doesn’t make you do the right thing….”

However, if verification helps drive a “race to the top” in terms of quality of monitoring systems, the sector will begin to have better data on which to base decisions. Better data about what kinds of programmes produce what kinds of outcomes in which contexts could help donors to fund, and programmers to implement, more of the “the right thing”. And the police officers will feel their job has been worthwhile.

---

Catherine Fisher, Learning Advisor, Monitoring and Verification Team for the WASH Results Programme. This post draws on a reflection process involving members of the Monitoring and Verification team for the WASH Results Programme (Alison Barrett, Amy Weaving, Andy Robinson, Ben Harris, Cheryl Brown, Don Brown, Joe Gomme and Kathi Welle).

---

Want to learn more about the experience of the WASH Results Programme? Join us in Stockholm during World Water Week for ‘The Rewards and Realities of Payment by Results in WASH’

Truly exceptional? Handling misfortune within Payment by Results

An exceptional event or a predictable adversity? The difference matters more in a Payment by Results (PbR) setting, as this blog post explores.

Conflict, political upheaval, epidemic, drought, flooding and earthquake; the WASH Results Programme has been hit by a wide range of disasters across the 12 countries in which it operates. All these adversities had an impact on the populations involved: some hit the programme’s implementation, some the sustainability of its achievements and others have affected the ability to monitor and verify those achievements.

Discussions on how to deal with these events have involved considering what is within the reasonable expectation of a Supplier to anticipate and deal with, and what is a truly exceptional event for which there should be flexibility around what Suppliers are expected to deliver – whether in the quantity, scope or timing of results.

The challenge of responding to exceptional events is not new for development programmers, but like many challenges, it takes a different shape in the context of a PbR programme. In such programmes, payment is linked to achieving and verifying results, and an impact on results ultimately leads to a financial impact for one or more of the parties. This is particularly challenging in the second phase of WASH Results, where Suppliers are paid for sustaining results achieved in the first “Outputs” phase. The passage of time means that programme achievements (whether physical infrastructure or behaviour change) are more likely to be affected by exceptional events, and suppliers may not have the resources (such as programme field staff) in place to undertake substantial mitigating actions.

Members of our team (Monitoring and Verification for the WASH Results Programme) recently met to discuss their experience of exceptional events in the programme and how these were tackled. Here are three of the examples they discussed followed by the team’s reflections on the issues they raise:

1) The moving population In this example a conflict-affected community relocated out of the proposed project area. In response, the Supplier closed the project in that area but thanks to overachievement of results in other locations, the overall outputs of the programme (in terms of beneficiaries reached) were not reduced and the Supplier did not suffer financially. In this case, the flexibility of PbR meant the risk to Supplier, Donor and Verifier could be effectively mitigated, although some of the original intended beneficiaries were not reached.

2) Destroyed infrastructure, different decisions In one instance, WASH infrastructure built in the first (output) phase of the WASH Results Programme was destroyed when a river eroded part of a village.  Although there was a known risk of erosion, the timing of erosion could not be foreseen nor the risk mitigated. The people living in this area were some of the poorest and most vulnerable whom the Supplier did not want to exclude from the programme.  The erosion was considered extreme, as evidenced by newspaper reports and other local data and it was agreed the area with the destroyed infrastructure would not be included in the sample frame for outcome surveys and so would not affect outcome results.

Meanwhile, in the same country, infrastructure was damaged by flooding, but this was considered expected, not extreme. In contexts where flooding can be expected, the demand for sustained outcomes (in which payment is linked to the sustained use of infrastructure) requires that infrastructure is built in such a way that it can withstand expected levels of flooding, or that plans for reconstruction or repair in the case of damage should be integral to programming. Consequently, areas in which infrastructure was affected by flooding were to be included in the sample frame for the outcome survey, which was amended to include questions about flood damage and beneficiary priorities for reconstruction.

3) When verification is too risky  When conflict erupted in one project location, the programme was able to implement activities regardless and continued to achieve results. However, the security situation on the ground made it too risky (for programme staff and the verification team) for the results to be independently verified through a field visit. In this case, alternative and less representative forms of verification were accepted. In this example, there was no adverse impact on the results achieved, or reduction in payment to the Supplier, but there was increased risk around the confidence that could be placed in the results.

Making decisions about risk

In exceptional circumstances, decisions need to be made about who bears the risk (Donor, Supplier, Verifier, Beneficiaries) and what kind of risk (physical, financial, reputational). If financial risk falls exclusively on Suppliers, they need to factor that into their “price per beneficiary” across the programme. Alternatively, Suppliers may choose not to operate in riskier areas, with potential negative consequences for the equity of programme interventions. If donors accept all risk, there is little incentive for Suppliers to programme in ways that account for predictable risks, such as flooding, particularly over the longer term.

Reflections and suggestions emerging from the discussion of these cases included the following:

• There are different types of impact to consider: effect on population, effect on ability to deliver activities, effect on achievement of results, and effect on ability to verify results. Being clear on the type of impact might aid decisions about who bears the risk and the mitigation strategy.
• Discussions about risk need to happen during the design phase; one approach is to use a risk matrix that explores what level of risk each party is going to absorb (and so design into the programme) and what would be considered ‘exceptional’.
• Programmes need to include within their design plans for responding to anticipated events e.g. in areas at risk of flood, include help for villages to cope with normal levels of flooding.
• Suppliers can minimise their financial and operational risk by balancing their work across a range of fragile and more secure areas, so enabling them to pull out of more hazardous areas in extreme circumstances and achieve results elsewhere. However, if the Supplier commits to working with specific communities in conflict-affected areas, then incentives will need to be set up differently within the results framework.
• In fragile contexts, a compromise may need to be made on rigour of verification and plans made for reliance on remote verification from the start, e.g. analysis of systems, remote data collection through phone or satellite, and beneficiary monitoring.

Our conclusions about exceptional events in the context of a PbR programme in WASH echo those in many of our previous blog posts. PbR throws a spotlight on an issue, raises questions of how risk is shared between stakeholders, and highlights the importance of planning at design phase and of flexibility, should the unforeseen occur.

If you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us

Recent views and news on Payment by Results – February 2018

What are people saying about Payment by Results (PBR) and development in 2018?

“We can’t ignore the importance of the process”, David Archer of ActionAid is reported to have said during a debate on Results Based Financing at the Global Partnership for Education’s Financing Conference. As well as more views from ActionAid, the live blog of the session on February 1st, also shares Norad’s experiences of Results Based Financing.

A policy paper published by UNESCO the previous month, questions the sustainability of impact on education systems achieved via results-based payments and urges donors to use caution. The paper looks at why there are so few evaluations in this area and the efforts being made to collect more evidence.

Also in January, the IDB published a Working Paper that seeks to answer the question ‘Is Results-Based Aid More Effective than Conventional Aid?’ by reviewing a natural experiment in the health sector in El Salvador where municipalities were funded through different mechanisms, including Results-Based Aid. The authors found that compared to conventional funding, financing that involved results-based conditionality roughly doubled aid effectiveness and conclude that “The effects appear to have been driven by a more rapid expansion of health infrastructure and qualified personnel by motivated national authorities”.

Coming up: ‘Outcomes-based commissioning: learning from Payment by Results and Social Impact Bonds’. On March 27th, Professor Chris Fox, Director of Policy Evaluation and Research Unit, at Manchester Metropolitan University, will share lessons learned from the UK’s use of Payment by Results and Social Impact Bonds since 2010.

From PERU’s website, you can take a look at earlier work and presentations by Professor Fox and colleagues on this topic, including a 2017 paper on evaluating outcome-based payment programmes.

Cheryl Brown, Communications Manager, WASH Results MVE Team.

 

 

 

WASH Results’ evaluation highlights differences in design

The WASH Results Evaluation Team (led by Oxford Policy Management) shared their midline evaluation findings with the programme’s Suppliers through a webinar earlier this year and the slides are now available online together with the full report.

The WASH Results Midline Evaluation was carried out mid-2016 and explored:

• The relevance of programme design (theory of change).
• To what extent the verification systems were fit-for purpose (relevance and efficiency).
• In what ways the Payment by Results (PBR) modality affected implementation.
• Which outputs were achieved (those paid for, and those outside the PBR framework).

The evaluation showcases how the three Suppliers designed their PBR systems in different ways, to suit the different WASH activities being implemented across 12 countries between them. As a result, a tailored verification system was also implemented by its Third Party Verifier (led by Itad) to fit this variation. This diversity is summarised nicely in the slides that accompanied the webinar.

The planned output targets of the programme were achieved by December 2015 – with significant over-achievement (though over-achievement was not paid for under the PBR modality). They were achieved off the back of results-oriented problem-solving by the Suppliers who adjusted activities to achieve agreed results. The increased scrutiny of the third-party verification mechanism also helped strengthen monitoring systems of the Suppliers, resulting in more regular and higher-quality monitoring data. However, the evaluators also found that the pressure of achieving results put significant burden on programme staff and partners.

Suppliers managed the risks of PBR in a variety of ways, including front-loading payments and moving output targets between countries if one was under-achieving. With a few exceptions, the Suppliers also opted to pre-finance their local implementation partners and thus hold the PBR risks at the headquarters level, instead of passing them down to local implementation partners.

The midline evaluation webinar concluded with a suite of recommendations for PBR suppliers (NGOs), for DFID and for independent verifiers. The endline evaluation will be carried out in 2018 and will focus on outcome achievement, impact and sustainability aspects.

If you would like to be notified when the WASH Results Endline Evaluation Report is available, please email Cheryl Brown, WASH Results MVE Team Communications Manager.  You may also be interested in the Evaluation Team’s Short Reading List on PBR.

 

 

 

Reflections on our experience of DFID’s results agenda

As verifiers of a DFID Results Based Finance programme, ODI’s research on the UK’s results agenda prompted us to reflect on our experience.

 

Why context matters when you focus on results #1: Some latrine building projects have to allow for the impact of hungry goats. Photo credit: Chamia Mutuku

 

In their report ‘The Politics of the Results Agenda in DFID: 1997 to 2017’, Craig Valters and Brendan Whitty argue that 2007 saw a new explicit focus from DFID on aggressively implementing results-based management. 10 years later, we have WASH Results: a DFID-funded programme where financial risk has been completely transferred away from UK taxpayers to the international NGOs who deliver the work and who only get paid for results that have been checked by a third party – us. However, as its name promises, the programme is delivering results in water, sanitation and hygiene (WASH). DFID was able to read in the programme’s 2017 annual report (with great confidence in the figures), for example, that WASH Results had reached over 1.1 million people with improved water supply, more than 4.7 million people with improved sanitation, and over 14.9 million people with hygiene promotion.

In our role as Monitoring, Verification and Evaluation Team for the WASH Results programme, our attention is less focused on the politics of the results agenda, and more in how results are monitored and verified and the very real impact that this approach has on ongoing programme delivery. However, we read the report and blog post by Valters and Whitty with great interest.

After more than three years of supporting the programme, how does our experience compare with the conclusions and recommendations of the ODI report? One key finding from the research is that some DFID staff have found ways to adhere to the results agenda, while retaining flexibility. This theme of the ways in which both donors and programme implementors are working creatively around the “tyranny of results” was one that we heard during last year’s BOND Conference session ‘How to crack Results 2.0’.

How can PBR be adapted to address the imbalance in accountability?

We absolutely agree with Valters and Whitty about the importance of finding a balance between being accountable to UK citizens and to the beneficiaries (poor people abroad). This time last year, we shared our opinion that if verification was designed to include beneficiary feedback and this was linked to payment, Payment by Results (PBR) could actually generate more downward accountability than other funding modalities. However, our team of verifiers felt that the demands of verification for large scale, representative, quantitative information on which to base payment decisions may leave less time, money and inclination to undertake more qualitative work with beneficiaries. So, we suggested that a resource-effective solution to upholding downwards accountability through verification would be to include payment for the existence and effective functioning of a beneficiary feedback system (rather than the results of that system). Payment would be made on verification of the effectiveness of the system in promoting downwards accountability.

We welcome the authors’ call to “Create a results agenda fit for purpose”. Our first reflection would be that a results agenda, at least one hard-wired into a PBR modality, is not going to be appropriate in all contexts and for all intended outcomes, particularly those where outcomes are difficult to predict or challenging to measure. Our set of recommendations to commissioners of PBR programmes, complement several of those made by ODI, for example, their suggestion that DFID spend more time considering whether its aid spending has the right mix of risks and the view that regular testing (that leads to course-correction) is important.

The challenge of communicating about costs and value

The authors also call on ministers to be honest with the British public about aid. Part of this, we feel, is making it clearer that Value for Money (VFM) is not synonymous with “cheap”. We feel that the results agenda, particularly a PBR model, should require donors/commissioners to clearly articulate the “value” they expect to see in VFM. Otherwise the importance placed by a donor on achieving clearly costed, verified results could risk squeezing out other values and principles that are central to development programming. A central theme in last year’s WASH Results learning workshop was the ongoing puzzle of how to place value (both in commercial/financial and VFM terms) on intangible aspirations and benefits, such as reaching the most vulnerable and investing in the processes and social capital that underpin effective programming. This is particularly important in an increasingly commercialised aid context, where one supplier’s approach would be to parachute in and build many toilets very quickly and cheaply, whereas another proposes taking longer to work with local stakeholders, norms and materials. This articulation of value may not be as simple as it sounds, when every commitment in a PBR programme, such as reaching the poorest, gender equity, national ownership, sustainable outcomes, etc. needs to be reflected in meaningful and measurable indicators.

Payment By Results can aid course correction

Interestingly, one of the reforms that the authors call for may be an inherent feature of the results framework itself. They say that “interventions need to be based on the best available information, with regular testing to see if they are on the right track”. We have found that a product of the PBR modality is that much greater emphasis is placed on monitoring systems and the generation of reliable data about what is happening within programmes. In WASH Results we have seen cases where the rigorous (compulsive?) tracking of results has identified areas where programmes are failing to deliver and rapid action has then been taken to address that failure. As verification agents we argue that this is due not only to the link between results and payment but also the independent verification of data and systems that has led to better information on which to base decision-making.

Benefits of the results agenda

In this way we think that the focus on monitoring within the results agenda, can, in some cases, enable flexibility and innovation. In its reliance on high quality data, it contains within it a driver that could improve the way that development work happens. The results agenda brings benefits – some of which we did not see reflected in the article – but it comes with risks; both ideological about the ambitions for UK Aid and practical for those involved in its delivery. And so we welcome this debate.

Catherine Fisher, Learning Advisor, WASH Results MVE Team

If you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us

Why is it so hard to measure handwashing with soap?

As Global Handwashing Day approaches, the WASH Results MV team explores the challenges of verifying changes in handwashing behaviour.

Hygiene has tended to be the Cinderella of WASH (Water, Sanitation & Hygiene): relegated to the end of the acronym and with hygiene promotion often having been treated as an afterthought in larger water and sanitation programmes. But good hygiene is recognised as an essential component in addressing the burden of disease, and hygiene promotion can be a cost-effective way of improving health outcomes. In linking payment milestones to sustained handwashing behaviour the WASH Results Programme is breaking new ground, and pushing against some of the limits of understanding in the sector. The programme has faced key challenges in how to establish what reasonable handwashing outcomes might look like, and the most appropriate way to measure them. This post draws on discussions within the Monitoring and Verification (MV) team on some of these issues, and resulting internal briefings.

What is a good handwashing outcome?

Across the WASH sector there is little established knowledge on the sustainability of WASH outcomes. Whilst the well-known (and well-resourced) SuperAmma campaign saw a sustained increase in handwashing practice of 28% after 12 months, it’s still an open question as to what level of behaviour change it might be reasonable to expect across differing countries, contexts and projects. This matters, because in a payment by results (PBR) context, payment is linked to achieving targets. Set targets too high and suppliers risk being penalised for failing to reach unrealistic expectations. Set targets too low and donors risk overpaying.

How do we measure handwashing?

Compounding the uncertainty over what reasonable targets may be, is uncertainty on how to measure achievement against those targets. There are several commonly accepted methodologies, but they all involve compromises. At one end of the scale there is structured observation of household handwashing behaviour. Considered the gold standard in measuring behaviour change, this can provide a wealth of data on the handwashing practices of individuals. But it’s also prohibitive to undertake at scale – the most rigorous designs can involve hundreds of enumerators making simultaneous observations. The feasibility of conducting such measurements regularly for a PBR programme is questionable.

A much simpler (and on the face of it, more objective) indicator might be the presence of handwashing facilities (a water container and soap). This is used by the Joint Monitoring Programme (JMP) to measure progress against SDG 6.2 because it is the most feasible proxy indicator to include in large, nationally-representative household surveys which form the basis for the Sustainable Development Goal monitoring (and typically collect data on WASH as just one of several health topics). However, these observations tell us nothing about individuals’ handwashing behaviour, or if the facilities are even being used.

Setting indicators for handwashing outcomes

Within the WASH Results Programme, the suppliers have tended to define handwashing outputs in terms of the reach of handwashing promotion (though evidencing how a programme has reached hundreds of thousands or even millions of beneficiaries leads to its own challenges). They have defined outcomes in terms of:

• knowledge of handwashing behaviour,
• self-reported behaviour, and
• the presence of handwashing facilities.

The suppliers have taken different approaches to considering these indicators separately or as part of a composite indicator to attempt to reach a reasonable conclusion on what the sustained change in handwashing practice has been. Each of the indicators has drawbacks (self-reporting, for example, has been shown to consistently over-estimate handwashing behaviour), but by considering all of them, a more reliable estimate may be reached.

Where this process has become particularly challenging for the WASH Results programme, is in attempting to measure outcomes at scale and across highly diverse contexts. All too often the devil is in the detail and small details in how indicators are set and measured could lead to huge variations in results, in turn leading to major differences in what suppliers get paid for. For example, we may wish to define a standard for the presence of a handwashing facility as a water container and soap, but very quickly some of the following issues arise:

• In some countries, ash is commonly used instead of soap: national governments may actively promote the use of ash, and in some areas, it’s almost impossible to even find soap to buy. But at the same time there is evidence that using ash (or soil, or sand) is less effective than using soap: and for this reason, handwashing with soap is the indicator for basic handwashing facilities for SDG 6.2. Should payment targets follow national norms, or would this be a case of paying for lower-quality outcomes?
• In other areas households choose not to store soap outside because it can be eaten by livestock or taken by crows (a far more common problem than one might imagine). Do we adopt a strict definition and record that there is no soap (so hence no handwashing facility) or is it acceptable if a household can quickly access soap from within a building? Do we need to see evidence that this soap has been used?
• Local practice may mean that handwashing facilities do not have water in-situ, instead people carry a jug of water to the latrine from a kitchen. Recording a handwashing facility only if there is water present may risk a significant underestimate of handwashing practice, but how can we determine if people actually do carry water? And does this practice adversely impact on marginalised groups such as the elderly or people living with disabilities?

And this is just for one indicator. Across all three (knowledge of handwashing behaviour, self-reported behaviour, and the presence of handwashing facilities) the complexity of the methodology for measuring the results can quite quickly become unwieldy. There are tensions between wanting to adopt the most rigorous assessment of handwashing possible (to be more certain that the results do reflect changed behaviour), what data it is feasible to collect, and what it is reasonable for the suppliers to achieve. The answers to these will depend on what the programme is trying to achieve, the results of baseline surveys, and the costs of measurement and verification of results.

There may not be an easy answer to the question of how to measure handwashing outcomes, but the experience of the WASH Results Programme suppliers has provided useful learning on some of the aspects that need to be thought through. As the programme progresses, the suppliers are continuing to refine their understanding of how best to approach this issue in the countries and contexts they work in, and iterate the indicators they use. What’s next? Well, exploring how the various handwashing indicators relate to improved public health could be interesting…

What can we learn from the Government Outcomes Lab?

Our Learning Advisor, Catherine Fisher, investigates a new research centre on outcomes-based commissioning and finds plenty of interest for the WASH Results Programme.

You know an idea has traction when the University of Oxford sets up a research centre to investigate it. This is the case for outcomes-based commissioning (aka Payment by Results), which is the focus of the new Government Outcomes Lab (GO Lab) based at the University of Oxford’s Blavatnik School of Government.

The GO Lab focusses on the interests and experience of government departments in procuring services in an outcomes-based way, rather than those of contractors (or suppliers, in WASH Results terminology) in providing them. It is a collaboration between Blavatnik School of Government and Her Majesty’s Government. The focus of the research centre is on outcomes-based commissioning that uses Social Impact Bonds (SIBs), a model in which external investors provide the initial investment for programme implementation which is repaid on achievement of outcomes. However, the GO Lab will also look at other models, presumably including that used in the WASH Results Programme in which the suppliers themselves provide the upfront investment.

The rationale for the GO Lab is as follows: “While numbers of, and funding for, outcomes-based approaches have increased over recent years, research has not kept pace with this speed of growth. Much is still unknown about whether outcomes-based commissioning will meet its promise….Through rigorous academic research, the GO Lab will deepen the understanding of outcomes-based commissioning and provide independent support, data and evidence on what works, and what doesn’t.”  (GO Lab FAQ) .

So far, the GO Lab has organised three “Better Commissioning” events looking at outcomes-based commissioning in different sectors in a UK context, namely Children’s Services, Older People’s Services and promotion of Healthy Lives.

A quick skim of the interesting post-event reports suggests that outcomes-based commissioning is seen as a way of promoting a greater focus on outcomes by providers (who may not already think in this way), of prompting innovation in service provision and of transferring the risk of new approaches from commissioners to socially-minded private enterprises. Similar themes occur in Results Based Aid discussions, although I’d suggest that the international development sector places a slightly greater emphasis on incentivising delivery, value for money and accountability to commissioners.

One aspect of the GO Lab work that caught my eye is their interest in the creation of Outcomes Frameworks which were discussed at each event: “Developing an outcomes framework is a key part of any SIB or outcome based contract, but accessing data and articulating robust metrics that can be rigorously defined and measured is often seen as a challenge by commissioning authorities.”  (Better Commissioning for Healthy Lives: a Summary Report, p 13).

This process of articulating appropriate metrics and identifying indicators has been a key area of learning within the WASH Results Programme and continues to be discussed. It was reassuring to see others grappling with similar challenges in different sectors, such as challenges of creating indicators that:

• are both unambiguous enough to trigger payment and reflective of the outcomes the programme is intended to achieve (Children’s Services Workshop Report, p. 5);
• reflect the context and needs of beneficiaries and are also able be aggregated at a higher level (and internationally in the case of the WASH Results Programme) (Healthy Lives Symposium Report, p.14  and Services for Older People Symposium Report, p.14 );
• reflect the sustainability of outcomes beyond the term of the contract.

During this, the outcomes-focused period of the WASH Results Programme, we will be following the progress of the GO Lab with interest, and hope to find opportunities to exchange learning with them, and others researching innovative funding approaches. Our team is particularly interested in contributing to, and benefiting from, learning around:

• independent monitoring and verification of outcomes-based contracts;
• creating outcomes frameworks that reflect sustained outcomes in areas such as behaviour change (e.g. handwashing behaviours) and institutional change (e.g. ability of district stakeholders to manage water systems);
• streamlining metrics and indicators while balancing needs of all parties: beneficiaries, service providers, commissioners and, in the case of international development programmes, national stakeholders and global commitments such as Sustainable Development Goals.

Catherine Fisher, Learning Advisor, WASH Results MVE Team

If you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us