Tag Archives: Fragile States

Oxfam’s experience with the SWIFT Consortium for Sustainable WASH in Fragile Contexts

What is Payment by Results like for Suppliers? In this post (first published by SNV) the SWIFT Consortium’s Global Programme Manager reflects on Oxfam’s experience.

In 2014 Oxfam made the decision to bid for a big commercial tender. It was not just any commercial tender but one that was structured along a Payment by Results (PbR) process. This meant that all programme funding had to be pre-financed by Oxfam and its partners. Once results have been proven, Oxfam gets paid by DFID (Department for International Development). This process introduced a new way of working for a lot of teams: we had to provide concrete evidence to prove our success.

Oxfam took a lead in this PbR-funded project as member of the SWIFT Consortium for Sustainable WASH in Fragile Contexts: a consortium with partners from Kenya, DRC and Liberia. Our ambition was to reach over 1 million people across these three countries over a four-year period. Unfortunately, the Ebola crisis in Liberia halted programming there, but we continued to work in DRC and Kenya.

Between 2014 and 2016 the programme had reached over 850,000 people in DRC and Kenya, providing safe water or undertaking sanitation and hygiene promotion activities. During this period, Oxfam achieved its first set of goals. Despite the immense pressure on our teams in countries, we received 100% payment for the initial phase.

Unlike many traditional implementation programmes, DFID included a sustainability phase in its programming. During this phase – between 2016 and 2018 – teams supported local capacity for the management of systems. Through working with water utility companies, we sought to improve services and introduce an e-billing service to improve efficiency and decrease corruption. Both water utility companies that Oxfam worked with now have improved revenue collection systems; staff of one company now have their own uniforms.

In Nairobi, Oxfam partner WSUP (Water and Sanitation for the Urban Poor) worked with Nairobi Water Company to bring water to individual compounds in some of the poorest neighbourhoods of the city. Oxfam also expanded Water ATMs, which allowed for people to access water whenever it is convenient, and ensured that payments are clear and transparent.

107058-flipped_0

Arron Gesar holds token to collect water from a Water ATM in Hadado Kenya

In DRC Oxfam piloted ‘ASUREPs’, a professional water management model, which has made significant improvements to the maintenance of water systems, increased the proportion of people paying for the service and, in some places, raised money to extend the system. This model was implemented in semi-urban localities: small towns or high growth areas. It has been praised by the government, leading to an opportunity to influence WASH policies at national level.

new_anne

Ms Kahindo Byamungu, Sake ASUREP worker, opens water point at scheduled times

The benefits of this programme, beyond its achievements, have been for all of us – from head offices to country teams and partners – to really think about sustainability, about WASH, and look at how we are measuring success.

All these activities took place during a drought in Kenya, insecurity in DRC, and heightened political tensions during the election period for both countries. The teams, throughout the programmes, went above and beyond to deliver a successful programme, which has been called a ‘high performing programme’ by DFID.

Finally, this risk has been paid off with DFID extending our DRC programme for another three years.

About the Author: Joanna Trevor has over 15 years of experience working in fragile states, holding a range of advocacy and programme management roles across East and Central Africa. She is currently Global Programme Manager for the SWIFT Consortium, which aims to deliver sustainable access to safe water and sanitation, and to encourage the adoption of basic hygiene practices in the Democratic Republic of Congo (DRC) and Kenya. Having joined Oxfam in 2013, Joanna has previously worked for World Vision, Mercy Corps and as a consultant for the British Government’s Stabilisation Unit. Most recently, she was Oxfam’s Programme Director in DRC. She holds a degree in Social Policy from Swansea University.

Photo credits: Banner and 1st photo by Katie G. Nelson/Oxfam | 2nd photo by Anne Cossutta/Oxfam

This post was originally published on the SNV website and is one of a series leading up to World Water Week 2018.

Going to #WWWeek? Come to the WASH Results session on Sunday morning and find out more about the realities of Payment by Results.

Truly exceptional? Handling misfortune within Payment by Results

An exceptional event or a predictable adversity? The difference matters more in a Payment by Results (PbR) setting, as this blog post explores.

Conflict, political upheaval, epidemic, drought, flooding and earthquake; the WASH Results Programme has been hit by a wide range of disasters across the 12 countries in which it operates. All these adversities had an impact on the populations involved: some hit the programme’s implementation, some the sustainability of its achievements and others have affected the ability to monitor and verify those achievements.

Discussions on how to deal with these events have involved considering what is within the reasonable expectation of a Supplier to anticipate and deal with, and what is a truly exceptional event for which there should be flexibility around what Suppliers are expected to deliver – whether in the quantity, scope or timing of results.

The challenge of responding to exceptional events is not new for development programmers, but like many challenges, it takes a different shape in the context of a PbR programme. In such programmes, payment is linked to achieving and verifying results, and an impact on results ultimately leads to a financial impact for one or more of the parties. This is particularly challenging in the second phase of WASH Results, where Suppliers are paid for sustaining results achieved in the first “Outputs” phase. The passage of time means that programme achievements (whether physical infrastructure or behaviour change) are more likely to be affected by exceptional events, and suppliers may not have the resources (such as programme field staff) in place to undertake substantial mitigating actions.

Members of our team (Monitoring and Verification for the WASH Results Programme) recently met to discuss their experience of exceptional events in the programme and how these were tackled. Here are three of the examples they discussed followed by the team’s reflections on the issues they raise:

1) The moving population In this example a conflict-affected community relocated out of the proposed project area. In response, the Supplier closed the project in that area but thanks to overachievement of results in other locations, the overall outputs of the programme (in terms of beneficiaries reached) were not reduced and the Supplier did not suffer financially. In this case, the flexibility of PbR meant the risk to Supplier, Donor and Verifier could be effectively mitigated, although some of the original intended beneficiaries were not reached.

2) Destroyed infrastructure, different decisions In one instance, WASH infrastructure built in the first (output) phase of the WASH Results Programme was destroyed when a river eroded part of a village.  Although there was a known risk of erosion, the timing of erosion could not be foreseen nor the risk mitigated. The people living in this area were some of the poorest and most vulnerable whom the Supplier did not want to exclude from the programme.  The erosion was considered extreme, as evidenced by newspaper reports and other local data and it was agreed the area with the destroyed infrastructure would not be included in the sample frame for outcome surveys and so would not affect outcome results.

Meanwhile, in the same country, infrastructure was damaged by flooding, but this was considered expected, not extreme. In contexts where flooding can be expected, the demand for sustained outcomes (in which payment is linked to the sustained use of infrastructure) requires that infrastructure is built in such a way that it can withstand expected levels of flooding, or that plans for reconstruction or repair in the case of damage should be integral to programming. Consequently, areas in which infrastructure was affected by flooding were to be included in the sample frame for the outcome survey, which was amended to include questions about flood damage and beneficiary priorities for reconstruction.

3) When verification is too risky  When conflict erupted in one project location, the programme was able to implement activities regardless and continued to achieve results. However, the security situation on the ground made it too risky (for programme staff and the verification team) for the results to be independently verified through a field visit. In this case, alternative and less representative forms of verification were accepted. In this example, there was no adverse impact on the results achieved, or reduction in payment to the Supplier, but there was increased risk around the confidence that could be placed in the results.

Making decisions about risk

In exceptional circumstances, decisions need to be made about who bears the risk (Donor, Supplier, Verifier, Beneficiaries) and what kind of risk (physical, financial, reputational). If financial risk falls exclusively on Suppliers, they need to factor that into their “price per beneficiary” across the programme. Alternatively, Suppliers may choose not to operate in riskier areas, with potential negative consequences for the equity of programme interventions. If donors accept all risk, there is little incentive for Suppliers to programme in ways that account for predictable risks, such as flooding, particularly over the longer term.

Reflections and suggestions emerging from the discussion of these cases included the following:

  • There are different types of impact to consider: effect on population, effect on ability to deliver activities, effect on achievement of results, and effect on ability to verify results. Being clear on the type of impact might aid decisions about who bears the risk and the mitigation strategy.
  • Discussions about risk need to happen during the design phase; one approach is to use a risk matrix that explores what level of risk each party is going to absorb (and so design into the programme) and what would be considered ‘exceptional’.
  • Programmes need to include within their design plans for responding to anticipated events e.g. in areas at risk of flood, include help for villages to cope with normal levels of flooding.
  • Suppliers can minimise their financial and operational risk by balancing their work across a range of fragile and more secure areas, so enabling them to pull out of more hazardous areas in extreme circumstances and achieve results elsewhere. However, if the Supplier commits to working with specific communities in conflict-affected areas, then incentives will need to be set up differently within the results framework.
  • In fragile contexts, a compromise may need to be made on rigour of verification and plans made for reliance on remote verification from the start, e.g. analysis of systems, remote data collection through phone or satellite, and beneficiary monitoring.

Our conclusions about exceptional events in the context of a PbR programme in WASH echo those in many of our previous blog posts. PbR throws a spotlight on an issue, raises questions of how risk is shared between stakeholders, and highlights the importance of planning at design phase and of flexibility, should the unforeseen occur.

If you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us

Comparative experiences of evaluating and verifying Payment by Results programmes

Highlights of the discussion about verification and evaluation of Payment By Results in our chat show at the UK Evaluation Society Conference.

A session at the UK Evaluation Society conference in May 2015 compared two Payment by Results (PbR) programmes in different sectors. The discussion revealed that although PbR programmes can be set up in very different ways, the management, evaluation and verification face similar tensions. These include balancing cash flows with the focus on “real outcomes” that PbR is designed to encourage.

Using an informal chat show approach, the session explored practical experiences of monitoring and verification of two DFID funded PbR programmes: the Water, Sanitation and Hygiene (WASH) Results Programme and the Girls Education Challenge (GEC). The chat show was attended by one of the WASH Results Programme service providers thus providing a different perspective on the issues.

Chat show participants were:
Dr Katharina Welle, Deputy Team Leader, WASH Results Monitoring, Verification & Evaluation (MVE) Team, (Itad)
Dr Lucrezia Tincani, Manager, WASH Results MVE Team, (Oxford Policy Management)
Jason Calvert, Global Monitoring & Evaluation (M&E) Lead, Girls’ Education Challenge
The host was me, Catherine Fisher, Learning Adviser, WASH MVE Team, (Independent Consultant).

The discussion focused on the practicalities of implementing programmes financed through a Payment by Results funding mechanism (specifically Results Based Financing, a term used by DFID, among others, when the payments from funders or government go to service providers or suppliers). This blog post shares a few of the areas that were discussed.

There are huge variations in the set-up of PbR programmes
Perhaps unsurprisingly for two programmes of different sizes in different sectors, the WASH Results Programme (£68.7 million over approximately four years) and GEC (£330 million over four years) are managed, monitored and evaluated in very different ways.

GEC is managed by Price Waterhouse Coopers (PwC) who manage the funding to the 37 GEC projects, set suppliers’ targets and create M&E frameworks, while suppliers themselves contract external evaluators to monitor and evaluate their progress. By contrast the three WASH Results Programme supplier contracts are managed in-house by DFID and the Monitoring, Verification and Evaluation is contracted out to a consortium: e-Pact. There are no doubt pros and cons of each approach which the chat show did not explore. But a key advantage of the GEC set-up (from the perspective of the e-Pact WASH MVE team) was the involvement of PwC from the start of the GEC programme. This meant PwC could shape the “rules of the game” for verification and were able to set standardised targets up front.

Differences in amount of funding subject to PbR
Another key difference between the two programmes was the amount of the supplier funding that was subject to PbR. For GEC suppliers it is on average 10%, whereas WASH Results Programme suppliers see 100% of their funding dependent on verified results. But this startling figure may mask some similarities….

GEC differentiates between payment against inputs (not called PbR) and payment against outcomes, the “real” PbR which constitutes 10% of the total funding. By contrast, the WASH programme has a broader definition of results that includes outputs, inputs and outcomes, and varies across suppliers. So in practice the programmes may be more similar than they appear.

Balancing cash flow and adaptive programming in a PbR context
An important driver for the broader interpretation of “results” within the WASH Results programme was the very real need for some suppliers to maintain a steady cash flow across the course of the programme in a 100% PbR context.   The temptation for suppliers can be to set lots of milestones that trigger payment. However the need to stick to and demonstrate achievement of these milestones may inhibit flexibility in delivery. This tension between maintaining cash flow and the adaptive programming that PbR is intended to foster has been experienced by both programmes.

Rigour in measurement
The ability to effectively measure results is at the heart of PbR. For PwC, this means that every project subject to PbR is monitored through use of a quasi-experimental Randomised Controlled Trial (RCT) that measures outcomes in the project site with a control group. One reason PwC insist on an RCT for each project is to protect themselves against the risk of being sued for incorrectly withholding payment.

Where an RCT is not possible, for example in Afghanistan where security risks for implementers and cultural considerations mean that control groups are not feasible, these projects are removed from PbR. A number of the 37 GEC projects have been taken off PbR due to cultural considerations and challenging environments.

The ability to measure results is also dependent on the existence of consensus and evidence about expected results and effective means of measurement. This is more the case in the education sector than the WASH sector and makes setting targets and assessing progress towards them more difficult for a WASH programme, particularly in hygiene promotion activities such as those promoting hand washing.

As a result of these discussions, participants suggested the use of a spectrum (see Figure 1) that matches the proportion of programme funding dependent on PbR to how to how easy it is to measure results in that particular sector.

A potential PbR - measurement spectrum

Does PbR generate perverse incentives?
One of the much talked about cons of PbR is that it will create perverse incentives among stakeholders that drive them to behave in the opposite way than intended. Participants shared stories that included examples of the PbR mechanism inhibiting innovation and encouraging suppliers to focus on “low hanging fruit” rather than greatest need. A review of PbR in Swiss cantons suggested it didn’t work at all in terms of generating efficiencies and effectiveness.

PbR is not a one-size-fits-all solution; there remains lots to learn about when and where it can work
There was consensus that PbR cannot be used effectively in all contexts. The risks and uncertainties of working in fragile states are one setting in which PbR can be difficult (but not impossible?) to implement, however even in more stable contexts, issues around organisational capacity and motivations can inhibit its effective implementation.

Participants agreed that there are probably sets of circumstances in which PbR can be effective. People working in countries and sectors in which PbR has been used for many years have spent time identifying what those contexts are – see for example this paper about PbR in UK community work. For those of us who are new to the challenges of commissioning, designing, implementing, monitoring, evaluating or verifying PbR projects and are doing it in contexts in which it has not been tried, there is a lot to learn.  This chat show illustrates that there is great value in coming together to do so.

What do you think?
What are you learning about managing, implementing or verifying Payment By Results programmes? Does this discussion reflect your experience? Would you like to learn with us?  Please use the comment box to share your thoughts.

Catherine Fisher, Learning Adviser, WASH Results MVE Team.

See also: Payment-by-results: the elixir or the poison?, 6 January 2015, Jason Calvert