Tag Archives: indicators

What can we learn from the Government Outcomes Lab?

Our Learning Advisor, Catherine Fisher, investigates a new research centre on outcomes-based commissioning and finds plenty of interest for the WASH Results Programme.

You know an idea has traction when the University of Oxford sets up a research centre to investigate it. This is the case for outcomes-based commissioning (aka Payment by Results), which is the focus of the new Government Outcomes Lab (GO Lab) based at the University of Oxford’s Blavatnik School of Government.

The GO Lab focusses on the interests and experience of government departments in procuring services in an outcomes-based way, rather than those of contractors (or suppliers, in WASH Results terminology) in providing them. It is a collaboration between Blavatnik School of Government and Her Majesty’s Government. The focus of the research centre is on outcomes-based commissioning that uses Social Impact Bonds (SIBs), a model in which external investors provide the initial investment for programme implementation which is repaid on achievement of outcomes. However, the GO Lab will also look at other models, presumably including that used in the WASH Results Programme in which the suppliers themselves provide the upfront investment.

The rationale for the GO Lab is as follows: “While numbers of, and funding for, outcomes-based approaches have increased over recent years, research has not kept pace with this speed of growth. Much is still unknown about whether outcomes-based commissioning will meet its promise….Through rigorous academic research, the GO Lab will deepen the understanding of outcomes-based commissioning and provide independent support, data and evidence on what works, and what doesn’t.”  (GO Lab FAQ) .

So far, the GO Lab has organised three “Better Commissioning” events looking at outcomes-based commissioning in different sectors in a UK context, namely Children’s Services, Older People’s Services and promotion of Healthy Lives.

A quick skim of the interesting post-event reports suggests that outcomes-based commissioning is seen as a way of promoting a greater focus on outcomes by providers (who may not already think in this way), of prompting innovation in service provision and of transferring the risk of new approaches from commissioners to socially-minded private enterprises. Similar themes occur in Results Based Aid discussions, although I’d suggest that the international development sector places a slightly greater emphasis on incentivising delivery, value for money and accountability to commissioners.

One aspect of the GO Lab work that caught my eye is their interest in the creation of Outcomes Frameworks which were discussed at each event: “Developing an outcomes framework is a key part of any SIB or outcome based contract, but accessing data and articulating robust metrics that can be rigorously defined and measured is often seen as a challenge by commissioning authorities.”  (Better Commissioning for Healthy Lives: a Summary Report, p 13).

This process of articulating appropriate metrics and identifying indicators has been a key area of learning within the WASH Results Programme and continues to be discussed. It was reassuring to see others grappling with similar challenges in different sectors, such as challenges of creating indicators that:

During this, the outcomes-focused period of the WASH Results Programme, we will be following the progress of the GO Lab with interest, and hope to find opportunities to exchange learning with them, and others researching innovative funding approaches. Our team is particularly interested in contributing to, and benefiting from, learning around:

  • independent monitoring and verification of outcomes-based contracts;
  • creating outcomes frameworks that reflect sustained outcomes in areas such as behaviour change (e.g. handwashing behaviours) and institutional change (e.g. ability of district stakeholders to manage water systems);
  • streamlining metrics and indicators while balancing needs of all parties: beneficiaries, service providers, commissioners and, in the case of international development programmes, national stakeholders and global commitments such as Sustainable Development Goals.

Catherine Fisher, Learning Advisor, WASH Results MVE Team

If you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us

Measuring progress towards SDGs: a Payment by Results perspective

Attending the 2017 WEDC Conference prompted our team members to share their reflections on measuring progress towards SDGs from a Payment by Results (PBR) perspective.

Some of the e-Pact Monitoring and Verification (MV) team recently attended the WEDC Conference – an annual international event focused on water, sanitation and hygiene (WASH), organised by the Water, Engineering and Development Centre (WEDC) at Loughborough University. One of the key themes this year was the Sustainable Development Goals (SDGs): what they are, how close we are to achieving them, and how we are going to monitor them. The SDGs are important for PBR programmes because they influence what programmes aspire to achieve and how they measure their progress.

The recent publication of the first report (and effective baseline) on SDG 6, covering drinking water, sanitation and hygiene, marked a watershed. With the shift to understanding universal and equitable access, the inclusion of hygiene and a focus on ‘safely managed’ and ‘affordable access’, the breadth and depth of data we aspire to have on water and sanitation services is unprecedented. But the first SDG progress report also highlights a yawning data gap: for example, estimates for safely managed drinking water are only available for one third of the global population, and we are only starting to get to grips with how to measure affordable levels of service.

As part of the WASH Results Programme, the three consortia are constantly grappling with how to objectively measure complex outputs and outcomes linked to water, sanitation and hygiene. At the same time our MV team is trying to understand how we can verify such measures, and if they are sufficiently robust to make payments against. How do the SDGs influence this process? We have three reflections from our experience of verifying results under the WASH Results Programme:

Reflection 1: the relationship between the SDGs and PBR-programming can be mutually beneficial.

The SDGs help PBR programmes to set ambitious benchmarks

It’s clear that to track progress against the SDGs, the WASH sector is going to have to become a lot better at collecting, managing and analysing an awful lot of data. One of the learning points from the WASH Results Programme is that the verification process requires the consortia (and in-country partners) to take data far more seriously.

Compared to more conventional grant programmes, Monitoring and Verification functions take on the importance of financial reporting. One function of this, is that everyone has more confidence that reported results (whether access to water, number of latrines built or handwashing behaviour) accurately reflect reality. As such, PBR programmes can help focusing peoples’ attention on improving service levels.

Conversely, the SDGs help PBR programmes to set ambitious benchmarks and provide an orientation on how to measure them. This is proving important under the WASH Results Programme, which has, at times, struggled with aligning definitions, targets, indicators and how to measure them.

Reflection 2: some of the SDG targets are hard to incorporate into a PBR programme

Physical evidence of a handwashing facility doesn’t guarantee use at critical times

Measuring hygiene behaviour change illustrates this point neatly: the simplest way to understand if people are washing their hand with soap may appear to be just to go out and ask them. Yet self-reported behaviour indicators are notoriously unreliable. Looking for physical evidence of a handwashing facility (with water and soap) is the approach currently suggested by the WHO/UNICEF Joint Monitoring Program (JMP), but there is no guarantee that people use such facilities at the most critical times, for example, after defecation or before handling food.

Under a PBR programme (where implementers get paid against pre-defined results) the temptation to take the shortest route to success, namely focusing on getting the hardware in place, may be high. Therefore, it may be important to complement this indicator with a knowledge-related indicator to also capture behaviour change albeit in a crude way. This brings along another challenge: how to agree on appropriate, payment-related targets in a situation where experience on how to accurately measure behaviour change is still in its infancy?

Reflection 3: keeping indicators lean is challenging when faced with the breadth and depth of the SDGs

Hygiene behaviour change is just one indicator. Attempting to robustly measure changes across three consortia, eight result areas and two phases (output and outcome) has resulted in the MV team reviewing a large amount of surveys, databases, and supporting evidence since 2014.

Under the WASH Results programme, the sustainability of services is incentivised via payment against outcomes: people continuing to access water and sanitation facilities and handwashing stations for up to two years after they gained access to improved services. In the meantime, between the final MDG report, and the initial SDG report, the number of data sources used by JMP to produce estimates for the water, sanitation and hygiene estimates has more than doubled[1]. Instead of more traditional household surveys, increasingly, data is obtained from administrative sources such as utilities, regulators and governments.

How to marry these new data ambitions with the necessary goal to keep the number of indicators manageable under a PBR programme will be an interesting challenge going forward.

Katharina Welle and Ben Harris, MV Team, WASH Results

[1] Progress on drinking water, sanitation and hygiene: 2017 update and SDG baselines (p50) https://washdata.org/reports 

Is it time for Results 2.0?

Catherine Fisher reports back on a Bond Conference panel discussion about the origins and future of the ‘results agenda’.

Aid should generate results. On the face of it, an indisputably good idea, but the ‘results agenda’ is anything but uncontroversial and can spark “epic” debates. In the WASH Results Programme, this agenda is manifest in the funding relationship – Results Based Financing (RBF) – a form of Payment By Results (PBR) through which DFID makes payments to Suppliers contingent on the independent verification of results. As the Monitoring, Verification and Evaluation (MVE) providers for WASH Results, we’re keen to exchange the programme’s insights with those of other people who have first-hand experience of the results agenda.

25093906970_1f78a254e6_o

Credit: Bond/Vicky Couchman. Jessica Horn and Irene Guijt sharing their views on the ‘results agenda’ at the Bond Conference 2016.

One such opportunity arose this week at the Bond Conference, in a session entitled ‘How to crack Results 2.0’ chaired by Michael O’Donnell, Head of Effectiveness and Learning at Bond and author of ‘Payment by Results: What it means for UK NGOs’. The session considered the origins and implications of the results agenda and looked ahead to the next version. Catherine Fisher, Learning Advisor for the WASH Results MVE team, reports on a lively discussion about how results agendas could be aligned with work on social transformation, enable learning and reflection within programmes, and provide value for money themselves.

Looking back to the origins of results approaches in DFID

Opening the session, Kevin Quinlan, Head of Finance, Performance and Impact at DFID explained how, in 2010, DFID encountered two opposing forces: increased funding in order to meet the UK’s legal commitment to spending 0.7 percent of national income on Overseas Development Aid alongside the introduction of austerity measures that required cuts in, and increased scrutiny on, public spending. Results and transparency agendas were DFID’s response to those competing demands and made a shift towards delivering results now in comparison with systems strengthening for results in future. This implied a corresponding shift to talking about the results DFID is going to support rather than the activities they would support to achieve results in future. Six years on, DFID is reassessing its approach.

Can results approaches be reconciled with the “art of transformation”?

Earlier that day, Dr Danny Sriskandarajah, Secretary General of CIVICUS, told conference delegates that INGOs had become too focused on the “science of delivery” (which he described as the achievement of impact by any means) as opposed to the “art of transformation” – the work of bringing about social change. This theme re-emerged during the ‘Results 2.0’ discussion: how could the focus on hard results, embedded in results frameworks, be reconciled with the messy business of social transformation that is at the heart of struggles for equity and rights?

Jessica Horn, Director of Programmes at the African Women’s Development Fund, noted that results frameworks do not acknowledge power or monitor how it is transformed. Consequently she and her colleagues resort to what she called “feminist martial arts” – twisting and turning, blocking and jabbing to defend the transformative work they do, from the “tyranny of results”.  Often, Jessica argued, the politics of the process are as important as the politics of the outcome and asked “how does the results framework capture that?” Yet as Irene Guijt, newly appointed Head of Research at Oxfam GB, argued, being forced to think about results even in the social transformation context helps to make things clearer. Between them, they had some suggestions about how it could be done.

Irene contended that there needed to be greater differentiation of what kind of data we need for different reasons, rather than a one-size-fits-all approach to accountability. She argued that “results” are too often about numbers and we need to bring humans back in and tell the story of change. Irene recommended using the tool SenseMaker to bring together multiple qualitative stories which, through their scale, become quantifiable. Jessica shared some frameworks for approaching monitoring and reporting on social transformation more systematically and in ways that consider power, such as Making the Case: the five social change shifts and the Gender at Work Framework.

Does focusing on monitoring results for accountability squeeze out reflection and learning?

This criticism is often levelled at results-based approaches and their associated heavy reporting requirements. Irene commented that “learning and data are mates but compete for space”. To align learning and reflection with results monitoring, she advised focusing on collective sense-making of reporting data, a process that enables evidence-based reflection and learning. She also suggested streamlining indicators focussing on those with most potential for learning, a point echoed by Kevin from DFID who emphasised the need to select indicators that are most meaningful to the people implementing programmes (rather than donors).

Do results agendas themselves demonstrate value for money?

This question resonated with the participants, triggering musings on the value of randomised controlled trials and the cost of management agents from the private sector. One point emerging from this discussion was that often what is asked for in results monitoring is difficult to achieve. Indeed, this has at times, been the experience of the WASH Results Programme, particularly in fragile contexts (see for example, the SWIFT Consortium’s report [PDF]) . Both Irene and Jessica talked of the need to use a range of different tools for different purposes and Irene made reference to her recent work on balancing feasibility, inclusiveness and rigour in impact assessments.

What is the trajectory for DFID and the results agenda?

Kevin Quinlan took this question head on, agreeing that this is something DFID needs to decide in the next few months. He suggested that some of the areas for discussion were:

  • Getting to a more appropriate place on the spectrum between communication (to tax-payers) and better programme design; results are part of communicating to tax-payers but not the only part;
  • Reducing standard indicators in favour of flexible local indicators; each project would need at least one standard indicator to allow aggregation but there should be more local indicators to enable learning;
  • Alleviating the torture of results – “rightsizing” the reporting burden and reducing the transaction costs of results reporting; thinking about what results can do alongside other tools.
  • Adopting a principles-based approach rather than a set of rules.

Meanwhile the Evaluation Team for WASH Results is investigating some of the issues raised during the panel such as examining the effect of results verification on Suppliers’ learning and reflection, and seeking to explore the value for money of verification.

So it sounds like there will be more interesting discussions about the results agenda in the near future and we look forward to contributing insights from WASH Results*. Whether Results 2.0 is on the horizon remains to be seen.

* Please email the MVE Team if you would like us to let you know when our evaluation findings are available.