Category Archives: Learning from others

Reflections on our experience of DFID’s results agenda

As verifiers of a DFID Results Based Finance programme, ODI’s research on the UK’s results agenda prompted us to reflect on our experience.

 

Kakimat latrine eaten by goats

Why context matters when you focus on results #1: Some latrine building projects have to allow for the impact of hungry goats. Photo credit: Chamia Mutuku

 

In their report ‘The Politics of the Results Agenda in DFID: 1997 to 2017’, Craig Valters and Brendan Whitty argue that 2007 saw a new explicit focus from DFID on aggressively implementing results-based management. 10 years later, we have WASH Results: a DFID-funded programme where financial risk has been completely transferred away from UK taxpayers to the international NGOs who deliver the work and who only get paid for results that have been checked by a third party – us. However, as its name promises, the programme is delivering results in water, sanitation and hygiene (WASH). DFID was able to read in the programme’s 2017 annual report (with great confidence in the figures), for example, that WASH Results had reached over 1.1 million people with improved water supply, more than 4.7 million people with improved sanitation, and over 14.9 million people with hygiene promotion.

In our role as Monitoring, Verification and Evaluation Team for the WASH Results programme, our attention is less focused on the politics of the results agenda, and more in how results are monitored and verified and the very real impact that this approach has on ongoing programme delivery. However, we read the report and blog post by Valters and Whitty with great interest.

After more than three years of supporting the programme, how does our experience compare with the conclusions and recommendations of the ODI report? One key finding from the research is that some DFID staff have found ways to adhere to the results agenda, while retaining flexibility. This theme of the ways in which both donors and programme implementors are working creatively around the “tyranny of results” was one that we heard during last year’s BOND Conference session ‘How to crack Results 2.0’.

How can PBR be adapted to address the imbalance in accountability?

We absolutely agree with Valters and Whitty about the importance of finding a balance between being accountable to UK citizens and to the beneficiaries (poor people abroad). This time last year, we shared our opinion that if verification was designed to include beneficiary feedback and this was linked to payment, Payment by Results (PBR) could actually generate more downward accountability than other funding modalities. However, our team of verifiers felt that the demands of verification for large scale, representative, quantitative information on which to base payment decisions may leave less time, money and inclination to undertake more qualitative work with beneficiaries. So, we suggested that a resource-effective solution to upholding downwards accountability through verification would be to include payment for the existence and effective functioning of a beneficiary feedback system (rather than the results of that system). Payment would be made on verification of the effectiveness of the system in promoting downwards accountability.

We welcome the authors’ call to “Create a results agenda fit for purpose”. Our first reflection would be that a results agenda, at least one hard-wired into a PBR modality, is not going to be appropriate in all contexts and for all intended outcomes, particularly those where outcomes are difficult to predict or challenging to measure. Our set of recommendations to commissioners of PBR programmes, complement several of those made by ODI, for example, their suggestion that DFID spend more time considering whether its aid spending has the right mix of risks and the view that regular testing (that leads to course-correction) is important.

The challenge of communicating about costs and value

The authors also call on ministers to be honest with the British public about aid. Part of this, we feel, is making it clearer that Value for Money (VFM) is not synonymous with “cheap”. We feel that the results agenda, particularly a PBR model, should require donors/commissioners to clearly articulate the “value” they expect to see in VFM. Otherwise the importance placed by a donor on achieving clearly costed, verified results could risk squeezing out other values and principles that are central to development programming. A central theme in last year’s WASH Results learning workshop was the ongoing puzzle of how to place value (both in commercial/financial and VFM terms) on intangible aspirations and benefits, such as reaching the most vulnerable and investing in the processes and social capital that underpin effective programming. This is particularly important in an increasingly commercialised aid context, where one supplier’s approach would be to parachute in and build many toilets very quickly and cheaply, whereas another proposes taking longer to work with local stakeholders, norms and materials. This articulation of value may not be as simple as it sounds, when every commitment in a PBR programme, such as reaching the poorest, gender equity, national ownership, sustainable outcomes, etc. needs to be reflected in meaningful and measurable indicators.

Payment By Results can aid course correction

Interestingly, one of the reforms that the authors call for may be an inherent feature of the results framework itself. They say that “interventions need to be based on the best available information, with regular testing to see if they are on the right track”. We have found that a product of the PBR modality is that much greater emphasis is placed on monitoring systems and the generation of reliable data about what is happening within programmes. In WASH Results we have seen cases where the rigorous (compulsive?) tracking of results has identified areas where programmes are failing to deliver and rapid action has then been taken to address that failure. As verification agents we argue that this is due not only to the link between results and payment but also the independent verification of data and systems that has led to better information on which to base decision-making.

Benefits of the results agenda

In this way we think that the focus on monitoring within the results agenda, can, in some cases, enable flexibility and innovation. In its reliance on high quality data, it contains within it a driver that could improve the way that development work happens. The results agenda brings benefits – some of which we did not see reflected in the article – but it comes with risks; both ideological about the ambitions for UK Aid and practical for those involved in its delivery. And so we welcome this debate.

Catherine Fisher, Learning Advisor, WASH Results MVE Team

If you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us

What can we learn from the Government Outcomes Lab?

Our Learning Advisor, Catherine Fisher, investigates a new research centre on outcomes-based commissioning and finds plenty of interest for the WASH Results Programme.

You know an idea has traction when the University of Oxford sets up a research centre to investigate it. This is the case for outcomes-based commissioning (aka Payment by Results), which is the focus of the new Government Outcomes Lab (GO Lab) based at the University of Oxford’s Blavatnik School of Government.

The GO Lab focusses on the interests and experience of government departments in procuring services in an outcomes-based way, rather than those of contractors (or suppliers, in WASH Results terminology) in providing them. It is a collaboration between Blavatnik School of Government and Her Majesty’s Government. The focus of the research centre is on outcomes-based commissioning that uses Social Impact Bonds (SIBs), a model in which external investors provide the initial investment for programme implementation which is repaid on achievement of outcomes. However, the GO Lab will also look at other models, presumably including that used in the WASH Results Programme in which the suppliers themselves provide the upfront investment.

The rationale for the GO Lab is as follows: “While numbers of, and funding for, outcomes-based approaches have increased over recent years, research has not kept pace with this speed of growth. Much is still unknown about whether outcomes-based commissioning will meet its promise….Through rigorous academic research, the GO Lab will deepen the understanding of outcomes-based commissioning and provide independent support, data and evidence on what works, and what doesn’t.”  (GO Lab FAQ) .

So far, the GO Lab has organised three “Better Commissioning” events looking at outcomes-based commissioning in different sectors in a UK context, namely Children’s Services, Older People’s Services and promotion of Healthy Lives.

A quick skim of the interesting post-event reports suggests that outcomes-based commissioning is seen as a way of promoting a greater focus on outcomes by providers (who may not already think in this way), of prompting innovation in service provision and of transferring the risk of new approaches from commissioners to socially-minded private enterprises. Similar themes occur in Results Based Aid discussions, although I’d suggest that the international development sector places a slightly greater emphasis on incentivising delivery, value for money and accountability to commissioners.

One aspect of the GO Lab work that caught my eye is their interest in the creation of Outcomes Frameworks which were discussed at each event: “Developing an outcomes framework is a key part of any SIB or outcome based contract, but accessing data and articulating robust metrics that can be rigorously defined and measured is often seen as a challenge by commissioning authorities.”  (Better Commissioning for Healthy Lives: a Summary Report, p 13).

This process of articulating appropriate metrics and identifying indicators has been a key area of learning within the WASH Results Programme and continues to be discussed. It was reassuring to see others grappling with similar challenges in different sectors, such as challenges of creating indicators that:

During this, the outcomes-focused period of the WASH Results Programme, we will be following the progress of the GO Lab with interest, and hope to find opportunities to exchange learning with them, and others researching innovative funding approaches. Our team is particularly interested in contributing to, and benefiting from, learning around:

  • independent monitoring and verification of outcomes-based contracts;
  • creating outcomes frameworks that reflect sustained outcomes in areas such as behaviour change (e.g. handwashing behaviours) and institutional change (e.g. ability of district stakeholders to manage water systems);
  • streamlining metrics and indicators while balancing needs of all parties: beneficiaries, service providers, commissioners and, in the case of international development programmes, national stakeholders and global commitments such as Sustainable Development Goals.

Catherine Fisher, Learning Advisor, WASH Results MVE Team

If you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us