Tag Archives: government

Reflections on our experience of DFID’s results agenda

As verifiers of a DFID Results Based Finance programme, ODI’s research on the UK’s results agenda prompted us to reflect on our experience.

 

Kakimat latrine eaten by goats

Why context matters when you focus on results #1: Some latrine building projects have to allow for the impact of hungry goats. Photo credit: Chamia Mutuku

 

In their report ‘The Politics of the Results Agenda in DFID: 1997 to 2017’, Craig Valters and Brendan Whitty argue that 2007 saw a new explicit focus from DFID on aggressively implementing results-based management. 10 years later, we have WASH Results: a DFID-funded programme where financial risk has been completely transferred away from UK taxpayers to the international NGOs who deliver the work and who only get paid for results that have been checked by a third party – us. However, as its name promises, the programme is delivering results in water, sanitation and hygiene (WASH). DFID was able to read in the programme’s 2017 annual report (with great confidence in the figures), for example, that WASH Results had reached over 1.1 million people with improved water supply, more than 4.7 million people with improved sanitation, and over 14.9 million people with hygiene promotion.

In our role as Monitoring, Verification and Evaluation Team for the WASH Results programme, our attention is less focused on the politics of the results agenda, and more in how results are monitored and verified and the very real impact that this approach has on ongoing programme delivery. However, we read the report and blog post by Valters and Whitty with great interest.

After more than three years of supporting the programme, how does our experience compare with the conclusions and recommendations of the ODI report? One key finding from the research is that some DFID staff have found ways to adhere to the results agenda, while retaining flexibility. This theme of the ways in which both donors and programme implementors are working creatively around the “tyranny of results” was one that we heard during last year’s BOND Conference session ‘How to crack Results 2.0’.

How can PBR be adapted to address the imbalance in accountability?

We absolutely agree with Valters and Whitty about the importance of finding a balance between being accountable to UK citizens and to the beneficiaries (poor people abroad). This time last year, we shared our opinion that if verification was designed to include beneficiary feedback and this was linked to payment, Payment by Results (PBR) could actually generate more downward accountability than other funding modalities. However, our team of verifiers felt that the demands of verification for large scale, representative, quantitative information on which to base payment decisions may leave less time, money and inclination to undertake more qualitative work with beneficiaries. So, we suggested that a resource-effective solution to upholding downwards accountability through verification would be to include payment for the existence and effective functioning of a beneficiary feedback system (rather than the results of that system). Payment would be made on verification of the effectiveness of the system in promoting downwards accountability.

We welcome the authors’ call to “Create a results agenda fit for purpose”. Our first reflection would be that a results agenda, at least one hard-wired into a PBR modality, is not going to be appropriate in all contexts and for all intended outcomes, particularly those where outcomes are difficult to predict or challenging to measure. Our set of recommendations to commissioners of PBR programmes, complement several of those made by ODI, for example, their suggestion that DFID spend more time considering whether its aid spending has the right mix of risks and the view that regular testing (that leads to course-correction) is important.

The challenge of communicating about costs and value

The authors also call on ministers to be honest with the British public about aid. Part of this, we feel, is making it clearer that Value for Money (VFM) is not synonymous with “cheap”. We feel that the results agenda, particularly a PBR model, should require donors/commissioners to clearly articulate the “value” they expect to see in VFM. Otherwise the importance placed by a donor on achieving clearly costed, verified results could risk squeezing out other values and principles that are central to development programming. A central theme in last year’s WASH Results learning workshop was the ongoing puzzle of how to place value (both in commercial/financial and VFM terms) on intangible aspirations and benefits, such as reaching the most vulnerable and investing in the processes and social capital that underpin effective programming. This is particularly important in an increasingly commercialised aid context, where one supplier’s approach would be to parachute in and build many toilets very quickly and cheaply, whereas another proposes taking longer to work with local stakeholders, norms and materials. This articulation of value may not be as simple as it sounds, when every commitment in a PBR programme, such as reaching the poorest, gender equity, national ownership, sustainable outcomes, etc. needs to be reflected in meaningful and measurable indicators.

Payment By Results can aid course correction

Interestingly, one of the reforms that the authors call for may be an inherent feature of the results framework itself. They say that “interventions need to be based on the best available information, with regular testing to see if they are on the right track”. We have found that a product of the PBR modality is that much greater emphasis is placed on monitoring systems and the generation of reliable data about what is happening within programmes. In WASH Results we have seen cases where the rigorous (compulsive?) tracking of results has identified areas where programmes are failing to deliver and rapid action has then been taken to address that failure. As verification agents we argue that this is due not only to the link between results and payment but also the independent verification of data and systems that has led to better information on which to base decision-making.

Benefits of the results agenda

In this way we think that the focus on monitoring within the results agenda, can, in some cases, enable flexibility and innovation. In its reliance on high quality data, it contains within it a driver that could improve the way that development work happens. The results agenda brings benefits – some of which we did not see reflected in the article – but it comes with risks; both ideological about the ambitions for UK Aid and practical for those involved in its delivery. And so we welcome this debate.

Catherine Fisher, Learning Advisor, WASH Results MVE Team

If you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us

Alignment, aid effectiveness and Payment by Results

To what extent does the Payment by Results approach of the WASH Results Programme follow the aid effectiveness principle of alignment?
One argument for Payment by Results (PBR) is that it can promote “alignment”, which is also an important principle in aid effectiveness. So that’s good, right? But, a closer look at how this slippery term is used reveals differences in understanding that are particularly relevant to the use of PBR in International Development.

According to some PBR commentators, PBR can bring advantages in situations where there is misalignment between the objectives of donors and implementers, but there is some debate about this argument, (see, for example, CGD’s commentary on principle 5 of Clist & Dercon’s 12 principles of PBR).  Either way, the alignment in question here is between the objectives of donor and implementer (or Suppliers as we call them in WASH Results; in our case either individual – SNV, or consortia of, non-governmental organisations – SAWRP and SWIFT).

Compare this with the understanding in the Paris Declaration in which alignment is one of the five principles of Aid Effectiveness. The first principle of Ownership states: “Developing countries set their own strategies for poverty reduction, improve their institutions and tackle corruption.”. The second principle, Alignment, builds on this: “Donor countries align behind these objectives and use local systems.”. In this case, the alignment is that of donors behind national strategies and objectives.

The PBR funding mechanism for the WASH Results Programme is the type that DFID calls Results Based Finance. Under this approach, the contract is between a donor and a service provider, not recipient governments (DFID calls the latter Results Based Aid*). In this context, the term “alignment” as used in the PBR literature may be at odds with the concept of alignment in the Paris Declaration for Aid Effectiveness as it encourages alignment between service provider and donor rather than donor with national stakeholders and priorities. This has led some people to claim that PBR promotes upwards accountability to donors at the expense of accountability to national and local stakeholders.

Experience of alignment with national government stakeholders under the WASH Results Programme

At a WASH Results learning workshop held earlier this year, participants shared their views on alignment in the context of the WASH Results Programme. Over the last year of implementation, some concerns were raised that the PBR modality was a barrier to alignment with national priorities and stakeholders in the countries in which the WASH Results is being implemented. During the learning workshop a nuanced picture emerged of the programme’s experience to date as this extract from the workshop report demonstrates:


Alignment is happening, whether incentivised by PBR or not:  All of the Suppliers work with local stakeholders as a matter of course. However differences in programme design affected to what extent this was incentivised or recognised in payment packages. All of the Suppliers had experienced positive reactions from local stakeholders to the principle of PBR – with one Supplier being asked by local government officials for support in rolling out PBR in one of their programmes. 

Value of building alignment into results packages: There was some sense that the focus on outputs in the first phase of WASH Results had taken attention away from areas such as alignment that are not so easily linked to milestones and so opportunities for alignment had been missed. However, SNV took a different approach to other suppliers by building concrete items into their results packages to reflect their work with local partners, e.g. district plans in each of 60 districts in which they work. While this was felt to be a “smart” approach – SNV warned that there are also disadvantages: “We think we have found some meaningful ways to address elements of alignment, but let’s not be too optimistic about these instruments; they focus attention on direct deliverables” (Jan Ubels, SNV).

Flexibility supports alignment: Suppliers value being allowed to change their approach without going through a contract amendment process. In one case a Supplier was able to change definitions of results to better align with national government definitions.  However, there are potential risks in this approach: “Alignment to what? If the government has a much lower CLTS standard than the SDGs – is that still the alignment we are trying to encourage?” (Louise Medland, SAWRP) 

 Challenge of timelines:  WASH Results has tight deadlines and an emphasis on deliverables while partners, e.g. water authorities, are working to a different longer timeline and may not deliver at the pace required. There is a limit to how much risk can be transferred to partners in this context.

PBR risks limiting Suppliers to existing relationships: participants agreed that PBR can only be introduced where there is existing relationships and social capital and it would be risky to try to implement PBR in places where there was no established relationship.

Additional demands of monitoring for PBR: One Supplier felt that the kind of monitoring carried out for WASH Results could never be the same as that carried out at a local level; it would always be additional to, rather than aligned with, that of the government, although it might stimulate M&E at a local level.

Ways in which alignment could be promoted in future Results Based Finance forms of PBR

To support alignment within a PBR mechanism, participants in the workshop suggested:

  • Valuing alignment at tendering and contracting stage:   Alignment should be considered when costing at the tendering and contracting stage so that prospective Suppliers are competing on an equal basis, given the additional cost (and value) alignment brings.
  • Defining specific hard deliverables, perhaps during a pre-inception phase, that were somewhere between output and outcome phase e.g. district plans.
  • Including specific rewards or incentives in the programme aimed at government to encourage their buy-in to the programme
See especially pages 9 – 10 of DFID WASH Results Programme: Learning Event. e-Pact Consortium, Hove, UK (2016)

Conclusions and looking forward

For PBR to be accepted as an effective form of aid financing it will need to follow all the principles of aid effectiveness, including alignment. The experience of WASH Results so far suggests that this is possible but requires careful consideration of how alignment can be promoted during the design of programmes, contracting and tendering processes,  definition of results and design of verification systems.

Another, more macro, way of supporting alignment using PBR, is for the Independent Verifiers to work much more closely with the national government monitoring systems to verify results. In this model, significant support is given by the donor to improve the national systems and then recipient countries themselves can do the verification. In this case, the use of PBR to fund service delivery would act as a catalyst for strengthening monitoring systems at a national level (although the PBR programmes would need to be of significant scale to be an effective catalyst). This sector-strengthening approach requires long-term investment and a multi-pronged approach, within which PBR projects may only be one element, albeit a potentially catalytic one.

We have not seen much focus on this area so far in the debates around PBR (please alert us to it if we are wrong!). We hope that the experience of our programme will help contribute to that understanding. We will continue to share ongoing lessons learned from implementation as well as findings from the evaluation.

Catherine Fisher, Learning Advisor, WASH Results MVE Team

* For an example of a Payment By Results programme in WASH that uses Results Based Aid (where payment goes from the donor to a recipient government), we suggest readers take a look at DFID’s Support to Rural Water Supply, Sanitation & Hygiene in Tanzania.

The report from the WASH Results learning workshop is available to download from DFID’s Research for Development website. 

As always, if you have any ideas or observations about this topic, we encourage you to Leave A Reply (below), or email us.