Leveraging the OECD/DAC criteria for impact evaluation; from principles to practice

This guide demystifies the OECD/DAC criteria, providing actionable insights for crafting impactful, context-sensitive evaluations that align with stakeholder needs.

Key Takeaways

  • Evaluation Framework: The OECD/DAC criteria cover effectiveness, impact, efficiency, relevance, coherence, and sustainability.
  • Practical Guidance: Criteria must be tailored to context and stakeholder priorities for meaningful evaluation.
  • Strategic Utility: Framework supports accountability, decision-making, and long-term intervention success.

In our previous article, we introduced the role of the OECD/DAC criteria in impact evaluation. In this article, we will delve into these criteria, from their underlying principles to their practical application, which can significantly enhance impact evaluations.

The purpose of the OECD/DAC criteria

The OECD/DAC criteria are designed to provide a multifaceted lens through which interventions can be evaluated. They not only assess the value and worth of interventions but also play a normative role in setting evaluation standards. Additionally, they aid in ensuring accountability by providing valuable information to donors and stakeholders. These criteria extend beyond evaluation and find utility in monitoring, results management, strategic planning, and intervention design.

Example of the OECD-DAC criteria conceptualized in cash and voucher assistance (CVA) use case
Example of the OECD-DAC criteria conceptualized in cash and voucher assistance (CVA) use case

OECD-DAC criteria principles

Understanding the principles underlying the OECD/DAC criteria is paramount for their effective application. The OECD-DAC criteria should be understood within a broader context and work in conjunction with evaluation principles. There are two principles that you need to keep in mind with the OECD evaluation process.

Principle one emphasizes that the criteria should be applied thoughtfully and contextualized. This means that you have to take into account the specific evaluation, the intervention that is being evaluated and the stakeholders that are involved. The interpretation and the evaluation of your criteria should then be informed by the evaluation questions and the intended use of the evaluation findings.

Principle two highlights that the application of the criteria should align with the purpose of the evaluation. The criteria should be tailored to meet the needs of the relevant stakeholders and the specific evaluation context. The amount of time and the amount of resources that are allocated for analyzing the criteria may vary depending on the evaluation's purpose. So factors such as data availability, resource constraints, the timing etc, all influence to a large extent whether a particular criterion is covered and how it is addressed.

It is important to keep in mind that other organizations have adapted these criteria and have expanded them. There can be other principles of evaluation or other criteria that might be relevant to your work and may not be covered by the six criteria.

Criterion 1: Effectiveness

The first criterion we have is effectiveness, and here we look at whether the intervention is achieving its objectives. This analysis takes into consideration the outputs and outcomes of the intervention. One important aspect of effectiveness is timeliness. This considers whether the intervention was implemented within the appropriate time frame. Issues of preparedness should also be addressed under this criteria. It assesses whether the intervention was sufficiently prepared and equipped to meet its objectives. The assessment helps you determine your overall success.

When you are applying effectiveness, there are few more considerations according to DAC:

  • Understanding intervention outcomes: It's essential to recognize not just if an intervention succeeded but also the reasons behind its success or failure. So an effective approach or a good approach to explore the factors is to utilize a checklist of cross cutting themes which can then help you examine aspects like stakeholder participation and the influence of local context.

  • Crafting measurable results statements: Frequently, result statements lack clear measurability, as objectives are often framed as activities rather than outcomes.

  • Analyzing objective formulation: During the evaluation process, it is vital for evaluators to analyze how objectives were formulated, who participated in the process, and the level of engagement from primary stakeholders in designing the intervention. This examination offers valuable insight into the intervention's effectiveness.

  • Assessing effectiveness with Logical Framework Approach (LFA): Utilizing LFA to simplify the evaluation of effectiveness involves analyzing specific objective statements. This process utilizes the logical framework approach, which simplifies the evaluation of effectiveness by focusing on these specific objective statements. Consequently, this approach aids evaluators in assessing the extent to which the intervention achieved its intended outcomes as outlined in the framework.

  • Evaluating resource utilization and benefits: Apart from assessing activities, evaluations should look into who utilizes and benefits from the allocated resources. Ideally, this analysis should include data segmented by gender, socio-economic status, and ethnicity to establish connections between effectiveness and impact.

  • Importance of timeliness in effectiveness: Timeliness is essential for effectiveness. Evaluations should assess whether interventions were timed appropriately to assist the affected population through various stages of crises, including promptly delivering support based on stakeholder perceptions.

Criterion 2: Impact

For impact, we ask; what difference does the intervention make? The evaluation of impact assesses the extent to which the intervention has generated or is expected to generate significant positive or negative effects at a higher level. These effects can be intended or unintended, but we have to keep in mind that it examines the overall impact of the intervention beyond its immediate outcomes, and it considers the broader consequences this intervention has on the target group or the larger context.

It's crucial to highlight that while evaluating effectiveness focuses on meeting intervention outputs and achieving objectives, assessing impact goes further by examining the long-term consequences of meeting or failing to meet those objectives.

When you are applying impact, there are few more considerations according to DAC:

  • Relevance of impact evaluation: The significance of impact evaluation depends on the timing and the availability of resources. Assessing impact may not be important for evaluations conducted immediately after an intervention, since the socio-economic and political changes take time to manifest. So impact evaluation requires a longitudinal approach where sufficient data for long term analysis is available, there's specialized expertise and then a commitment from either the donor or whoever is the commissioning body.

  • Addressing attribution challenges: It is essential to tackle attribution challenges when assessing long-term transformations. As time progresses, pinpointing the precise impact of an evaluation becomes intricate due to potential interferences from other initiatives or larger contextual shifts. Engaging with control groups, those not benefiting from the intervention, can aid in navigating this challenge.

  • Articulating results and using the Logical Framework Approach: While you are evaluating impact, it's very important to link it to your goal and to the purpose section of your logical framework (LogFrame). This allows you to understand how socio-political changes have occurred and what factors influence these changes in the long term. For example, an impact evaluation of a poverty alleviation program may not be relevant immediately after its intervention because poverty reduction takes time to measure. However, when you're conducting a longitudinal study with data collected over several years, then you can provide insights into the program's long term impact on poverty levels.

  • Considering livelihoods: When evaluating things like livelihood, the focus is on assessing the support provided by the intervention to the primary stakeholders. So this involves you considering the long term adaptive strategies for sustainable development such as promoting income generating activities or enhancing skills training.

Criterion 3: Efficiency

For efficiency, we ask, how well are resources being used? This inquiry examines the degree to which an intervention delivers results in a cost-effective and timely manner. Efficiency is measured by comparing the outputs, both qualitative and quantitative, to inputs that were used. It involves assessing whether the intervention has used the most efficient process in achieving the desired outcomes, often by comparing different approaches to achieving the same outputs. Cost-effectiveness also goes beyond the conversion of inputs into outputs, and it looks at whether different outputs could have been produced to have a greater impact in achieving the project's purpose. It involves the value of money and whether the intervention has achieved the desired outcomes and impacts in the most cost-effective way.

When you are applying efficiency, there are few more considerations according to DAC:

  • Influence of political factors: For instance, we must consider the political factors that can have a significant impact on the efficiency of the interventions. Government agenda, donor preference, all of this can lead to some inefficient practices such as discouraging long term settlements for refugees or prioritizing high visibility aid delivery methods.

  • Origin of inputs: The origin of inputs is another important aspect to consider. Here, for example, you examine whether goods and inputs were sourced locally or imported as this can have a significant impact on efficiency. For example, if you are procuring supplies for an intervention in Nairobi, in Kenya, and you go all the way to Geneva when you could have gone to a closer or more efficient location, that can affect efficiency. So, the use of local tenders is also a factor to consider.

  • Financial aspects for efficiency evaluation: Financial aspects play a crucial role as well. It's imperative to evaluate the comprehensive cost of the intervention across sectors, encompassing both local and international inputs, transportation expenses, and staff costs.

  • Logical Framework Approach and Efficiency: Evaluators with expertise in economics or accounting might be required to properly evaluate efficiency in some cases.

Criterion 4: Relevance

The question for this criterion is whether the intervention is doing the right thing. Here, we examine the extent to which the intervention's objectives and design is responding to beneficiaries’ needs, global needs, country needs, and policy needs priorities. If the circumstances change, this is a crucial aspect of relevance assessment. It involves considering whether the objectives and the design of the intervention are sensitive to context.

Relevance and appropriateness also go hand in hand in a lot of cases because they are interrelated criteria that are applicable at various stages of assessment. So relevance is particularly suitable for examining the broader aspects of the intervention including its overarching objectives to ensure alignments with local needs, priorities, and donor policies. On the other hand, appropriateness is more suited to assessing the specific inputs and actions that are involved in the intervention, ensuring that they are suitable and they are appropriate for achieving the intended outcomes.

When you are applying relevance, there are few more considerations according to DAC:

  • Importance of contextual analysis: It's very important for us to understand the local context and conduct a thorough needs assessment because these are essential for developing interventions that are relevant and appropriate. Evaluators need to carefully examine how well the planning, design and execution of interventions have taken into account local factors.

  • Adequacy of needs assessment: The adequacy of needs assessment is determined by its ability to identify and understand the diverse needs of the affected population and how external assistance can enhance livelihood strategies in many cases. So it should be participatory and the beneficiaries should be involved in determining the design of the project.

Criterion 5: Coherence

For coherence, we are assessing how well the intervention fits, so the compatibility of the intervention with other interventions in the country, sector or institution is a very important aspect to consider. It involves assessing how well the intervention aligns with and supports other interventions, policies, initiatives, and vice versa.

The assessment encompasses both internal and external coherence. Internal coherence focuses on the synergies and interlinkages between the intervention and other interventions that are carried out by the same institution or government. And it examines whether the intervention is consistent with the relevant international norms and standards that the institution or government adheres to. External coherence is considering the consistency of the intervention with the interventions of other factors within the same context. So it involves you assessing the complementarity, harmonization, coordination with other interventions. For instance, if multiple organizations are working on improving access to clean water in a community, the intervention should be coordinated to avoid duplication and to maximize impact.

Criterion 6: Sustainability

And lastly, we are dealing with sustainability, where you are asking the question, will the benefits last? So the extent to which the net benefits of an intervention continue, or are likely to continue, is very important. It involves you examining the financial, economic, social, environmental and institutional capacities of the systems required to sustain the net benefits over time.

Conclusion:

In summary, the OECD/DAC criteria offer a comprehensive framework for evaluating the impact of interventions, from principles to practical application. These criteria serve multiple purposes, including assessing value, ensuring accountability, and guiding strategic planning and intervention design.

Built upon the principles of thoughtful application and alignment with evaluation purposes, the OECD/DAC criteria provide a structured approach to evaluating effectiveness, impact, efficiency, relevance, coherence, and sustainability. By considering these criteria within the broader context of evaluation principles, organizations can tailor their evaluations to specific contexts and stakeholder needs.

Moreover, the criteria facilitate a deeper understanding of intervention outcomes, resource utilization, contextual relevance, and long-term sustainability. They also encourage coordination and coherence among interventions to maximize impact and minimize duplication.

In leveraging the OECD/DAC criteria, organizations can enhance their ability to assess the value and significance of interventions, ultimately contributing to more effective and sustainable development outcomes.

Resources and further reading: