Evaluating: criteria, terms of reference, matrix and use

1Why and when to evaluate

An evaluation is a systematic and objective assessment of an intervention, its design, its implementation and its results. It differs from monitoring: monitoring is continuous and internal, and asks "are we doing what we planned?"; evaluation is periodic, often external, and asks "are we doing the right thing, in the right way, and what difference does it make?".

It has two functions, sometimes in tension: accountability (reporting to people, donors and the board) and learning (improving this programme and future ones). An evaluation designed only for accountability produces scores; an evaluation designed for learning produces decisions.

Before commissioning an evaluation, write down who will use it and for what decision. If no one can answer, the evaluation will be one more report on a shelf.

2The OECD-DAC criteria

The evaluation criteria of the OECD Development Assistance Committee, revised in 2019, are six in number:

• Relevance: does the intervention respond to people's needs and priorities, and does it remain relevant if the context changes?
• Coherence (added in 2019): is it compatible with other interventions in the same context, both internal and external to the organisation?
• Effectiveness: does it achieve its objectives, including for different groups?
• Efficiency: does it deliver its results economically and on time?
• Impact: what longer-term effects, positive or negative, intended or not?
• Sustainability: will the benefits continue after funding ends?

In humanitarian work, the criteria adapted by ALNAP are also used: coverage (have all people in need been reached, without exclusion?), connectedness (link between the emergency and the longer term) and coordination.

The criteria do not all carry the same weight. A real-time evaluation of a three-month response focuses on relevance, coverage and effectiveness; talking about sustainability at that stage makes little sense. Choosing three or four criteria and going into them in depth is better than skimming over six.

3Terms of reference that get a good evaluation

The terms of reference (ToR) are the intellectual contract of the evaluation. Vague ToR produce vague evaluations. They contain:

• Context and description of the intervention (with the theory of change and the logframe in annex).
• Purpose and users: why now, for whom, for which decisions.
• Scope: period, areas, components, what is excluded.
• Evaluation questions: five to eight main questions, not thirty. Each question can be evaluated with the means and time available.
• Expected approach and methods, leaving the team to make proposals, and the existing data available.
• Ethics: consent, data protection, protocols for sensitive topics, participation of people.
• Deliverables and timeline: inception report, debriefing, draft report, final report, summary.
• Team profile, budget, governance (reference group) and independence rules.

Order of magnitude. A good-quality external final evaluation of a multisectoral project often costs between 25,000 and 80,000 USD depending on the field setting and methods. Too small a budget leads to a few days of fieldwork and a report without evidence; fewer evaluations, better done, is preferable.

4The evaluation matrix and the inception report

The evaluation matrix is the central tool of the inception report: for each question, it specifies the sub-questions, the indicators or judgement criteria, the data sources, the data collection and analysis methods, and the limitations.

Example. Question: "To what extent have cash transfers improved the food security of the most vulnerable households?" Judgement criteria: change in the food consumption score and rCSI between baseline and endline, compared with a comparison area; differences according to the sex of the head of household and the presence of a person with a disability. Sources: baseline and endline surveys, post-distribution monitoring, focus group discussions. Methods: difference-in-differences, qualitative analysis by group. Limitations: comparison area partially assisted by other actors from month 6.

The inception report is the moment to correct course: the team confirms what is feasible, flags questions that cannot be addressed, and proposes the sampling. The evaluation manager validates it with the reference group before fieldwork. It is the best investment of time in the whole evaluation.

The quality of an evaluation is judged on the clarity of the link between findings (supported by data), conclusions (judgements based on the findings) and recommendations (actions that flow from the conclusions). A recommendation without a corresponding finding is the evaluator's opinion.

5Independence, ethics and evaluation management

Independence protects credibility: the evaluation team took no part in the design or implementation; the evaluation manager is not the head of the project being evaluated; the team has free access to data and people; the programme may comment on the facts in the draft report, but may not impose changes to the conclusions.

The United Nations Evaluation Group (UNEG) norms and standards for evaluation and the ALNAP principles insist on ethics: do no harm to the people interviewed, informed consent, confidentiality, attention to people in distress and to risks of retaliation, feedback to communities.

The evaluation manager has an active role: preparing data, organising field access, convening the reference group, checking quality at key stages, protecting the team's independence, and organising the debriefing. The main risks are well known: start-up delays that shorten fieldwork, non-existent baseline data, limited access, pressure to soften findings.

An evaluation that finds only positive points is not credible. A programme that accepts well-supported negative findings shows that it is serious.

6Management response and use

The management response is the document that turns an evaluation into decisions. For each recommendation: accepted, partially accepted or rejected, with the justification; planned actions; person responsible; deadline. It is written within a month of the final report and published with it.

Follow-up of actions takes place at quarterly management reviews: where does each action stand? Mature organisations publish an annual implementation rate for accepted recommendations.

A recommendation can legitimately be rejected: too costly, outside the mandate, already under way. But the justification must be written down. What is not acceptable is the absence of a response.

Finally, dissemination adapts the format to the audience: a two-page summary for management and donors, a presentation for teams, a simple oral version for communities, and publication of the full report (for example in the ALNAP library) for the sector. Publishing evaluations, including critical ones, is a Grand Bargain commitment on transparency.