Measuring impact: counterfactual, methods and contribution

1The fundamental problem: what would have happened without the programme?

The impact of a programme is the difference between what happened with the programme and what would have happened without it: the counterfactual. It is never possible to observe the same household directly both with and without the programme; all impact evaluation methods are ways of building a credible estimate of this counterfactual.

Two naive comparisons are misleading:

• Before-after among beneficiaries: the observed change mixes the effect of the programme with everything else that changed at the same time (harvest, prices, security, other aid). If the harvest was good, food security improves even without the programme; if it was poor, an effective programme can appear to have no effect.
• Beneficiaries versus non-beneficiaries after the programme: the two groups were different at the outset, precisely because targeting selected the most vulnerable. The comparison then underestimates the effect, or overestimates it if it was the most motivated who signed up (selection bias).

The question to ask of any impact claim: "compared with what?". If the answer is "compared with before", it is not a measure of impact.

2The randomised controlled trial and its use in humanitarian settings

In a randomised controlled trial (RCT), among the eligible people, those who receive the programme (treatment group) and those who do not receive it or receive it later (control group) are selected by lottery. Random assignment makes the two groups comparable on average on all characteristics, observed or not. The effect is simply the difference in average outcomes at the end.

In humanitarian settings, denying life-saving aid to eligible people in order to form a control group is unethical. But RCTs remain possible and have produced major knowledge (on cash, nutrition, mental health) in specific situations:

• Insufficient resources for everyone in any case: random selection is then a fair way of allocating, and the group not served acts as the control.
• Phased roll-out: everyone will receive the programme, but the order is drawn by lot (stepped wedge design).
• Comparison of two variants of a programme, both considered acceptable: monthly or lump-sum cash, with or without accompanying support.

RCTs are expensive, take time and answer specific questions; they are justified when the answer will serve many programmes. For most projects, quasi-experimental methods are sufficient.

3Difference-in-differences

Difference-in-differences compares the change in an assisted group with the change in a non-assisted comparison group: effect = (after − before) in the assisted group − (after − before) in the comparison group. It eliminates stable initial differences between the groups and common trends (harvest, prices).

Example. Proportion of households with acceptable food consumption. Assisted group: 28% before, 52% after (+24 points). Comparison group: 35% before, 44% after (+9 points). Estimated effect: 24 − 9 = 15 points. A before-after comparison would have attributed 24 points to the programme; a comparison of final results alone would have attributed 8.

Validity condition: the parallel trends assumption. Without the programme, the two groups would have changed in the same way. It is checked where possible with several measurements before the programme, and one makes sure that the comparison group does not receive other significant aid during the period (contamination) and does not suffer a specific shock.

In humanitarian settings, the comparison group is often a neighbouring area not covered for lack of funding, or eligible households on a waiting list. A comparison group is never created by denying aid for the purposes of the study.

4Matching and regression discontinuity

Matching builds a comparison group by pairing each assisted household with one or more non-assisted households that resemble it on characteristics measured before the programme (size, assets, income, status, area). The propensity score method summarises these characteristics as a probability of being assisted, and households with close scores are compared. Its limitation: it corrects only observed differences. If motivation or social networks influenced registration and are not measured, the bias remains.

Regression discontinuity applies when eligibility depends on a threshold on a score: for example, households with a vulnerability score of 60 or more are assisted, those below are not. Households just above and just below the threshold (57-59 and 60-62) are almost identical; the difference in outcomes between them estimates the effect of the programme around the threshold. Conditions: the threshold is strictly applied, and households cannot manipulate their score to get above it.

Example. Households with a score of 57-59: 38% with acceptable consumption. Households with a score of 60-62 (assisted): 51%. Estimated effect around the threshold: about 13 points. This result holds for households close to the threshold, not necessarily for the most vulnerable.

The interrupted time series compares the trend of an indicator before and after the start of a programme, over many measurement points (for example weekly nutrition admissions). It is useful when no comparison group exists, provided there are enough points before the intervention.

5Contribution approaches: when attribution is impossible

Many important effects do not lend themselves to statistical methods: advocacy that changes a law, capacity strengthening of local organisations, improved coordination. They cannot be compared with a quantified counterfactual. The aim is then to establish credibly that the programme contributed to the change.

• Contribution analysis (Mayne): set out the theory of change, gather evidence for each link, examine alternative explanations, and build a "contribution story" that is then critiqued by stakeholders.
• Process tracing: causal hypotheses are tested by looking for evidence whose presence or absence makes it possible to confirm or rule out an explanation (for example, the word-for-word reuse of an advocacy note in a decree).
• Outcome harvesting: first identify the changes observed among actors, then work backwards to determine whether and how the programme contributed to them.
• Most significant change: stories of change are collected from people, and panels select the most significant ones, explaining why. Useful for discovering unintended effects.

Rigour does not mean quantitative. A well-conducted contribution analysis that seriously examines alternative explanations is more credible than a quantified before-after comparison.

6Choosing the method and deciding what needs proving

The choice of method depends on four questions:

• What decision? Scaling up a pilot model nationally justifies strong evidence; adjusting an ongoing programme can make do with sufficient evidence.
• Does the evidence already exist? The effect of cash on food consumption is very well documented: there is no need to demonstrate it again with an RCT in every project. It is often enough to check that the programme is faithful to the model and that the intermediate effects are present.
• What is feasible and ethical? Is there a comparison group without denying aid? A clear eligibility threshold? A phased roll-out?
• What budget and what timeframe? A serious impact evaluation requires a baseline before the start: it is designed with the programme, not at the end.

As regards sample size, an impact evaluation must have sufficient power to detect the expected effect. Detecting a 15-point effect requires far fewer households than a 5-point effect. If the expected effect is small and the sample is small, the study will almost certainly conclude "no significant effect", which will be misinterpreted as "no effect".

A final point: present the level of evidence honestly. "Assisted households improved their consumption by 24 points; compared with a comparison area, the attributable effect is estimated at 15 points, under the parallel trends assumption supported by data from two previous rounds" is a sentence that inspires confidence.