Managing a final evaluation: terms of reference, impact analysis and management response

Weaknesses: (1) the purpose "to demonstrate the positive impact" prejudges the result: an evaluation serves learning and accountability, not proof; (2) the questions are too many, closed and unrelated to the criteria; (3) a survey limited to beneficiaries allows no attribution, even though comparison data exist; (4) July is in the middle of the lean season and the rainy season: the results would not be comparable with the November surveys and access would be difficult; (5) major conflict of interest: the former project manager would be evaluating his own work; (6) report in English, whereas the authorities, partners and communities read French or Arabic; (7) no ethical provisions (consent, confidentiality, data protection, what to do if abuse is reported); (8) no feedback to communities and no management response planned.

Realistic budget: fees 25 × 450 = USD 11,250; data collection 800 / 8 = 100 enumerator-days × 20 = USD 2,000; training 12 enumerators × 3 days × 20 = USD 720; vehicles 2 × 12 days × 120 = USD 2,880. Total of about USD 16,850 before data entry, supervision and feedback, i.e. nearly double the USD 9,000 planned. Either the budget must be increased (a final evaluation commonly represents a few per cent of a project's budget), or the sample must be reduced, justified by a power calculation, while keeping the comparison group.

Reworded purpose: "To assess independently the relevance, effectiveness, efficiency, impact and sustainability of the project, in particular for female-headed households, in order to draw lessons for phase 2 and to account to communities, authorities and the donor."

Questions: (relevance) to what extent did the chosen value chains respond to the priorities and constraints of refugee and host households, including women? (coherence) was the project aligned with the other livelihood interventions in the area? (effectiveness) to what extent were the expected results in income and dietary diversity achieved, for which groups, and why? (efficiency) what was the cost per household whose income increased, compared with other modalities? (impact) what share of the observed changes can be attributed to the project, and what unintended effects were there? (sustainability) which activities continue without support, and under what conditions?

Difference-in-differences = (endline − baseline) intervention − (endline − baseline) comparison.

Interpretation: a before-after comparison would attribute to the project twice the real effect on dietary diversity (+1.8 instead of +0.9) and more than twice on income (+13,000 instead of +5,500). The effect on female heads of household is almost nil: the project did not reach this group, which supports recommendation R1.

Validity: (1) the parallel trends assumption cannot be tested with two measurement points; the baseline levels are close (3.8 and 4.1; 28,000 and 30,000), which is reassuring but not conclusive; (2) the seed distribution in 5 of the 9 comparison villages probably improves their results: the project's effect is therefore probably underestimated; an analysis separating the 4 uncontaminated villages would be useful, with caution about sample size; (3) both surveys in November, after the harvest, make the measurements comparable with each other, but say nothing about the lean season; (4) significance should be checked (confidence intervals, regression with cluster effect) before any strong claim, and sample attrition should be checked to be low.

Wording: "The project is associated with an improvement in the dietary diversity score of about 0.9 point and in income of about XAF 5,500 per month beyond the change in the comparison villages; this estimate is probably conservative because some of those villages received assistance. No effect is observed for female-headed households."

Team comments: they are defensive and partly wrong. A 60% participation of women does not prove inclusion: the difference-in-differences shows almost no effect for female-headed households. The price drop cannot be controlled, but the risk was foreseeable and manageable. The donor's supposed preference does not justify a dispersion that reduces effectiveness.

The management response is published with the report and its follow-up is reviewed every quarter.

I point out that the evaluator is independent: neither the project manager nor I can ask him to remove a finding. The project manager may, however, make factual comments on the draft report; the evaluator decides whether or not to include them and documents his choices. If sample size is a real limitation, the right response is to state it (confidence interval, note of caution), not to delete the finding. I refuse to hold back the report: the donor is expecting it, and hiding it before a funding request would be misleading. Instead, I propose sending the report with the management response, which shows how phase 2 corrects the weakness: that is an argument in favour of the proposal. I inform the country director of the request and, if it is maintained, I escalate it in line with the organisation's evaluation policy.

"(1) The project had a measurable positive effect, beyond the change in the comparison villages: +0.9 point of dietary diversity, +XAF 5,500 of monthly income (about +20%) and 9 points fewer households using crisis strategies. (2) This estimate is probably conservative, as some of the comparison villages received other assistance. (3) Female-headed households benefited very little: phase 2 will adapt the package to their constraints. (4) Dispersion across 8 value chains and the lack of price monitoring reduced the effect (30% drop in the price of groundnuts). (5) Phase 2 will concentrate support on 4 or 5 value chains, with market analysis and participatory targeting."