AI module · complements the 12 weeks · about 4 hours

Monitoring, evaluation and MEAL with artificial intelligence

Mini MBA Programmes and MEAL

Module objectives
  • Use AI to speed up qualitative and quantitative analysis while keeping results verifiable.
  • Measure the reliability of AI-assisted coding through double coding of a sample.
  • Protect the data of respondents and complainants at every step.

This module complements the 12 weeks of the programme. It adds no exam: certificates and the diploma are earned as before. Tools are described as known in November 2026; they change quickly.

1Where AI helps the MEAL team

In the MEAL cycle, generative AI is useful for repetitive text and structuring tasks:

  • Qualitative analysis: proposing a first codebook from 50 open-ended answers, classifying verbatims, summarising focus group discussions.
  • Data collection tools: reviewing a questionnaire (double-barrelled, leading or overly long questions), suggesting answer options, helping write the logic of a KoboToolbox or ODK form.
  • Quantitative analysis: writing Excel formulas, R or Python scripts, explaining a statistical test.
  • Reporting: a first draft of a report, a summary sheet, key messages for communities.

It is not reliable for producing figures: asking "what is the satisfaction rate?" of an assistant that has read a file can give a wrong result. Have it calculated by a formula or script that you can read and rerun.

2AI-assisted qualitative coding: the method

  1. Anonymise the answers before sending anything: remove names, numbers, precise locations and identifying details.
  2. Build the codebook with AI on a sample, then revise it as a team: clear definitions, examples, an "other" code.
  3. Code the full set with AI, giving it the codebook.
  4. Verify: an analyst codes a random sample by hand (for example 10%, at least 50 answers) and the agreement rate is calculated. Below about 80%, revise the definitions and recode.
  5. Document in the report: tool used, size of the verified sample, agreement rate, limitations.

Be careful with languages: a model often understands answers in local languages or phonetic transcriptions less well; check those answers more closely.

3Quantitative data: cleaning and calculating without inventing

Good practice:

  • Ask the AI for the code or formula, not the result. Example: "Write an Excel formula that counts households with a food consumption score below 21".
  • Test the code on a small sample whose answer you know.
  • Keep a record: commented script, data version, cleaning steps (duplicates, outliers, inconsistencies).
  • Never send the raw dataset with identifiers to a consumer tool; work on an anonymised dataset or an approved professional account.

For questionnaire translation, AI can produce a first draft, but back-translation and testing with a few respondents remain essential: one badly chosen word can change the meaning of an indicator.

4Complaints, feedback and sensitive data

Feedback and complaints mechanisms sometimes receive highly sensitive messages: sexual exploitation, abuse and harassment (PSEA), fraud, threats. Rules:

  • PSEA allegations, serious fraud or threats are handled by trained people, following the confidential procedure, and never pasted into an AI tool.
  • For routine feedback (distribution times, quality of an item), assisted classification can help, on a professional account, after anonymisation, and with human review of the "sensitive" categories.
  • An automated system must never make a complaint "disappear": every message receives an acknowledgement and follow-up.

Remember: people who complain often take a risk. Trust in the mechanism depends on confidentiality.

5Reports, dashboards and evaluation ethics

AI can propose the structure of a report, rephrase for a non-technical audience, or turn findings into key messages. But:

  • Every figure in the report must come from your analysis, with its source.
  • Participant quotes must be real, never generated or "improved".
  • An evaluation remains independent: do not let the tool soften negative findings.
  • State the use of AI and its limitations in the methodology, as you would for statistical software.

Finally, share results with communities in their language: AI can help simplify a message, but a team member who speaks the language validates the final version.

Exercise: checking AI-assisted coding

Situation. A post-distribution survey collected 640 open-ended answers. AI coded them using an 8-theme codebook. You recode a random sample of 64 answers by hand: you agree with the AI on 50 of them. Of the 14 disagreements, 9 concern answers in a local language.

Questions. 1) What is the sample size as a percentage? 2) What is the agreement rate? 3) What do you do?

Answer key. 1) 64 / 640 = 10%. 2) 50 / 64 ≈ 78%, below the threshold of about 80%. 3) Revise the definitions of the codes concerned, have the local-language answers coded by an analyst who speaks the language (9 of the 14 disagreements, about 64%), recode and then check a new sample; document the approach and the final rate in the report.

Key points

  • AI helps code, structure and draft; figures come from formulas or scripts that can be read again.
  • Anonymise before sending anything; never a raw dataset with identifiers in a consumer tool.
  • Check coding by double coding a sample and aim for at least about 80% agreement.
  • Sensitive complaints (PSEA, fraud, threats) are handled by trained people, never by AI.
  • Real quotes, findings not softened, use of AI stated in the methodology.

Self-assessment quiz

6 self-assessment questions, corrected immediately. They do not count towards the exams.

1To get a reliable satisfaction rate from a dataset, it is better to:
2Verification sample of 64 answers, 50 agreements. Agreement rate:
3An allegation of sexual exploitation arrives through the complaints line. What do you do?
4After AI translation of a questionnaire, the essential step is:
5Can you ask AI to make participants' quotes "more striking"?
6Which answers need closer checking after assisted coding?
Going further

The Mini MBA AI and Humanitarian Action (Claude and Gemini) covers these topics in depth over 12 weeks.

See the AI Mini MBA