M&E Fundamentals — Module 4
M&E Data Sources: Finding, Evaluating & Collecting the Right Evidence
Mediacrest College — Department of Monitoring, Evaluation & Learning
July 07, 2026
🎯 Module 4 Overview
Module: M&E Fundamentals — Module 4 Focus: Identifying, evaluating, and designing the evidence pipeline that feeds your indicators — from source selection to data collection planning. Target: Entry-Level M&E Officers, Data Managers & Program Coordinators Delivery: Facilitator-led instructional tabs with applied multi-sector activities
By the end of this module, you will be able to:
- Define what a data source is and explain why source selection is as important as indicator selection.
- Distinguish between primary and secondary data sources, and routine and nonroutine sources.
- Evaluate a data source against key quality criteria: validity, reliability, timeliness, completeness, and accessibility.
- Select the right data collection method for a given indicator and context.
- Design a basic data collection tool and avoid the most common tool-design mistakes.
- Build a complete Data Collection Plan mapping each indicator to its source, method, frequency, and responsible person.
📚 Module Sessions
📋 Recap: Module 3
Before we move into data sources, let’s consolidate the key ideas from Module 3: M&E Indicators — because you cannot design a good data source without first knowing precisely what you are trying to measure.
Six Things to Carry Forward from Module 3
1. The Golden Rule of Indicators An indicator is a neutral variable, not a target. It must be able to move in any direction. “Number of clients counseled” is a valid indicator. “Increased number of clients counseled” is not — it bakes a direction into the name and fails the moment the number drops.
2. The Results Chain Every indicator belongs to a level of your results chain. Process indicators track Inputs and Activities. Output indicators track immediate, controlled deliverables. Evaluation indicators track Outcomes and Impact — where external factors increasingly interfere.
3. The Anatomy of a Metric For any percentage indicator, the numerator and denominator must align perfectly in population, time window, and geographic boundary. A mismatch produces a number that looks precise but means nothing.
4. The Seven Characteristics Strong indicators are Objective, Direct, Practical, Adequate, Useful for Management, Attributable, and properly Disaggregated. These criteria are often in tension — the most direct indicator is not always the most practical one to collect.
5. Target Setting Targets must be data-driven, not aspirational guesses. Floor targets are contractual minimums. Stretch targets are internal motivators and should never be reported to donors as firm commitments.
6. Common Pitfalls Avoid indicators you cannot influence, denominators that misrepresent your population, and indicator drift — when a definition silently changes mid-program, making trend data incomparable across periods.
The Bridge Into Module 4
You now know what to measure. Module 4 answers the next logical question: where does the number actually come from?
A well-designed indicator with a poorly chosen data source is like a precise recipe using unreliable ingredients — the formula is sound, but the output cannot be trusted. Every indicator you defined in Module 3 needs a specific, credible, accessible source behind it. That source selection is not a technical afterthought; it determines your program’s cost, ethical obligations, and the trustworthiness of every number you report. This is what Module 4 is about.
🗂️ 1. Understanding Data Sources
What Is a Data Source?
A data source is any system, person, document, or process from which you obtain the information needed to calculate your indicators. Choosing the wrong source — even for a perfectly designed indicator — produces numbers that are misleading, untimely, or impossible to verify.
💡 Expert note — Source selection is an ethical decision, not just a logistical one: Standard global guidance (USAID ADS 201, Global Fund M&E frameworks) requires programs to consider who bears the burden of data collection before selecting a source. A source that relies on already-overstretched frontline workers to fill in extra forms creates reporting fatigue, degrades data quality over time, and shifts a program cost onto the people it is meant to serve.
Two Ways to Classify Data Sources
M&E practice uses two overlapping classification systems. Understanding both helps you make better source decisions.
Classification 1: Primary vs. Secondary
- Primary sources are data your program collects directly — surveys, observation checklists, interviews, focus groups, and routine service registers that your team designs and controls.
- Secondary sources are data collected by someone else for their own purposes — national demographic surveys, government administrative records, published research, and census data.
💡 Expert note — Secondary sources save money but carry hidden risks: A national population survey may be free to access, but if it was conducted three years ago, it may no longer reflect current realities in your target area. Always check the reference year and geographic granularity of any secondary source before building it into your indicator plan.
Classification 2: Routine vs. Nonroutine
- Routine sources generate data continuously as part of normal program operations — daily tally sheets, weekly registers, monthly summary forms. They are the backbone of most monitoring systems.
- Nonroutine sources generate data at specific points in time for a specific purpose — a baseline survey, a mid-term evaluation, or a post-project focus group.
🌡️ The Weather Station Analogy
A weather station that records temperature every hour is a routine source — it runs continuously, captures trends over time, and costs nothing extra per reading once it is set up. A meteorologist who flies into a region to conduct a special atmospheric study is a nonroutine source — expensive, time-limited, and producing deep data that the weather station cannot.
Your M&E system needs both. Routine sources tell you what is happening week to week. Nonroutine sources tell you why it is happening and whether the big-picture change you hoped for is actually occurring.
The Four-Quadrant Source Map
✅ 2. Assessing Data Quality
Even the most perfectly designed indicator is only as trustworthy as the quality of the data feeding it. Before committing to a source, M&E practitioners evaluate it against a standard set of quality dimensions.
The Seven Quality Dimensions
Are We Measuring the Right Thing?
The source must actually capture what the indicator requires — not a close approximation.
Same Result, Same Conditions?
If two data collectors use the same tool on the same day at the same site, they should get the same number.
Are There Gaps?
Missing records, blank fields, or facilities that never submitted reports all create holes in the data picture.
Available When Needed?
Data that arrives after the decision has already been made is useless for management, no matter how accurate it is.
Can You Actually Get It?
Some sources exist but are locked behind bureaucratic processes, cost barriers, or data-sharing restrictions.
Does It Reach Everyone?
A source that only captures data from urban facilities misrepresents a program serving both urban and rural populations.
Is the Collection Process Neutral?
Sources where the person being measured also records the measurement are particularly vulnerable to bias.
💡 Expert note — Validity vs. Reliability are not the same thing, and both can fail independently: A bathroom scale that consistently reads 5kg too heavy is reliable (same result every time) but not valid (not measuring your true weight). A sleepy data clerk who records different numbers on the same form each time is neither reliable nor valid. You need both. In M&E, validity asks “does this source actually measure our indicator concept?” while reliability asks “does it produce consistent results across collectors, sites, and time?”
Three Types of Bias to Know
1. Sampling Bias — The sample taken to represent the population is not actually representative. Example: surveying only market vendors on a Tuesday morning systematically excludes farmers who are in the field on weekdays.
2. Nonsampling Bias — Errors that occur during data collection regardless of who is selected. Example: a questionnaire with leading questions (“Don’t you agree that the training was useful?”) pushes respondents toward positive answers.
3. Subjective Measurement Bias — When the person measuring is also the person being measured, or has a stake in the result. Example: asking project staff to self-report whether they followed the standard operating procedure creates a systematic tendency to over-report compliance.
⚖️ The Exam Marking Analogy
Imagine a teacher marking their own students’ exams, knowing that low scores reflect badly on their own teaching. Even with the best intentions, the marking is vulnerable to generous interpretation of borderline answers. This is subjective measurement bias — not dishonesty, just a structural conflict of interest built into who is doing the measuring.
The solution in M&E is the same as in education: wherever possible, separate the person who collects or enters data from the person or program whose performance that data reflects.
| Quality Dimension | What It Means | Common Red Flag |
|---|---|---|
| Validity | Source captures what the indicator actually requires. | Proxy measure used because true measure is too expensive. |
| Reliability | Repeated measurement under same conditions gives same result. | Different collectors get different numbers from the same records. |
| Completeness | No significant gaps, missing records, or unreported units. | Facilities or participants systematically missing from records. |
| Timeliness | Data arrives in time to inform the relevant decision. | National survey published every 5 years used for annual reporting. |
| Accessibility | Source can be obtained within budget and without legal barriers. | Government database requires ministerial approval to access. |
| Coverage | Source represents all subgroups in the target population. | Only urban sites reporting; rural sites excluded. |
| Freedom from Bias | Collection process is not distorted by the collector’s interest in the result. | Program staff self-report their own compliance with protocols. |
🛠️ 3. Data Collection Methods
The method is how you extract information from a source. The source tells you where the data lives; the method tells you how you get it out. Choosing the wrong method for your indicator is as costly as choosing the wrong source.
The Seven Core Methods
| Method | Data Type | Relative Cost | Expertise Needed | Best Used For |
|---|---|---|---|---|
| Structured Survey | Quantitative | Medium | Medium | Measuring prevalence, coverage, or percentages across a population. |
| Focus Group Discussion | Qualitative | Medium | High | Exploring why a behavior or outcome occurred; generating hypotheses. |
| Key Informant Interview | Qualitative | Low–Medium | Medium | Getting detailed insight from one person with specific knowledge. |
| Observation Checklist | Quantitative / Qualitative | Low | Low | Verifying whether a process or standard was actually followed. |
| Document / Record Review | Quantitative | Low | Low | Extracting data from existing registers, files, or administrative records. |
| Lab / Physical Measurement | Quantitative | High | High | Obtaining objective biological or physical measurements. |
| Community Score Card | Qualitative / Quantitative | Medium | Medium | Getting community feedback on service quality in a structured format. |
💡 Expert note — Qualitative methods cannot produce percentages: A focus group can tell you why adoption rates are low. It cannot tell you how many people have adopted. Students frequently confuse these. The rule is simple: if your indicator requires a number or percentage, you need a quantitative method. If your indicator requires an explanation, pattern, or community perspective, qualitative methods are appropriate — often as a companion to quantitative data, not a replacement for it.
🎙️ The Reporter’s Toolkit Analogy
A journalist covering a factory fire uses different tools for different questions. A count of injuries comes from the hospital’s admission log — a record review. A survivor’s account of what caused the fire comes from an interview. A photo of the safety equipment that was missing comes from direct observation. The question determines the tool; a journalist who only ever uses one method will always miss part of the story.
M&E works the same way. A project that only runs surveys will miss the why. A project that only runs focus groups will never be able to report a defensible percentage. Match the method to the question, not to whichever method is most familiar.
When to Use Qualitative vs. Quantitative
| Criterion | Quantitative Methods | Qualitative Methods |
|---|---|---|
| Produces a number/percentage | ✓ | ✗ |
| Explains why something happened | ✗ | ✓ |
| Representative of a population | ✓ | Partially |
| Captures community voice | Partially | ✓ |
| Suitable for logframe reporting | ✓ | Partially |
| Generates hypotheses for further study | Partially | ✓ |
📝 4. Designing Data Collection Tools
A data collection tool is the physical or digital instrument your team uses to capture information — a paper form, a mobile data entry screen, an interview guide, or an observation checklist. A badly designed tool is one of the most reliable ways to ruin otherwise good data.
Seven Principles of Good Tool Design
- Group related indicators together. Don’t jump between topics randomly — organize the form by subject area so the collector moves logically through the interview or record.
- Collect only what you will use. Every field on a form is a burden. If no indicator in your M&E plan uses the data from a field, remove it.
- Capture metadata on every tool. Date of collection, name of collector, site or location, and tool version number. Without metadata, a completed form cannot be traced, verified, or corrected.
- Use standardized codes, not free text, wherever possible. Free text (“male,” “Male,” “M,” “man”) creates cleaning headaches. Codes (“1 = Male, 2 = Female”) eliminate ambiguity.
- Pre-test before deployment. Run the tool with five to ten real respondents before scale-up. Pre-testing consistently reveals questions that respondents misinterpret and fields that collectors skip.
- Train every person who will use the tool. A well-designed tool operated by an untrained collector produces poor data.
- Build in an ethics and consent step. Before any primary data collection begins, the respondent must understand what data is being collected, why, and how it will be stored and used.
⚠️ Gap from standard practice — Data privacy and ethics are not optional: Your program’s raw notes did not include a data ethics section. All major donors — Global Fund, USAID, UNDP — and most national data protection laws require informed consent, data anonymization wherever possible, and secure storage protocols. Skipping this step is not a design oversight; it is a compliance failure. Build a consent step into every primary data collection tool before anything else.
💡 Expert note — “Tool bloat” is the most common tool design failure: Teams routinely add fields to a form because the data might be useful someday, or because a donor asked a question in passing, without checking whether any indicator in the M&E plan actually requires it. The result is a ten-page form where three pages of data are never entered into any database. Every extra field is borrowed time from the person filling it in — time that comes at the cost of data quality on the fields that actually matter.
Metadata Fields: The Minimum Requirement
| Metadata Field | Purpose | Consequence if Missing |
|---|---|---|
| Collection Date | Enables timeliness checks and links the record to the correct reporting period. | Cannot assign record to correct period; may be double-counted. |
| Data Collector Name / ID | Enables traceability — if a data quality issue is found, the collector can be identified and retrained. | Cannot investigate or correct data quality issues. |
| Site / Location Code | Enables geographic disaggregation and cross-site comparisons. | Cannot disaggregate by geography. |
| Tool Version Number | Identifies which version of the tool was used, critical if the tool was revised mid-program. | Cannot identify whether old and new data are comparable. |
| Supervisor Review Signature | Confirms that a supervisor checked the form before it entered the system. | No quality control checkpoint before data entry. |
📊 5. The Data Collection Plan
A Data Collection Plan (DCP) is the operational document that ties your entire measurement system together. For every indicator in your M&E plan, it specifies exactly what data will be collected, from whom, using which method, how often, and who is responsible.
The five questions every Data Collection Plan must answer for each indicator:
- What is being measured? (The indicator and its precise definition)
- Where does the data come from? (The source)
- How is the data collected? (The method and tool)
- How often is data collected and reported? (Frequency)
- Who is responsible for collection, entry, and verification?
💡 Expert note — “Can we actually do this?” is the most important question teams skip: Before finalizing the DCP, run a feasibility check on every row. Ask: does the responsible person have the time, the access, and the training to collect this data at this frequency? A plan that looks complete on paper but is operationally impossible will produce missing data, fabricated data, or burned-out frontline workers. USAID’s Data Quality Assessment framework explicitly requires this feasibility review before any indicator is considered “reportable.”
Sample Data Collection Plan
| Indicator | Data Source | Collection Method | Frequency | Responsible Person | Stored In |
|---|---|---|---|---|---|
| % of farmers who adopted improved seed varieties | Farmer household survey (primary, nonroutine) | Structured household survey | Annual (endline) | M&E Officer | Survey database |
| # of community workshops conducted | Workshop attendance register (primary, routine) | Document / record review | Monthly | Field Coordinator | Programme register file |
| % of participants satisfied with training quality | Post-training feedback form (primary, nonroutine) | Structured survey (Likert scale) | Per workshop | Facilitator / M&E Assistant | Feedback form database |
Participant Tracking and Coding
When your program tracks individual participants over time, using identity codes rather than names protects privacy while maintaining traceability.
| Participant Code | Site | Sex | Age | Enrolment Date | Visit 1 | Visit 2 | Visit 3 |
|---|---|---|---|---|---|---|---|
| KLF-001-F-28 | KLF | F | 28 | 2025-01-10 | ✓ | ✓ | ✓ |
| KLF-002-M-34 | KLF | M | 34 | 2025-01-10 | ✓ | ✗ | ✓ |
| KLF-003-F-22 | KLF | F | 22 | 2025-01-11 | ✓ | ✓ | ✗ |
💼 6. Practical Group Activities
Activity 1: The Bias Trap (Enterprise Logistics)
A commercial enterprise selling Mwea Pishori rice at wholesale and retail levels relies entirely on its delivery drivers to self-report “customer satisfaction” via an observation checklist they fill in after each delivery. Over three months, satisfaction scores average 94%. The manager is delighted — until customer complaints begin rising sharply in the same period.
Task: Identify the type of bias present in this data source. Explain specifically why asking the delivery driver to self-report on an outcome they are being evaluated on produces structurally distorted data. Propose two alternative routine data sources that would measure satisfaction more objectively.
Debrief: Which of the seven quality dimensions does the original source fail? Which does your proposed alternative satisfy?
Activity 2: Method Matching (Agriculture / Livestock)
A cross-breeding project wants to understand whether farmers feel confident raising pedigree Dorper sheep after a six-month technical support program. The M&E team has a limited budget, no vehicle for field travel, and needs information rich enough to redesign the training if confidence is low. Farmers work long hours and are rarely available for extended sessions.
Task: From the seven methods in Section 3, select the most appropriate method for this context. Justify your choice against the constraints given. Explain why a structured survey alone would be insufficient for this particular research question.
Debrief: Is this question calling for a quantitative indicator or a qualitative one? How does your method choice reflect that?
Activity 3: Tool Bloat Audit (Public Health)
Your facilitator will provide your group with a deliberately over-designed clinic tracking form containing 38 fields. The program’s M&E plan has six indicators, none of which require demographic fields beyond age group and sex.
Task: Audit the form as a group. Mark each field as: Keep (required by an indicator), Remove (no indicator uses this data), or Add (a required metadata field that is missing). Redesign the form to contain only essential fields, add the minimum metadata requirements from Section 4, and replace any free-text fields with standardized codes.
Debrief: How many fields did you remove? What is the likely impact on data quality and collector burden of the original 38-field version compared to your redesigned version?
📝 7. Knowledge Review
Module 4 Mastery Assessment
Question 1 A project uses a national demographic survey published every five years to measure its annual program impact. What is the primary data quality issue?
- Validity — the survey does not measure the right concept.
- Timeliness — the frequency of the source does not match the annual reporting need. ✓
- Subjective measurement bias.
- Coverage — the survey does not reach the target population.
Question 2 Which of the following is an example of a routine data source?
- A mid-term facility assessment survey.
- A national population census.
- Daily participant attendance and tracking registers. ✓
- A post-project community focus group.
Question 3 True or False: A focus group discussion can be used to calculate the exact percentage of a community that adopted a new farming practice.
- True — focus groups give representative community data.
- False — focus groups produce qualitative insight, not statistically representative percentages. ✓
Question 4 What is the purpose of capturing metadata on a data collection tool?
- To make the form longer and more comprehensive.
- To record who collected the data, when, and where — enabling traceability and quality review. ✓
- To replace the need for a data collection plan.
- To anonymize participant information.
Question 5 A reliable data source is one that:
- Measures exactly the right concept for the indicator.
- Is free to access without bureaucratic barriers.
- Produces consistent results when used by different collectors under the same conditions. ✓
- Covers 100% of the target population without gaps.
🗝️ Answer Key & Trainer Rationales
| Question | Correct Choice | Technical Trainer Rationale |
|---|---|---|
| 1 | B | The survey may be perfectly valid and reliable — it is measuring the right thing accurately. The problem is frequency: a five-year publication cycle cannot serve annual reporting needs. This is a textbook timeliness failure. |
| 2 | C | Routine sources are those that generate data continuously as part of normal operations. Daily registers are filled in as part of the program’s standard workflow — not commissioned for a specific study purpose. Options A, B, and D are all time-limited, purpose-specific events. |
| 3 | B | Focus groups are purposefully small and non-representative — participants are selected to generate depth, not breadth. Calculating a percentage from a focus group of 8 people and applying it to a community of 800 is a methodological error. Surveys with probability sampling are the appropriate quantitative tool. |
| 4 | B | Metadata creates a chain of custody for every data record. If a data quality problem is found during a DQA, metadata tells you which collector, which site, and which time period to investigate. Without it, quality issues cannot be traced or corrected. |
| 5 | C | Reliability is about consistency, not accuracy of concept (that is validity) or access (that is accessibility) or coverage. A source is reliable if two different collectors using the same tool on the same records arrive at the same result. |
📚 References & Resources
| Resource | Relevance to Module 4 | Access |
|---|---|---|
| USAID — ADS Chapter 201: Program Cycle Operational Policy | Standard for Data Quality Assessments and feasibility reviews before indicator reporting. | usaid.gov |
| UNDP — Handbook on Planning, Monitoring & Evaluating for Development Results | Comprehensive guidance on Results-Based Management including data source selection. | undp.org |
| MEASURE Evaluation — Routine Health Information Systems (RHIS) Curriculum | Practical curriculum for designing and managing routine data collection systems. | measureevaluation.org |
| The Global Fund — M&E Framework and Data Quality Assurance Guidelines | Data quality assurance requirements for grant-funded programs. | theglobalfund.org |
| BetterEvaluation — Data Collection Methods Rainbow Framework | Comparative guide to selecting appropriate data collection methods by indicator type. | betterevaluation.org |
| WHO — Data Quality Review: A Toolkit for Facility Data Quality Assessment | Practical toolkit for assessing completeness, timeliness, and accuracy of facility-level data. | who.int |