Project Portfolio: Scalable M&E System for NGO Impact

GitHub

  |  

LinkedIn

  |  

Email

⚠️ Disclaimer

This project is for portfolio and learning purposes only. All datasets used are simulated and do not represent real beneficiaries or sensitive program data.

1. Project Overview

Small NGOs and community-based organizations (CBOs) often struggle with fragmented data systems, manual Excel reporting, and limited budgets.

This project demonstrates how a fully automated Monitoring & Evaluation (M&E) system can be built using:

• KoBoToolbox for offline data collection

• KoBo API for automated data retrieval

• R (tidyverse) for data processing

• Shiny for real-time dashboards

Goal: Replace manual reporting with a zero-cost, automated M&E pipeline.

2. Problem Statement

A typical CBO working with adolescent girls and young women (AGYW) faces:

• No budget for commercial M&E systems (e.g., DHIS2 customization, ActivityInfo)

• Heavy reliance on Excel-based reporting

• Delayed donor reporting cycles

• Offline field conditions (poor internet connectivity)

• High risk of data errors and duplication

3. Proposed Solution

Design an end-to-end digital M&E pipeline that automates the full data lifecycle:

Architecture Summary

Layer	Tool	Purpose
Data Collection	KoBoToolbox	Offline mobile data entry
Data Storage	KoBo Server	Centralized dataset + API access
Data Processing	R (tidyverse)	Cleaning, transformation, scoring
Visualization	Shiny Dashboard	Real-time monitoring
Deployment	shinyapps.io	Cloud access for stakeholders

4. Key Features of the System

• Offline-First Data Collection: Field workers collect data even without internet using KoBo forms.

• Automated Data Sync: Data is pulled directly from KoBo via API—no manual downloads.

• Real-Time Indicators: KPIs such as vulnerability scores and intervention coverage update automatically.

• Reproducible Analysis: All transformations are coded in R for transparency and auditability.

5. Data Pipeline Design

Field Data Collection (KoBoToolbox)         
        ↓ 
Cloud Storage (KoBo Server)         
        ↓ 
API Sync (R robotoolbox)         
        ↓ 
Data Cleaning & Scoring (dplyr)         
        ↓ 
Shiny Dashboard (Real-time insights)

6. Technical Implementation

Step 1: Connecting R to KoBoToolbox API

In traditional M&E systems, data is manually downloaded from Excel or emails.

Here, we automate data extraction directly from KoBoToolbox using an API.

To get started, you’ll first need your API key; here’s a step-by-step guide to obtaining it through Kobo Toolbox.

# Loading libraries. Ensure you have them installed
library(robotoolbox)
library(dplyr)
library(gt)

# Initializing secure connection to KoBo server
kobo_setup(
  url = "https://kf.kobotoolbox.org", 
  token = api_token # stored securely (NOT hardcoded in production systems). Replace with yours.
)

# Retrieving list of all available forms (assets)
# This elps identify dataset IDs (UIDs) for different program components
my_forms <- kobo_asset_list()

# Viewing the output in a table format
my_forms %>% 
  gt()

uid	name	asset_type	owner_username	date_created	date_modified	deployed	submissions
aKboNAnbeZsCN5nUjLBmNw	Binti Imara Screening Demo Form	survey	sufuri_org	2026-06-15 13:37:55	2026-06-15 13:55:15	TRUE	11
aDiwXzZfmtKdrybADM9YVX	Binti Imara Interventions Demo Form	survey	sufuri_org	2026-06-15 12:53:13	2026-06-15 13:10:18	TRUE	12
aAykYKBdBLzygcKjgHSgvY	Binti Imara Registry Demo Form	survey	sufuri_org	2026-06-15 09:56:55	2026-06-15 12:17:00	TRUE	7

Step 2: Selecting Data Sources (forms)

In real M&E systems, data is split across multiple program areas:
- Registry (beneficiaries)
- Screening (vulnerability assessment)
- Interventions (services delivered)

Each dataset is stored as a separate KoBo “asset and contains metadata about the form.”

# Selecting specific asset using its UID
registry_asset <- kobo_asset("aAykYKBdBLzygcKjgHSgvY")
interventions_asset <- kobo_asset("aDiwXzZfmtKdrybADM9YVX")
screening_asset <- kobo_asset("aKboNAnbeZsCN5nUjLBmNw")

# Viewing basic metadata (Submissions, creation date, status)
print(registry_asset)

## <robotoolbox asset>  aAykYKBdBLzygcKjgHSgvY 
##   Asset name: Binti Imara Registry Demo Form
##   Asset type: survey
##   Asset owner: sufuri_org
##   Created: 2026-06-15 09:56:55
##   Last modified: 2026-06-15 12:17:00
##   Submissions: 7

print(interventions_asset)

## <robotoolbox asset>  aDiwXzZfmtKdrybADM9YVX 
##   Asset name: Binti Imara Interventions Demo Form
##   Asset type: survey
##   Asset owner: sufuri_org
##   Created: 2026-06-15 12:53:13
##   Last modified: 2026-06-15 13:10:18
##   Submissions: 12

print(screening_asset)

## <robotoolbox asset>  aKboNAnbeZsCN5nUjLBmNw 
##   Asset name: Binti Imara Screening Demo Form
##   Asset type: survey
##   Asset owner: sufuri_org
##   Created: 2026-06-15 13:37:55
##   Last modified: 2026-06-15 13:55:15
##   Submissions: 11

Viewing full questionnaire structure:

# Viewing the actual form schema (questions, types, and labels)
form_structure <- kobo_form(registry_asset)

form_structure 

## # A tibble: 27 × 15
##    name        list_name type  label lang  kuid  xpath appearance required hint 
##    <chr>       <chr>     <chr> <chr> <chr> <chr> <chr> <chr>      <lgl>    <chr>
##  1 start       <NA>      start  <NA> Labe… mNkp… start <NA>       NA       <NA> 
##  2 end         <NA>      end    <NA> Labe… 0dR6… end   <NA>       NA       <NA> 
##  3 intro       <NA>      note  "**I… Labe… dnnd… intro <NA>       NA       <NA> 
##  4 registry_id <NA>      text  "Reg… Labe… RZuu… demo… <NA>       TRUE     <NA> 
##  5 county      <NA>      text  "Cou… Labe… qk89… demo… <NA>       TRUE     <NA> 
##  6 subcounty   <NA>      text  "Sub… Labe… hHRi… demo… <NA>       TRUE     <NA> 
##  7 ward        <NA>      text  "War… Labe… kLJt… demo… <NA>       TRUE     <NA> 
##  8 dob         <NA>      date  "Dat… Labe… VBO5… demo… <NA>       TRUE     This…
##  9 age         <NA>      calc… "Age" Labe… ehY1… demo… <NA>       NA       <NA> 
## 10 age_display <NA>      calc…  <NA> Labe… mZHw… demo… <NA>       NA       <NA> 
## # ℹ 17 more rows
## # ℹ 5 more variables: constraint <chr>, calculation <chr>, relevant <chr>,
## #   version <chr>, choices <list>

This step matters because it extracts full questionnaire structure from KoBoToolbox.

It is equivalent to viewing the “data dictionary” in a real M&E system.

USE CASE:
- Understand variables collected
- Validate form design
- Ensure alignment with program indicators

Step 3: Extracting Raw Data

This is the fun part because it replaces manual CSV/XLSX downloads.

Data is pulled directly from the server ensuring:
- Real-time updates
- No version confusion
- Reduced human error

# Downloading the data into a tidy data frame
registry_data <- kobo_data(registry_asset, all_versions = TRUE)

interventions_data <- kobo_data(interventions_asset, all_versions = TRUE)

screening_data <- kobo_data(screening_asset, all_versions = TRUE)

# Viewing the data frame
head(registry_data, 2) %>% 
  gt()

start	end	**Introduction** Welcome to the Binti Imara data collection tool. This is a simulated tool for a fictitious project, created solely for training, data analysis practice, tool development, and reporting purposes. It does not represent real beneficiaries or actual program activities. The Binti Imara Initiative aims to support adolescent girls and young women by strengthening their skills, confidence, and opportunities through targeted mentorship and capacity‑building. The initiative is inclusive of all girls, including those who are HIV positive, pregnant, or young mothers, ensuring they receive the guidance, protection, and resources needed to thrive. Thank you for using this tool responsibly and in accordance with its intended purpose.	Registry ID	County	Sub-County	Ward	Date of Birth	Age	age_display	Age: ${age_display}	School Level	Occupation	Name of Occupation	Has Birth Certificate	Birth Certificate Number	National ID	Why no ID	HIV Status	Date Linked	Pregnancy Status	Is AGYW the Household Head?	Has Stable Caregiver	Eligibility Criteria	Date Enrolled	Name of Field Worker	Name of Data Clerk	_id	uuid	__version__	instanceID	_xform_id_string	_uuid	rootUuid	_status	_submission_time	_validation_status	_submitted_by	deprecatedID	_attachments
2026-06-15 12:14:24.138	2026-06-15 12:16:12.272	NA	BII00001	Homa Bay	Homa Bay Town	Homa Bay Cebtral	2006-06-15	20	20 Years	NA	DROPPED OUT	STUDENT	NA	NO	NA	NO	NA	NEGATIVE	NA	NOT PREGNANT	NO	YES	OUT OF SCHOOL	2025-04-27	Sam	Doreen	781027340	888452f7f2cf438fb140bc45e1550b3c	vbqXtWy4P4cSuoGNafANyC	uuid:d1830d23-1ac0-4131-9f02-d5fe407ef552	aAykYKBdBLzygcKjgHSgvY	d1830d23-1ac0-4131-9f02-d5fe407ef552	uuid:d1830d23-1ac0-4131-9f02-d5fe407ef552	submitted_via_web	2026-06-15 12:16:14	NA	NA	NA
2026-06-15 11:05:51.907	2026-06-15 12:28:57.397	NA	BII00002	Mombasa	Mvita	Majengo	2009-07-09	16	16 Years	NA	SECONDARY	SELF EMPLOYED	NA	YES	803963/2021	NO	NA	NEGATIVE	NA	NOT PREGNANT	YES	YES	HOUSEHOLD HEAD	2024-07-31	Sam	Doreen	781030616	888452f7f2cf438fb140bc45e1550b3c	vb9XQSm5rxbQPmCGcYPYy2	uuid:865eeb74-d45d-41b9-ba69-944545b4cc47	aAykYKBdBLzygcKjgHSgvY	865eeb74-d45d-41b9-ba69-944545b4cc47	uuid:0354e69a-96e3-46a4-bd01-bfd6db6848c8	submitted_via_web	2026-06-15 12:19:18	NA	NA	uuid:0354e69a-96e3-46a4-bd01-bfd6db6848c8

head(interventions_data, 2) %>% 
  gt()

start	end	**Introduction** This simulated tool collects data on all interventions provided under the fictional Binti Imara Initiative, for training and analysis purposes only. It does not represent real beneficiaries or activities. The initiative supports all adolescent girls and young women, including those who are HIV positive, pregnant, or young mothers. Use responsibly.	Registry ID	Intervention Domain	Intervention Provided	Date of Service	Type of Provider	Referral Made	Follow up Needed	Date of Follow up	Service Status	_id	uuid	__version__	instanceID	school_level	_xform_id_string	_uuid	rootUuid	_status	_submission_time	_validation_status	_submitted_by	_attachments
2026-06-15 12:54:14.278	2026-06-15 12:59:49.755	NA	BII00005	GBV	Legal Referral	2025-12-12	PARALEGAL	YES	YES	2026-01-14	COMPLETED	781077786	83d3f433669241309347dd234afdb5b9	vKSiY7iGN9VP8CxaC88fCn	uuid:19efcb4f-8b45-47b7-ad29-4e5bb0d8b494	GBV	aDiwXzZfmtKdrybADM9YVX	19efcb4f-8b45-47b7-ad29-4e5bb0d8b494	uuid:19efcb4f-8b45-47b7-ad29-4e5bb0d8b494	submitted_via_web	2026-06-15 12:59:50	NA	NA
2026-06-15 12:59:49.805	2026-06-15 13:00:36.057	NA	BII00001	HEALTH	Psychosocial Support	2024-07-17	MENTOR	NO	YES	2024-08-21	ONGOING	781078822	83d3f433669241309347dd234afdb5b9	vKSiY7iGN9VP8CxaC88fCn	uuid:449b25ab-0878-4a4d-9223-564f41c4a334	NA	aDiwXzZfmtKdrybADM9YVX	449b25ab-0878-4a4d-9223-564f41c4a334	uuid:449b25ab-0878-4a4d-9223-564f41c4a334	submitted_via_web	2026-06-15 13:00:36	NA	NA

head(screening_data, 2) %>% 
  gt()

start	end	**Introduction** This is a simulated screening tool for the fictional Binti Imara Initiative, used to assess vulnerabilities and risks among adolescent girls and young women. Data collected is for training and analysis only and does not represent real beneficiaries. The initiative supports all girls, including those who are HIV positive, pregnant, or young mothers. Use responsibly.	Registry ID	Date of Assessment	Name of Assessor	Type of Assessment	School Dropout Risk	GBV Risk	Child Marriage Risk	Transactional Sex Risk	Food Insecurity	Caregiver Support	Depression Screening	Suicidal Ideation	vulnerability_score	vulnerability_level	Vulnerability Score: ${vulnerability_score}/10 — Level: ${vulnerability_level}	_id	uuid	__version__	instanceID	_xform_id_string	_uuid	rootUuid	_status	_submission_time	_validation_status	_submitted_by	_attachments
2026-06-15 13:38:24.603	2026-06-15 13:39:20.358	NA	BII00007	2025-01-13	Robert	Baseline	1	0	0	0	1	1	3	1	6	Moderate Vulnerability	NA	781125119	6efd0da9808d4ca9a3cc4c058f83ebf9	vHMCCwPX8xPonUPVC4ezbd	uuid:a83d45b6-b7a5-4263-a515-0e4af749cc39	aKboNAnbeZsCN5nUjLBmNw	a83d45b6-b7a5-4263-a515-0e4af749cc39	uuid:a83d45b6-b7a5-4263-a515-0e4af749cc39	submitted_via_web	2026-06-15 13:39:21	NA	NA
2026-06-15 13:39:20.4	2026-06-15 13:40:12.472	NA	BII00005	2026-04-16	Mercy	Baseline	0	1	0	0	1	1	1	1	4	Moderate Vulnerability	NA	781125947	6efd0da9808d4ca9a3cc4c058f83ebf9	vHMCCwPX8xPonUPVC4ezbd	uuid:2bad0faf-0cdf-403c-8c01-4f05378aefa4	aKboNAnbeZsCN5nUjLBmNw	2bad0faf-0cdf-403c-8c01-4f05378aefa4	uuid:2bad0faf-0cdf-403c-8c01-4f05378aefa4	submitted_via_web	2026-06-15 13:40:13	NA	NA

Each dataset is now available as a tidy R dataframe

Step 3: Output for Dashboard

The cleaned dataset feeds directly into a Shiny dashboard displaying:

• Active beneficiaries

• High-risk cases

• Service coverage

  

  Old Data (in RAM)

  ↓ click Refresh

  Download new data from KoBo

  ↓

  Replace old data in reactiveVal

  ↓

  Shiny automatically redraws everything

Live Dashboard

The interactive Monitoring & Evaluation dashboard is deployed on shinyapps.io and can be accessed below:

👉 Binti Imara M&E Dashboard

Further analysis and interactive visualizations will be progressively enhanced as additional data is captured through ongoing field entry.

🔗 Access the Data Collection Forms

The following KoBoToolbox forms power the end-to-end M&E data pipeline:

• Registry Form:

• Screening Form:

• Intervention Tracking Form:

7. Impact of the System

a. Cost Efficiency: Eliminates dependency on paid M&E software → $0 licensing cost

b. Operational Efficiency: Reduces reporting time from days → minutes

c. Data Quality Improvement: Automated validation reduces human entry errors

d. Real-Time Decision Making: Stakeholders access up-to-date program insights anytime

References

• KoboToolbox. (2026). KoboToolbox documentation and API guide.

• KoboToolbox Help Center. (2026). Getting started with KoBo API and data access.

• robotoolbox: A quick tutorial

• Chang, W., et al. (2025). shiny: Web application framework for R.

• RStudio/Posit. (2026). Shinyapps.io deployment platform.

• ggplot2 (tidyverse). (2026). Elegant data visualization in R.