This project is an opportunity to apply statistical learning to a real-world problem of your choice. You will work in groups on this task.
The project is deliberately open-ended. There is no prescribed dataset, statistical-learning method, or type of problem. You are expected to identify an interesting real-world problem, find suitable data, formulate appropriate statistical or machine-learning objectives, and carry out the analysis.
A major part of the project is independent learning. You are expected to study and apply some material beyond what is taught in MH4510.
Your project does not have to involve supervised learning. A completely unsupervised project is perfectly acceptable if it addresses an interesting real-world question. For example, a project may investigate spatial patterns in crime data represented by geographical coordinates using several clustering methods. In such a project, the objective may be to explore and understand the structure of the data rather than to predict a response variable.
The project is designed to assess your ability to:
You are expected to complete the following stages. The exact analysis will depend on your project, so not every project will follow exactly the same workflow.
Choose and formulate a real-world problem.
Explain the background of the problem and why it is worth investigating. Carry out an appropriate literature review.
Most importantly, translate the real-world problem into an appropriate statistical or machine-learning problem. Think carefully about what you actually want to learn from the data.
For example, suppose you are interested in understanding how lifestyle factors are related to mental-health problems. Training a classifier and maximizing its prediction accuracy does not necessarily answer this question. If the objective is to understand which factors are associated with mental health and how, interpretable models and statistical inference may be considerably more useful than a highly accurate black-box classifier.
Obtain and prepare appropriate data.
You may use publicly available datasets or collect data from multiple sources. You are encouraged to go beyond simply downloading a ready-made tabular dataset.
Clean the data as necessary and explain important preprocessing decisions. Identify the response variable, where applicable, and determine which variables should be used as features.
Pay particular attention to variables that should not be used. For example, variables that contain information that would not be available at prediction time may result in data leakage.
Perform exploratory data analysis.
Your report should contain both:
The purpose of exploratory data analysis is not to produce as many tables and plots as possible. The tables and visualizations should help the reader understand the data and should be relevant to the subsequent analysis.
Perform feature engineering where appropriate.
Identify and justify the features used in your analysis. Depending on the project, this may involve variable selection, transformations, construction of derived variables, encoding of categorical variables, text embeddings, dimension reduction, or other techniques.
Feature engineering should be motivated by the problem and the methods you intend to use rather than performed mechanically.
Choose methods that are appropriate for the objective of your project. These may be supervised, unsupervised, or a combination of both. You are not required to train a predictive model if prediction is not relevant to your real-world problem.
For supervised learning, you should normally separate the data used for training from the data used for final evaluation, address overfitting, and tune hyperparameters where necessary.
For unsupervised learning, you should similarly justify the methods and their parameters in relation to the objective of the project. For example, a clustering project should not simply run several clustering algorithms and report the resulting clusters; you should explain what structure you are trying to discover, why the chosen methods are appropriate, and what the resulting clusters tell you about the original problem.
Apply any preprocessing required by your methods. For example, methods based on distances or penalization may require variables to be appropriately rescaled.
You should be able to explain why each method was chosen and the basic theory behind the methods you use.
Choose an appropriate evaluation criterion.
Do not automatically compare models using a collection of standard metrics simply because they are commonly reported.
Your project should have one clearly identified primary evaluation criterion that reflects the real-world objective of the analysis.
For example, in a cancer-screening problem, a false negative may be substantially more serious than a false positive. In this situation, overall accuracy, F1-score or AUC may not adequately represent the objective of the problem. You may need to define an evaluation criterion that assigns an appropriate cost to different types of errors.
Other performance measures may still be reported when they are informative, but the selection of the preferred model should be driven by the primary objective of the project.
Learn material beyond the course syllabus.
Independent learning is a major component of the project. You are expected to study statistical or machine-learning methodology that is not taught in MH4510 and apply it correctly to your problem.
It is not sufficient simply to call an R function implementing a new method. You should understand and briefly explain the theory behind the method and demonstrate that you understand how it is used.
Write the final report.
Prepare a structured report using R Markdown and knit it into a PDF. Pure LaTeX and Jupyter Notebook reports are also acceptable.
Your report should explain the problem, data, methodology and conclusions in clear English and full sentences. Code should be included in the source document, but the written report should read as a coherent scientific report rather than as a sequence of R commands with comments.
Prepare the final presentation.
Prepare no more than 4 slides using R Markdown, Beamer, Jupyter, or another suitable reproducible system.
The presentation should last no more than 10 minutes. All team members should understand the entire project and should be prepared to answer questions, including questions about work primarily carried out by another team member.
An \(n\)-student team is expected to include at least \(n\) substantial analytical components.
These components do not have to be supervised-learning models and may occur at different stages of the project. For example:
What matters is that each component represents a substantial piece of analytical work. Several trivial variations of essentially the same analysis do not constitute several substantial components.
Although individual team members may take primary responsibility for different components, the final submission should form one coherent project, rather than a collection of independent analyses. Every team member is expected to understand the overall project and its main methodology and conclusions.
| Time | Milestone |
|---|---|
| Week 2 | Form project teams of 4 or 5 students and submit the list of team members to Fedor. |
| Week 3 | Fedor assigns teams to students who have not formed their own team. |
| Week 6 | Start researching possible project ideas and datasets. Explore Kaggle, data.gov.sg, and other suitable sources. |
| Week 7 | Come up with a real-world problem involving statistical learning and share your idea with Fedor. |
| Recess Week | Submit a formal project proposal containing an introduction to the problem, a description of the dataset, project objectives, proposed methods, and any progress made so far. |
| Week 12 | Meet Fedor for informal feedback on your progress. |
| Week 13 | Give the project presentation and submit a draft version of the report for preliminary grading. |
| Study Week | Submit the final version of the report for final grading. |
Each criterion is assessed at one of five levels:
The score for each criterion is its maximum mark multiplied by the corresponding value.
| Criterion | MAX | Unsatisfactory (0) | Poor (0.25) | Average (0.5) | Good (0.75) | Excellent (1) |
|---|---|---|---|---|---|---|
| Problem formulation / literature review | 10 | The problem was suggested to the students by the course instructor, or there is no meaningful attempt to formulate a real-world problem. | A real-world problem is stated, but the statistical / machine-learning task does not meaningfully address it. The analysis appears to be driven mainly by the available dataset or by standard prediction metrics rather than by the stated research question. | The real-world problem and its background are explained, but its translation into a statistical / machine-learning problem is incomplete or partly inappropriate. For example, the project may focus on predictive accuracy when explanation or statistical inference is needed to answer the stated question. | The real-world problem is well motivated by appropriate background research and is translated into an appropriate statistical / machine-learning problem. The choice between prediction, explanation, statistical inference, etc. is broadly appropriate, although some aspects of the analysis may not be fully aligned with the stated objectives. | The real-world problem is clearly motivated by appropriate background research and translated precisely into statistical / machine-learning objectives. The choice of response, predictors, modelling approach, evaluation criteria, interpretation and, where appropriate, statistical inference is consistently driven by the real-world question. |
| Exploratory data analysis | 5 | Data exploration is not included in the report. | There is an attempt at data exploration, but it is very limited or does not meaningfully help the reader understand the data. | Data exploration is reasonably relevant to the analysis, but either appropriate summary tables or appropriate visualizations are missing. | Both summary tables and visualizations are included, but their choice is not well aligned with the subsequent analysis; for example, some tables or plots appear to have been included without a clear purpose. | Both appropriate summary tables and visualizations are included, and their choice is well aligned with the subsequent analysis and helps the reader understand the data relevant to the problem. |
| Feature engineering | 10 | The response variable is not correctly identified, or the features used do not allow the stated problem to be meaningfully addressed. | The response variable is identified, but there is little or no meaningful consideration of which variables should be used as features. Important problems with the features are ignored. | The response variable and features are identified, but feature selection and feature engineering are largely mechanical or insufficiently justified. | The response variable and features are appropriately identified. Feature selection and, where appropriate, feature engineering are explained and broadly justified in relation to the problem, but there are some omissions or questionable choices. | The response variable and features are appropriately identified. Feature selection and feature engineering are carefully justified by the real-world problem, the nature of the variables and the intended models. Relevant transformations, derived variables, encoding and exclusion of inappropriate variables are properly considered. |
| Modelling | 15 | The modelling procedure contains fundamental errors that make the results invalid. | There are major flaws in the modelling procedure. For supervised learning, this includes evaluating models without a proper separation of training and test data. Other examples include choosing inappropriate models, selecting parameters essentially at random, or using evaluation criteria that are seriously inappropriate for the real-world problem. | The modelling procedure is broadly valid, but important parts are missing or seriously incomplete. For supervised learning, this includes not performing hyperparameter tuning when it is required. There may also be serious shortcomings in dealing with overfitting or choosing an appropriate evaluation criterion. Several standard performance metrics may be reported without identifying which one corresponds to the objective of the project. | The modelling procedure is generally appropriate. Models are trained and tuned reasonably, overfitting is addressed, and an appropriate primary evaluation criterion is identified. There may nevertheless be smaller methodological problems, such as failing to rescale variables when this is required by the method, in either supervised or unsupervised learning, or an incomplete justification of modelling choices. | The modelling procedure is fully appropriate for the problem. Models and hyperparameters are carefully selected, overfitting is properly addressed, and all necessary preprocessing for the chosen methods is performed. Model comparison is based on one clearly defined primary evaluation criterion derived from the real-world objective. Where standard metrics do not adequately represent the consequences of different errors, an appropriate custom evaluation criterion is defined and justified. |
| Coding | 10 | There is no R code at all, it contains syntax errors, or it does not do what it is supposed to do. | The code does what it is supposed to do but is either very inefficient or incomprehensible. This includes inappropriate recursion, breaking loops, creating matrices by assigning values entry by entry, etc. | The code does what it is supposed to do, but may be inefficient because of lack of vectorization, unnecessary loops or control statements, nested loops, or failure to use appropriate native functions. | The code does what it is supposed to do, but may be somewhat inefficient. Sometimes the code may be hard to understand because of a lack of commenting or obscure variable names. | The code is efficient, easy to understand and does what it is supposed to do. It makes appropriate use of existing R libraries and the tidyverse, and avoids loops whenever possible. Important variables have meaningful names, and there are comments explaining the code whenever needed. |
| Material beyond the course syllabus | 25 | The project is completely within the course syllabus. | The project includes a small amount of work beyond the course syllabus, but no substantial new statistical or machine-learning methodology is learned. Examples include combining several datasets or other somewhat unusual data preparation. | The team independently learns and applies a standard statistical or machine-learning method that is not covered in the course but is comparable in difficulty to methods that are covered. Examples include XGBoost, support vector regression, DBSCAN, or k-medoids clustering. The method is correctly implemented and its underlying theory is briefly explained. | The team independently learns and applies a substantially more advanced method that goes beyond standard introductory statistical-learning material. Examples include symbolic regression, recurrent neural networks, or clustering / unsupervised-learning methods based on more advanced ideas such as topological data analysis. The team demonstrates a good understanding of both the theory behind the method and its implementation. | The team undertakes unusually advanced, creative or original work substantially beyond the course syllabus. The work goes beyond simply applying a standard advanced method and demonstrates deep understanding of both the methodology and its implementation. This level is reserved for exceptional projects. |
| Presentation | 10 | The team was not able to prepare the presentation for class in Week 13. | The number of slides exceeds 4. The presentation is very messy and the speaker does not seem to be familiar with the scope of the project. | There are not more than 4 slides, which may be prepared in PowerPoint / Google Slides. The presentation may be very messy or substantially exceed 10 minutes. Some team members may not be familiar with the structure of the project. | There are not more than 4 slides, professionally prepared in R Markdown / Jupyter / Beamer. The presentation is very clear but may last slightly more than 10 minutes. Students may not be familiar with their teammates’ contributions. | There are not more than 4 slides, professionally prepared in R Markdown / Jupyter / Beamer. The presentation is very clear and lasts not more than 10 minutes. All team members are able to answer questions about the project. |
| Final report | 15 | There is no report at all; for example, it is not submitted or is submitted as commented R code rather than as a structured report knitted into a PDF. | The report is not written in full English sentences and is hard to follow. There may be serious mistakes in style, grammar and structure that make understanding the report challenging. The abstract or reference list may be missing, or the report may be excessively long. | The report is mostly written in full English sentences, but is not easy to follow because of mistakes in style or grammar. It may be too wordy, the abstract may be placed at the end, the report may be produced in MS Word, or sources may not be properly cited. | The report is written in full English sentences, is properly structured and is easy to follow. It has been prepared as an R Markdown / Jupyter Notebook / LaTeX document and is concise. There may be minor mistakes in style or grammar that do not substantially affect readability. Tables and figures may have minor formatting problems. The abstract may exceed 200 words or may not summarize the findings adequately. | The report is written in full English sentences, is properly structured and is easy to follow. It has been prepared as an R Markdown / Jupyter Notebook / LaTeX document and is concise. As a rule of thumb, a standard project should contain about 8 pages of text, tables and plots (excluding code), while 10–12 pages may be appropriate for a project containing substantial difficult material beyond the course syllabus. Tables and figures are professionally formatted. The abstract is not longer than 200 words and summarizes all important findings. All sources are appropriately cited. There are no significant mistakes in style or grammar. |
The project contributes 40% of the total course mark.
The rubric above is used to determine the team score. Individual students in the same team may receive different project marks depending on their contribution to the project and their participation in the peer-evaluation process.
Your individual project score is calculated as
\[ 0.35T + 0.55IT + 0.05C + 0.05R, \]
where:
Thus, 90% of the project score is based on the quality of the project itself, with 55% of the score adjusted according to your individual contribution. The remaining 10% rewards careful and constructive participation in peer evaluation.
At the end of the project, team members will evaluate each other’s contributions. Your individual contribution \(I\) is determined from these peer evaluations using a robust procedure based approximately on the median rather than the mean, so that a single unusually high or low evaluation has limited influence on the result.
The precise peer-evaluation algorithm is described in:
Fedor Duzhin, Peer Evaluation for Group Projects, in the paper available here.
Your individual project score cannot exceed 100. If the formula above produces a score greater than 100, it will be changed to exactly 100. In other words, it is impossible to obtain more than the full 40% of the total course mark allocated to the project.
If your individual contribution \(I\) is 0, your entire project score will be changed to 0, irrespective of the team score or the other components of the formula.
For the purpose of the rubric, methods found in standard introductory statistical-learning material, such as An Introduction to Statistical Learning, will normally be considered Average rather than Good, even if they are not explicitly covered in MH4510.
Examples of material that would normally fall into the Average category include XGBoost, support vector regression, DBSCAN and k-medoids clustering, provided that the method is properly implemented and its underlying theory is understood and explained.
The Good category is intended for substantially more advanced material, such as symbolic regression, recurrent neural networks or methods based on more advanced ideas such as topological data analysis.
The distinction between Average, Good and Excellent is primarily based on the depth and sophistication of the material learned, rather than on the number of additional methods used. One genuinely difficult piece of additional methodology may demonstrate substantially more independent learning than several routine extensions of methods covered in class.
The Excellent category is deliberately not associated with a fixed list of methods. It is reserved for unusually advanced, creative or original work and is not awarded routinely.
Exploratory data analysis versus presentation quality: The EDA criterion assesses whether appropriate tables and plots are present and whether they are useful for the analysis. The visual and technical quality of these outputs is assessed elsewhere. For example, a relevant but unattractive base R plot may still satisfy the EDA requirement, while reducing the Coding score. Similarly, raw R output may contain the necessary summary information but reduce the Final report score.
Programming quality: Use R in a clear, efficient and reproducible way. In particular, use the tidyverse where appropriate, avoid unnecessary loops and complicated control structures, and make use of suitable functions and existing R libraries rather than reimplementing standard functionality.
Tables and figures: Tables appearing in the final report should be professionally formatted rather than presented as raw R output. Figures should be clear, appropriately labelled and suitable for inclusion in a professional report.
Understanding your methods: Using an R package does not by itself demonstrate understanding of a method. You should be able to explain the main ideas behind every substantial method used in your project, including material that you learned independently.
Coherence: Do not treat the project as a competition to accumulate as many models, metrics or techniques as possible. Every substantial part of the analysis should serve the objective of the project.
You are allowed to use generative AI tools such as ChatGPT, Copilot or similar tools to assist with your project.
For example, they may be used to:
However, all use of generative AI tools must be declared in the final submission.
You remain fully responsible for everything contained in your project. In particular, you must understand the methods and code you submit and be able to explain them during the presentation. Incorrect statements, inappropriate methodology or non-working code remain errors regardless of whether they were produced with the assistance of an AI tool.
Fedor used ChatGPT to assist in preparing this document. Fedor determined the objectives of the project, project requirements, assessment criteria, grading philosophy and examples used in the rubric. ChatGPT was used to help refine the wording, organize the material, and format the document in R Markdown.