Research Design and Statistics I
(PST010)
Richard D. Morey, coordinator
Academic year 2018-2019
Lecturers
Your lecturers are Richard D. Morey (coordinator) and Michael Lewis. If necessary, you may contact them via email:
| Lecturer | Website | |
|---|---|---|
| R. Morey | [website] | moreyr@cardiff.ac.uk |
| M. Lewis | [website] | lewisMB@cardiff.ac.uk |
Forum information
The PST010 forum on Learning Central will act as a central conduit for information and discussion. If you have questions about the course please post them there, so that others can benefit from the answers. The lecturers will monitor the forums and provide help.
Software
Answers to the exercises will be offered using SPSS.
Brief schedule of topics
Lectures will be held in Lab 1.07 in the Tower building.
| Date | Time | Lecturer | Topics | |
|---|---|---|---|---|
| 0 | 4 Oct | 9:00-11:00 | R. Morey | Introduction/Refresher (optional) |
| 1 | 11 Oct | 9:00-11:00 | R. Morey | data management, summary and visualization |
| 2 | 25 Oct | 9:00-11:00 | R. Morey | probability, estimation, and hypothesis testing |
| 3 | 8 Nov | 9:00-11:00 | R. Morey | one- and two-sample designs |
| 4 | 22 Nov | 9:00-11:00 | R. Morey | one-way designs |
| 5 | 6 Dec | 9:00-11:00 | M. Lewis | correlation and regression |
| 6 | 13 Dec | 9:00-11:00 | R. Morey | Review, question and answer |
Each practical session will include the following components.
- Opportunity for questions
- Theoretical lecture (c. 45 minutes)
- Short break (c. 5 minutes)
- Review of assignment and practical lecture (c. 45 minutes)
Solutions will be provided to all assignments. Any of the material covered in the assignments may be examined, so please be sure you read the solutions.
Preparation for each lecture
It is expected that you will come to each lecture prepared. There are three components to prepare: background textbook readings, an essential reading, and an exercise.
The background text readings are optional in the sense that some students may have had this content in a previous course, and the essential material will be covered in the lecture. Some students may be therefore be comfortable skimming this material to refresh, others may need to more deeply read the background material. The lectures will cover some of this material, but a general familarity with the content background reading will be helpful for preparing for the lectures and thus the exam.
The background text will be Andy Field’s Discovering Statistics Using IBM SPSS Statistics, 5th edition. The “background text” below refer to sections of the Field text. Please note that the readings are not ordered as they are in the book.
Review after the practical
A “study question” for each practical is provided. In the days after the practical, try to answer this question in short essay form. Do your best to write a thorough, structured answer that shows that you grasped the material, and try to write it in 10 minutes, without access to your notes. The question is meant to make you think more deeply about the material.
The instructors will be happy to comment on any answer that you post to Learning Central. Please take advantage of this offer as you progress through the module; it will help you perform better on the exam.
Helpful additional reading
In addition to the review, a number of additional resources are offered. These may be readings to help solidify your understanding of the week’s content, interesting readings to spur your thinking about problems in statistical analysis, or even games or interactive applets.
The content in the further reading will not be examined.
Lecture 0
Preparation
Preparation for this introductory lecture is not strictly necessary, but if you have not had much exposure to statistical reasoning in science, you should read “Statistical Thinking” by Beth Chance and Allan Rossman:
- Chance, B. & Rossman, A. (2018). Statistical thinking. In R. Biswas-Diener & E. Diener (Eds), Noba textbook series: Psychology. Champaign, IL: DEF publishers. DOI:http://noba.to/ruaz6wjs
If you have never used SPSS before, it might be good for you to also read Chapter 4 from the Field textbook, “The IBM SPSS statistics environment”.
Lecture 1
Preparation
| Material type | Content | Link |
|---|---|---|
| Essential reading (4 pages) | Rousselet, Foxe, & Bolam (2016). A few simple steps to improve the description of group results in neuroscience | |
| Essential video (12 minutes) | R. Peng, “Principles of Graphics” YouTube | |
| Corresponding sections of background text | Chapters 5-6 |
Review
In the April, 2017 article “Ride-hailing apps may help to curb drunk driving”, The Economist’s data team writes:
According to a working paper by Jessica Lynn Peck of the Graduate Centre at the City University of New York, the arrival of Uber to New York City may have helped reduce alcohol-related traffic accidents by 25-35%. Uber was first introduced in the city in May 2011, but did not spread through the rest of the state. The study uses this as a natural experiment. To control for factors unrelated to Uber’s launch such as adverse weather conditions, Ms Peck compares accident rates in each of New York’s five boroughs to those in the counties where Uber was not present, picking those that had the most similar population density and pre-2011 drunk-driving rate.
The four boroughs which were quick to adopt Uber—Manhattan, Brooklyn, Queens and the Bronx—all saw decreases in alcohol-related car crashes relative to their controls. By contrast, Staten Island, where Uber caught on more slowly, saw no such decrease.
They present the following graphic:
Evaluate why this graphic works well in the context of the article. What features does the graphic have that help a reader understand the point being made?
Helpful resources
A. Abela (2006). Choosing a good chart PDF
The Financial Times’ Visual Vocabulary: a chart for data visualization options.
Further reading/watching
A. Reese (2017). Graphical interpretations of data: An introduction. Significance magazine.
Allen & Erhardt (2016). Visualizing Scientific data PDF
Lecture 2
Preparation
| Material type | Content | Link |
|---|---|---|
| Essential reading (3.5 pages) | Gelman & Stern (2006). The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant | |
| Corresponding sections of background text | Chapter 2 |
Review
Describe, in your own words, what Gelman and Stern mean when they say that “[t]he difference between ‘significant’ and ‘not significant’ is not itself statistically significant.”
Helpful resources
B. Foltz, “Type I and Type II errors” YouTube
B. Foltz, “Hypothesis testing: Single sample with known σ” YouTube
Wilkinson and the Task Force on Statistical Inference (1999). Statistical Methods in Psychology Journals: Guidelines and Explanations. American Psychologist (54), 594-604.
Further reading/watching
- Gelman & Weakliem (2009). Of Beauty, Sex, and Power. American Statistician (97), 310-316.
Lecture 3
Preparation
| Material type | Content | Link |
|---|---|---|
| Essential reading (9.5 pages) | LaFleur & Greevy (2009). Introduction to Permutation and Resampling-Based Hypothesis Tests | |
| Corresponding sections of background text | Chapter 10; Sections 7.1-7.5 |
Review
Describe two situations when you might use a two-sample permutation test over a parametric one (e.g., a t-test); detail your reasoning.
Helpful resources
- B. Foltz, “Two Populations, t-test with Hypothesis” YouTube
Further reading/watching
- Wasserstein & Lazar (2016). The ASA’s Statement on p-Values: Context, Process, and Purpose American Statistician (70), 129-133.
Lecture 4
Preparation
| Material type | Content | Link |
|---|---|---|
| Essential reading (10 pages) | Rousselet, Pernet, & Wilcox (2017). Beyond differences in means: robust graphical methods to compare two groups in neuroscience. PDF European Journal of Neuroscience (46), 1738–1748. | |
| Corresponding sections of background text | Chapter 12; Section 7.6 |
Review
In a one-way ANOVA, what are the two sums of squares used to calculate the F ratio, and what affects their size? Use a sketch if necessary. If the F ratio is around 1, how do we interpret the result?
Helpful resources
Further reading/watching
With more complicated designs, the opportunities for fooling ourselves and others increase. These two further readings relate to this problem.
Simmons, Nelson, & Simonsohn (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science (22), 1359-1366.
Schönbrodt, F. D. (2015). p-hacker: Train your p-hacking skills!.
Lecture 5
Preparation
| Material type | Content | Link |
|---|---|---|
| Essential reading (14 pages) | Ernst & Albers (2017). Regression assumptions in clinical psychology research practice — a systematic review of common misconceptions | |
| Corresponding sections of background text | Chapters 8-9 |
Review
Consider the following scatterplot, with the least-squares regression line in blue.
Sketch the corresponding residual plot. Does it have any characteristics you might be worried about from a linear regression perspective if these were actual data? Which, if any?
Helpful resources
B. Foltz, “Understanding Correlation” | YouTube
B. Foltz, “Simple Linear Regression, The Very Basics” | YouTube
“Guess the correlation”: A game where you get points by accurately guessing the Pearson correlation between two variables.
Further reading/watching
- M. McArdle (2012). When Correlation Is Not Causation, But Something Much More Screwy. The Atlantic.
Lecture 6
Lecture 6 is less a lecture and more a student-driven review session. Please come prepared with any clarification questions to ask. You can also post questions to the Learning Central with the subject “Review session question” and I will answer it in the session. There will not be prepared slides for this lecture; the topics to be discussed can be anything from the module that a student would like to review.
Assessment
The final module mark for the module will be based on 30% coursework (essay) and 70% exam.
Essay
Find an scientific article that interests you (either a published one or a preprint) that uses one of the designs (one- or two-sample, one-way, correlational) or techniques (t test, Wilcoxon test, permutation test, one-way ANOVA, Kruskall-Wallis test, correlation, regression) discussed in PST010.
Essay organization
The essay will be a critique of a primary data visualization in a published scientific paper, and should be about 600 words.
Organize your essay in the following way:
- Introduction: Describe the research question and analysis in the article, and why that analysis was chosen.
- Critique a central visualization: What visualization did the authors use to communicate their results, if any? What are the good and bad points of this visualization? Design a better visualization; or, if the authors did not use a visualization, create one that is appropriate.
- Discussion: justify your choices visualization using principles discussed during the module. Describe any difficulties you encountered during the process.
Essay Marking
Definition of key terms:
Knowledge and understanding: Using knowledge of relevant theories and literature, demonstrating an understanding of the relevance and success of theories in the broader dimensions of the area, evidence of understanding the limits of current understanding.
Analysis and evaluation: Demonstrating awareness of gaps or limits in the knowledge base, selecting appropriate methods of enquiry, presenting a lucid rational argument with clear well thought out conclusion and an appreciation of the future direction of work in the topic area.
Synthesis: Demonstrating the combination of different viewpoint/levels of analysis when dealing with complex and perhaps conflicting information. Considering and identifying the appropriateness of methods and/or experimental design.
Originality and innovation: The demonstration of original thinking – that is, evidence of independent analysis of information, and presenting new ideas and or the application of techniques in novel ways to address specific problems.
Independent learning: The piece of work demonstrates evidence of relevant reading and reflection on material outside the lecture content or essential reading.
Marking scale
A mark of 50 or above is considered “passing”.
Distinction (70-100): An essay marked with distinction should, at a minimum, be correctly formatted throughout with few minor errors. The rationale for the study and the statistical analysis should be developed accurately and succinctly. The work should show (1) evidence of accurate knowledge and understanding of the key statistical concepts. There should be (2) evidence of synthesis of material and (3) critical analysis (e.g., the strengths and limitations of various approaches one could take data analysis) of relevant material that is justified and that connects to the research question of the chosen paper. The work should demonstrate (4) independent learning and synthesis of material (5) and provide a logical discussion regarding the limits of the statistical approaches and implications that go beyond practical handouts/lectures or core texts.
Higher marks with distinction (80 and above) should display sustained quality in all of the areas described for distinction with no errors or critical omissions. Relevant concepts should be discussed and analysed in a critical fashion, e.g., competing/alternative analyses should be presented clearly and cogently discussed. There should be evidence of independent learning accompanied by original insight into problems and solutions. The consideration of potential alternative approaches should clearly display evidence of originality in terms of identifying problems and/or proposals for novel analyses.
Very high marks with distinction (90-100) should show all the characteristics of discussed above and, in addition, present clear evidence of sustained innovation and originality in terms of knowledge, understanding (e.g., linking concepts in novel ways) and critical analysis (insight into issues or the appropriateness of methods and analysis) that is clearly based upon independent learning and demonstrates novel synthesis of information. Consideration of potential alternative approaches should clearly demonstrate sustained originality and innovation.
Merit (60-69): A essay in the merit category will, at a minimum (60), display the majority of the following characteristics: (1) It should have only minor omissions or errors; (2) the content should be appropriate, accurate and convey the research question and statistical content clearly but may lack depth. The discussion will be largely appropriate but may need greater depth and scope. Consideration of the implication of the arguments will be limited and will reflect information provided in lecture content/hand out material. A better answer within this class should be more comprehensive, show evidence of depth of understanding of core material and (4) clear evaluation in developing the rationale for the report (e.g., contrasts are made between approaches). (5) Good answers within this class should develop its topic clearly and illustrate the above together with evidence for some independent learning and synthesis of relevant material. The general discussion should reflect on the rationale for the critiques and connection with broader scientific practice. The discussion should demonstrate critical understanding of lecture/core texts and recommended readings.
Pass (50-59): A minimum pass (50) should (1) be clearly and succinctly written and appropriately formatted but will have some errors and omissions. The essay should (2) show evidence of a basic understanding and application of relevant material to explain the statistical analyses. The level of knowledge (e.g., reference to key statistical concepts) shown in the introduction and discussion is limited in scope and depth, and may be incomplete and does not go beyond lecture material/core texts. Better answers will show evidence of synthesis but it will not typically extend beyond that provided in lectures/core texts. There may be some irrelevant arguments, and may contain factual errors and/or omissions. Higher marks within this category should demonstrate evidence of (1) an accurate understanding of the material and its appropriate application to the statistical report (e.g., demonstrate the knowledge of key concepts and their underpinnings), (2) but should lack depth and critical analysis and contain some errors or omissions. Any consideration of the broader implications of the arguments may be weak or not fully developed.
Failing (40-49): A failure with high mark (40-50) should be formatted correctly but with substantial omissions or inaccuracies. The introduction will show some but limited knowledge of material directly related module material. The development of the rationale for the study and analysis should be weak (there is no reflection on basic statistical concepts and their underpinnings) and the material should be limited to lecture/handout content. There should be no evidence for independent learning.
30-39: A piece of work that contains extensive errors and omissions and fails to address the question or topic. There is only a small amount of acceptable or relevant information. There is no evidence of appropriate critical awareness or analysis and scant use of lecture material. The answer will be poorly structured with minimal integration of relevant ideas and little or no evidence of understanding the wider context. The work may demonstrate a failure to follow instructions appropriately.
20-30: A piece of work at this level should fail to demonstrate sufficient relevant knowledge and understanding to address the question, and have serious errors and omissions. There may be evidence of misconceptions, errors and irrelevancies.
10-20: A piece of work that is based at this level will reflect minimal appropriate knowledge beyond that one might expect from a lay person. There is little, if any, appropriate argument and the answer does not address the question. There is no evidence of conceptual understanding. Any relevant knowledge is marred by serious errors.
1-9: A piece of work that displays only vague application of relevant knowledge. Any relevant knowledge should be compromised by serious errors and omissions and/or a failure to follow instructions. There are few if any relevant ideas and those present are vague and expressed with no evidence of critical awareness or the ability to link ideas.
0: There is no answer. There is no evidence of relevant knowledge and understanding for psychology, or does not answer the question set.
Essay feedback
Generic feedback will be given on essay marks; however, any student wishing to discuss their mark with one of the instructors should make an appointment via email.
Exam
The exam will consist of approximately ten short conceptual essay questions. These questions will be similar in form to the questions in the “Review” section of each lecture above. The exam content will cover the essential readings/videos, the conceptual content in the exercise solutions, and the lectures. You will be asked to write several paragraphs showing your understanding of the material in the preparatory readings and practicals; the expected length and depth of your answer will depend on the question. For general marking guidelines, please see your handbook. Last year’s exam questions and response guidance are also available.
In order to study for the exam, ensure that you have read the background reading, the essential reading, and the solutions to exercises. Time management is essential; it is highly recommended that you do this prior to the corresponding lecture to avoid having to cover all the material in the weeks before the exam.
Two good ways of studying for the exam are to attempt to answer the review questions and to generate your own questions, which you or others then attempt to answer. I will comment on any answers posted to Learning Central.
If you are unclear about any module content, please post a question on the Learning Central forums. If for some reason you’d prefer not to do this, please send me an email with your question, and I will either answer via email or suggest we meet in person. Learning Central my preferred method of answering questions; it has the advantage that other studens can benefit from my answers to questions.
This document was compiled Tue Sep 25 14:49:48 2018 (Europe/London).