How to Choose a Statistical Test
Choosing a statistical test is mostly a matter of answering four questions about your data in the right order. Once you can answer them, the choice narrows to one or two options.
Students often approach test selection as if there were a hidden correct answer to memorise. In reality, the choice follows logically from what you want to know and what kind of data you collected. This guide sets out the decisions in sequence, gives a reference table for common designs and explains how to justify your choice in a methods chapter.
Question one: what are you trying to find out?
Most analyses fall into one of three families. You are either comparing groups (do students who received feedback score higher than those who did not?), examining a relationship between variables (is study time associated with exam score?), or predicting an outcome from several variables at once (which factors predict retention?). Write your question in one of these forms before going further. If it fits none, it may need sharpening.
Stuck on your assignment? Our expert writers can help.
Question two: what type is your outcome variable?
The measurement level of your dependent variable drives most of the decision.
- Continuous — measured on a scale with meaningful intervals, such as test scores, blood pressure or time.
- Ordinal — ranked categories, such as satisfaction from “very poor” to “very good.”
- Categorical — groups with no order, such as pass or fail, or treatment type.
Likert items are technically ordinal, but summed scales of several items are often treated as continuous. Check your discipline’s convention and state your reasoning.
Question three: how many groups, and are they independent?
For comparisons, count the groups and ask whether the same participants appear in more than one. Two separate groups need a different test from the same group measured twice. Paired or repeated designs are more powerful because each person acts as their own control, but they must be analysed as such — treating repeated measures as independent inflates your chances of a false positive.
Question four: do the assumptions hold?
Parametric tests such as t-tests and ANOVA assume roughly normally distributed residuals and similar variance across groups. When those assumptions are badly violated, especially in small samples, non-parametric alternatives that work on ranks are safer. In larger samples, parametric tests are fairly robust to moderate departures from normality. Check with plots as well as formal tests, and report what you found.
A reference table for common designs
| Design | Parametric | Non-parametric |
|---|---|---|
| Two independent groups, continuous outcome | Independent t-test | Mann-Whitney U |
| Same group measured twice | Paired t-test | Wilcoxon signed-rank |
| Three or more independent groups | One-way ANOVA | Kruskal-Wallis |
| Same group, three or more times | Repeated-measures ANOVA | Friedman |
| Relationship between two continuous variables | Pearson correlation | Spearman correlation |
| Two categorical variables | Chi-square test of independence (Fisher’s exact for small counts) | |
| Predicting a continuous outcome | Multiple linear regression | |
| Predicting a binary outcome | Logistic regression | |
This covers the large majority of undergraduate and master’s projects. More complex designs — multilevel data, time series, survival outcomes — need specialist methods and usually a conversation with your supervisor.
Worked example: walking through the decision
Suppose you want to know whether three teaching formats — lecture, flipped and blended — produce different exam scores, and each student experienced only one format. The question is a comparison. The outcome, an exam score, is continuous. There are three groups, and they are independent. If residuals look roughly normal and variances are similar, a one-way ANOVA is appropriate, followed by post-hoc tests to see which formats differ. If the scores are heavily skewed in a small sample, Kruskal-Wallis is the safer choice. Four answers, one decision.
Watch for designs with more than one factor
If you have two grouping variables — say, teaching method and year of study — a two-way ANOVA lets you test each effect and their interaction in one model. Running separate t-tests for every combination is less efficient and inflates error rates. Similarly, if you want to control for a covariate such as prior attainment, ANCOVA or regression is more appropriate than comparing raw group means.
Choose before you collect, where possible
The best time to choose a test is at the design stage, because the test determines the sample size you need. A power analysis tells you how many participants are required to detect an effect of a given size. Choosing a test after collecting data is common in student projects, but it often reveals that the sample is too small for the comparison you wanted to make.
Ethics and proposals
Many ethics committees and proposal templates ask for your planned analysis. Having it settled early makes those documents faster to write and more convincing.
Do not choose a test because it gives significance
Switching from a parametric to a non-parametric test, or the reverse, because one produces p < .05 is a form of selective reporting. Make the choice on the basis of your design and assumptions, and stick with it. If you run a sensitivity analysis with an alternative test, report both results.
Remember that tests answer narrow questions
A test tells you whether the pattern in your sample is unlikely under a specific null hypothesis. It does not tell you why the pattern exists, whether it would hold in a different population, or whether it is large enough to matter. Choosing the correct test is necessary but not sufficient; you still need effect sizes, confidence intervals and a careful interpretation that takes your design and sample into account.
Justify your choice in the methods chapter
A strong justification names the test, links it to the research question and variable types, and notes how assumptions were checked. Two or three sentences per analysis is usually enough.
“Because the outcome was continuous and the two teaching groups were independent, an independent-samples t-test was planned. Normality was assessed using Q-Q plots and Levene’s test indicated unequal variances, so Welch’s correction was applied.”
When to ask for help
If your data does not fit any row of the table, if you have nested or repeated data across many time points, or if your supervisor’s feedback mentions “model specification,” it is time to get specialist advice. Statistics support services, methods textbooks for your discipline and worked examples from published papers in your field are all good starting points.