INDEPENDENT RESEARCH & ANALYTICS PARTNERResearch with purpose. Evidence for impact.
← Back to Publications
Statistical practice

Seven Statistical Mistakes Researchers Make—and How to Avoid Them

From causal overclaims to Simpson’s paradox, learn how common analytical errors can change a study’s conclusions.

6–7 min read

A researcher examining statistical charts on a computer

Statistical software makes it easier than ever to run an analysis. The harder task is deciding whether the analysis answers the research question and whether the result has been interpreted fairly. Many problems in research do not come from a complicated formula; they come from a small decision made too quickly. Here are seven common mistakes and practical ways to avoid them.

1. Treating association as causation

If two variables move together, it does not automatically mean that one causes the other. For example, employees reporting low support may also report high stress, but the relationship could reflect several pathways: low support may increase stress, stress may affect how support is perceived, or workload may influence both. Cross-sectional data are especially limited for establishing temporal order.

How to avoid it: Use language that matches the design. In an observational study, write “was associated with” rather than “caused” unless the design and assumptions support a causal claim. Explain plausible confounding, timing and alternative explanations. A regression coefficient by itself does not turn an observational association into a causal effect.

A checklist for reviewing statistical analysis
A checklist for reviewing statistical analysis

2. Treating normality as a box that every variable must pass

Researchers sometimes run a normality test on every variable and assume that a significant result means the data cannot be analysed. Many statistical methods do not require every measured variable to be normally distributed. In ordinary linear regression, for example, assumptions concern the model errors and the adequacy of the conditional mean structure, not a requirement that every predictor be normal. With large samples, formal normality tests may flag small, unimportant departures; with small samples, they may fail to detect meaningful departures.

How to avoid it: Check the assumptions of the specific method you plan to use. Inspect histograms, Q–Q plots and residual plots alongside substantive knowledge. Consider robust standard errors, transformations, bootstrap intervals or an alternative model when justified, rather than choosing a method solely because a test returned a particular p-value.

3. Ignoring model assumptions and data structure

A model may produce output even when its assumptions are badly violated. Examples include treating repeated observations from the same participant as independent, ignoring clustering within schools or clinics, assuming a straight-line relationship when the pattern is curved, or using a model poorly suited to a binary or count outcome. Standard errors and confidence intervals can be misleading when the data structure is ignored.

How to avoid it: Match the model to the outcome and sampling design. For clustered or repeated data, consider multilevel, mixed-effects, generalised estimating equation or other appropriate methods. Examine residuals and influential observations, check functional form, and document any model changes. Do not rely on a single diagnostic number.

4. Reporting only whether p is below .05

A p-value does not tell you the probability that your hypothesis is true, nor does a value just below .05 automatically mean a result is important. A tiny effect can be statistically significant in a large sample, while an important but imprecisely estimated effect may not cross a conventional threshold. Focusing only on significance can hide the size and uncertainty of the finding.

How to avoid it: Report effect sizes and confidence intervals alongside p-values. Explain whether the estimated difference is meaningful in the real setting, not just whether it meets a threshold. Be transparent about the number of tests performed and avoid presenting exploratory findings as if they were planned in advance.

5. Ignoring missing data

Deleting every record with any missing value may reduce sample size and introduce bias, particularly when missingness is related to the outcome or participant characteristics. But replacing all missing values with a mean is not a reliable universal fix either; it can distort variability and relationships.

How to avoid it: Describe how much data are missing, which variables are affected and whether missingness differs across groups. Think carefully about why data may be missing. Choose a method appropriate to the setting—such as multiple imputation or a likelihood-based approach when its assumptions are defensible—and include sensitivity checks where possible. Explain exclusions and report the final analytic sample clearly.

6. Overlooking subgroups and Simpson’s paradox

Sometimes a pattern observed in the combined data changes or reverses when the data are separated into meaningful groups. This is often called Simpson’s paradox. Imagine that one treatment appears more successful overall, but the people receiving it were mostly in an easier-to-treat group. Within each group, the comparison might tell a different story. The aggregate result is not necessarily wrong; it may answer a different question because the groups have different sizes or baseline risks.

How to avoid it: Think about important grouping variables before analysing the data. Show relevant stratified results when justified, explain why groups differ, and use an appropriate adjusted model if it matches the question. Do not split data into many subgroups simply until a preferred result appears. Subgroup findings should be interpreted cautiously, especially when sample sizes are small.

7. Confusing a statistically tidy result with a trustworthy study

Even a correctly fitted model cannot rescue poor measurement, a biased sample or a research question that changes after the results are seen. Testing many outcomes and reporting only the favourable ones can create a misleading picture. So can quietly changing exclusion rules or dropping inconvenient observations without a defensible reason.

How to avoid it: Plan primary outcomes and main analyses in advance when possible. Keep a record of data-cleaning decisions, exclusions and model specifications. Distinguish confirmatory analyses from exploratory work, and report limitations honestly. Reproducible scripts and independent checks help reveal errors before results are shared.

The habit that prevents many mistakes

Before opening statistical software, write down the question, the outcome, the key predictors, the unit of analysis and what evidence would support the conclusion. Then check whether the data and design can genuinely provide that evidence. Statistical analysis is not simply a search for a significant result; it is a disciplined way of reducing uncertainty. A careful, transparent analysis is more valuable than a complex model whose assumptions and limitations are unclear.

← Return to all publications