Learn — Inferential statistics & hypothesis testing

6 notes · 14 questions on the topic

Hypothesis testing — the p-value, alpha and statistical significance

Hypothesis testing is the procedure of inferential statistics by which a result observed in a sample is used to decide whether an effect or relationship exists in the population it came from, or whether the result could plausibly have arisen by chance. It tests the null hypothesis of no difference: the probability of obtaining the observed result, or one more extreme, if the null hypothesis were true is the p-value, and when that probability is smaller than the significance level set in advance the null hypothesis is rejected.

The steps run in order: state the null and the alternative hypotheses; set the significance level alpha before seeing the data — 0.05 by convention, 0.01 where a false positive would be costly — and decide on a one- or two-tailed test; choose the test from the level of measurement, the number of groups and whether they are independent or paired; compute the test statistic (t, chi-square, F, r); obtain the p-value or compare the statistic with the critical value; reject the null hypothesis if p is at or below alpha, otherwise fail to reject it; and state the conclusion in words.
The p-value is the probability of obtaining a result at least as extreme as the one observed if the null hypothesis is true — it is not the probability that the null hypothesis is true and not the probability that the result arose by chance in general; p = 0.03 with alpha at 0.05 means the difference is statistically significant and the null hypothesis is rejected, p = 0.08 means it is not rejected, and 'not significant' does not mean 'no difference'.
Statistical significance is not clinical importance: a very large sample makes a trivial difference significant and a small sample leaves an important difference non-significant, so the effect size — the difference in standard-deviation units (Cohen's d), the relative risk, the correlation — and the confidence interval are reported with p to say how big the effect is and how precisely it is known.
A directional hypothesis takes a one-tailed test, with the whole 5 per cent in one tail, and a non-directional hypothesis a two-tailed test with 2.5 per cent in each; the two-tailed test is the default and the conservative choice.
The level of significance is fixed before the data are seen, each test has its degrees of freedom and its critical value, testing many hypotheses on one data set inflates the chance of a false positive (the Bonferroni correction divides alpha by the number of tests), and a confidence interval that excludes the null value — zero for a difference, one for a ratio — is significant at the matching level.
A result is reported with the test, the statistic, the degrees of freedom, the exact p-value, the effect size and the confidence interval — t(58) = 2.31, p = 0.025, d = 0.6, 95 per cent CI 0.4 to 5.6 — so that the reader can judge both whether the effect is real and whether it matters.

Type I and Type II errors and the power of a test

A hypothesis test can be wrong in two ways: a Type I error rejects a null hypothesis that is actually true — finding an effect that does not exist — and a Type II error fails to reject a null hypothesis that is actually false — missing an effect that does exist. The probability of the first is alpha, the significance level the researcher sets; the probability of the second is beta, and one minus beta is the power of the test, the probability of detecting an effect that is truly there.

A Type I error (alpha, the false positive) is rejecting a true null hypothesis — concluding that the new dressing heals faster when it does not; its probability is the significance level, so alpha at 0.05 accepts a 5 per cent risk of it, and making alpha stricter (0.01) lowers that risk at the price of more Type II errors.
A Type II error (beta, the false negative) is failing to reject a false null hypothesis — concluding that there is no difference when the dressing really does heal wounds faster; its probability is conventionally held at 0.20, and it rises with a small sample, a small true effect, high variability in the outcome and a very strict alpha.
Power, one minus beta, is the probability that a study will detect an effect that truly exists, and 0.80 is the conventional minimum; a power analysis before the study fixes the sample size from the expected effect size, alpha and the power wanted, and a study that lacked the power to find a difference has shown nothing by failing to find one.
The two errors trade against each other for a fixed sample — lowering alpha raises beta — and only a larger sample, a larger true effect, less variability, a one-tailed test or a more efficient design such as pairing lowers both at once.
The four outcomes form a table of truth against decision: a true null not rejected is a correct decision; a true null rejected is a Type I error with probability alpha; a false null not rejected is a Type II error with probability beta; a false null rejected is the correct detection whose probability is the power; the courtroom parallel convicts the innocent (Type I) or acquits the guilty (Type II), and the screening parallel is the false positive and the false negative.
Which error matters more depends on its consequence — adopting a useless and costly treatment (Type I) against discarding a useful one (Type II) — and the conventional pairing of alpha at 0.05 with beta at 0.20 treats a false claim as four times worse than a missed effect.

Parametric tests — t-test, ANOVA and choosing the right test

Parametric tests are the tests of statistical inference that assume the data come from a population with a known distribution — normally distributed, at the interval or ratio level, with roughly equal variances between groups — and that use its parameters, the mean and the standard deviation; they are more powerful than their non-parametric counterparts when the assumptions hold. The t-test compares two means and analysis of variance compares three or more, and choosing among them is a matter of three questions: the level of the outcome, the number of groups, and whether the groups are independent or paired.

The independent-samples (unpaired) t-test compares the means of two separate groups on a continuous outcome — the mean pain score of patients given a new dressing against those given the standard one; it assumes normality, equal variances (checked by Levene's test, with Welch's correction when they differ) and independence, and its degrees of freedom are the two sample sizes added together minus two.
The paired (dependent) t-test compares two means from the same people or from matched pairs — anxiety before and after teaching in the same patients — by analysing the differences, with n − 1 degrees of freedom; it is more powerful because the variation between people is removed, and the one-sample t-test compares one mean with a known value.
Analysis of variance, the F-test, compares the means of three or more groups at once — pain scores under three analgesic protocols — as one-way for one factor, two-way for two factors and their interaction, and repeated-measures for the same people measured several times; a significant F says only that at least one mean differs, so a post-hoc test (Tukey, Bonferroni, Scheffé) finds which pair, and running several t-tests instead inflates the Type I error; analysis of covariance adjusts for a covariate.
Pearson's correlation coefficient tests the linear relationship between two continuous variables and linear regression predicts one from the other, and both belong to the parametric family with the same assumptions of normality and interval or ratio data.
The choice grid pairs each parametric test with its non-parametric counterpart: two independent groups — unpaired t-test or Mann–Whitney U; two related measures — paired t-test or Wilcoxon signed-rank; three or more independent groups — one-way ANOVA or Kruskal–Wallis; three or more related measures — repeated-measures ANOVA or Friedman; two continuous variables — Pearson's r or Spearman's rho; two nominal variables — chi-square, or Fisher's exact test when cells are small.
Normality is checked with a histogram or the Shapiro–Wilk test and the variances with Levene's test; skewed data, an ordinal outcome or a very small sample call for the non-parametric equivalent or a transformation, while with about thirty or more per group the parametric tests stand up to mild departures because of the central limit theorem.

Non-parametric tests — chi-square, Mann–Whitney and Wilcoxon

Non-parametric (distribution-free) tests make no assumption that the data follow a normal distribution and work on ranks or counts rather than on means, so they are the tests for nominal and ordinal data, for small samples and for continuous data that are badly skewed. Each parametric test has a non-parametric counterpart, and the chi-square test — the test of association between two categorical variables — is the one examinations ask most.

The chi-square test of independence cross-tabulates two categorical variables in a contingency table — the proportion of patients who develop a surgical-site infection in two groups makes a 2 × 2 table — and compares the observed with the expected frequencies, chi-square being the sum of (observed − expected)² ÷ expected, with degrees of freedom (rows − 1) × (columns − 1); it needs counts rather than percentages, independent observations and an expected frequency of at least five in every cell, failing which Fisher's exact test is used (or Yates' correction for a 2 × 2 table); the goodness-of-fit chi-square compares one variable with an expected distribution and McNemar's test handles paired nominal data.
The Mann–Whitney U test (the Wilcoxon rank-sum test) compares two independent groups on an ordinal or non-normal continuous outcome — pain graded mild, moderate or severe in two wards — by comparing the ranks, and it is the counterpart of the unpaired t-test.
The Wilcoxon signed-rank test compares two related measures — before and after in the same people — on ordinal or non-normal data and is the counterpart of the paired t-test; the sign test is its cruder form, using only the direction of each change.
The Kruskal–Wallis H test compares three or more independent groups (the counterpart of one-way ANOVA), the Friedman test three or more related measures (repeated-measures ANOVA), Spearman's rho correlates ranks (Pearson's r), and the Kolmogorov–Smirnov test compares distributions and checks normality.
Their advantages are few assumptions, use with ordinal data and with small or skewed samples, and simplicity; their disadvantages are less power than the parametric test when its assumptions hold — ranking discards information — fewer measures of effect size and awkwardness with complex designs.
The rule of choice is that a nominal outcome takes chi-square (or Fisher's or McNemar's), an ordinal or skewed outcome takes the rank test that matches the design, and the parametric test is used whenever its assumptions are met.

Pearson r, Spearman rho and linear regression

A correlation coefficient is a single number between −1 and +1 that describes the strength and the direction of the linear relationship between two variables, and regression is the equation that predicts one variable from the other once such a relationship exists. Pearson's r is the coefficient for two interval or ratio variables that are roughly normally distributed, Spearman's rho is its rank-order counterpart for ordinal or non-normal data, and neither of them proves that one variable causes the other.

The sign gives the direction — positive when the variables rise together, as height and weight do, negative when one rises as the other falls, as hours of sleep and pain score do, so a coefficient of −0.85 is a strong negative relationship — and a coefficient of zero means no linear relationship, though a curved one may still exist.
The size gives the strength whatever the sign — by the usual convention up to 0.25 is little or none, 0.26 to 0.50 weak, 0.51 to 0.75 moderate and above 0.75 strong — the coefficient is unit-free and unchanged by a change of scale, and its square, the coefficient of determination, is the share of the variance in one variable that the other explains: r = 0.70 explains 49 per cent.
Pearson's product-moment coefficient needs two continuous, roughly normal variables in a linear relationship and is pulled by outliers; Spearman's rank coefficient correlates the ranks and serves ordinal data, small or skewed samples and a monotonic but non-linear relationship; Kendall's tau is the other rank coefficient, the point-biserial coefficient pairs a dichotomous with a continuous variable and phi pairs two dichotomous ones.
A scatter diagram is drawn first — the slope of the cloud shows the direction, its tightness the strength and its shape whether the relationship is linear — and a significance test of r asks whether the coefficient differs from zero in the population, so a small r can be significant in a large sample and a large r non-significant in a tiny one.
Correlation is not causation: a third, confounding variable may drive both, the direction may run either way and the sample may be restricted, only an experiment shows cause, and the textbook warning is the spurious correlation between ice-cream sales and drownings.
Simple linear regression fits the line y = a + bx by least squares — a the intercept, b the slope or regression coefficient, the change in y for a unit change in x — to predict the dependent variable from the independent; multiple regression uses several predictors, logistic regression predicts a dichotomous outcome and gives odds ratios, and R² measures the fit.

Standard error and the confidence limits of an estimate

Estimation is the other half of inferential statistics: instead of testing whether a difference exists, it asks how large the true population value probably is, and answers with a point estimate — the sample mean or proportion — and an interval around it within which the population value lies with a stated probability, the confidence interval. The width of that interval comes from the standard error, which measures how much sample estimates would vary from sample to sample, and the two together say how precise a study's answer is.

The standard error of the mean is the standard deviation divided by the square root of the sample size — the standard deviation of the means of all possible samples of that size — and it falls as the sample grows, so that quadrupling the sample halves it, while the standard deviation of the individuals does not change; the standard error of a proportion is the square root of pq ÷ n.
The 95 per cent confidence interval of a mean is the mean plus or minus 1.96 standard errors (the t-value replaces 1.96 for small samples) and the 99 per cent interval uses 2.58, so a mean haemoglobin of 11 g/dL with a standard error of 0.5 has 95 per cent confidence limits of about 10 and 12 g/dL — the limits being the two ends of the interval.
The interval is read as follows: if the study were repeated many times, 95 per cent of the intervals so calculated would contain the true population value; a wide interval means low precision — a small sample or a large standard deviation — a narrow one high precision, and a 99 per cent interval is always wider than a 95 per cent one.
The interval and the test agree: a 95 per cent interval for a difference that excludes zero, or for a ratio such as a relative risk or an odds ratio that excludes one, corresponds to p below 0.05, an interval that includes the null value means a non-significant result, and the interval shows the whole range of effects compatible with the data, which the p-value cannot.
The standard error and the interval serve to report the precision of a prevalence, a mean, a difference between means or a relative risk, to calculate a sample size from the precision wanted, and to draw the error bars on a graph, which must say whether they show the standard deviation, the standard error or the confidence interval.
Behind both stands the central limit theorem: whatever the shape of the population, the sampling distribution of the mean is approximately normal once the sample is about thirty, with the population mean at its centre and the standard error as its spread, which is why the normal-curve multipliers work.

Free today: the 10-question daily quiz · the NORCET 11 recall paper · what ₹999 unlocks — once, no auto-renewal.