Regression, ANOVA, and the t-test are related

Source: How are regression, the t-test and ANOVA all versions of the general linear model?

In a new episode of things I keep forgetting: regression, ANOVA, and the \(t\)-test are not three unrelated procedures. With two groups they are different ways to describe the same linear model.

Suppose the response is \(y\) and the group indicator is \(x\). The regression model is

\[ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i. \]

The coefficient \(\beta_0\) is the mean of the reference group and \(\beta_1\) is the difference between the two group means. Testing whether \(\beta_1 = 0\) is therefore the same question asked by a two-sample \(t\)-test. ANOVA asks whether the group explains variation in the response. With only two groups, it is the same question again.

The data, according help(sleep):

Data which show the effect of two soporific drugs (increase in hours of sleep compared to control) on 10 patients — Scheffé, Henry (1959), The Analysis of Variance. New York, NY: Wiley.

Now, load packages and data.

Code

library(tibble)
library(broom)

data("sleep")

sleep <- as_tibble(sleep)

glimpse(sleep)

Rows: 20
Columns: 3
$ extra <dbl> 0.7, -1.6, -0.2, -1.2, -0.1, 3.4, 3.7, 0.8, 0.0, 2.0, 1.9, 0.8, …
$ group <fct> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2
$ ID    <fct> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

Regression

The regression makes the difference between group means explicit as a coefficient.

Code

linear_model <- lm(extra ~ group, data = sleep)

summary(linear_model)


Call:
lm(formula = extra ~ group, data = sleep)

Residuals:
   Min     1Q Median     3Q    Max 
-2.430 -1.305 -0.580  1.455  3.170 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)  
(Intercept)   0.7500     0.6004   1.249   0.2276  
group2        1.5800     0.8491   1.861   0.0792 .
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 1.899 on 18 degrees of freedom
Multiple R-squared:  0.1613,    Adjusted R-squared:  0.1147 
F-statistic: 3.463 on 1 and 18 DF,  p-value: 0.07919

Code

tidy(linear_model)

# A tibble: 2 × 5
  term        estimate std.error statistic p.value
  <chr>          <dbl>     <dbl>     <dbl>   <dbl>
1 (Intercept)     0.75     0.600      1.25  0.228 
2 group2          1.58     0.849      1.86  0.0792

ANOVA

ANOVA partitions the variability into variation between and within groups. With one two-level factor, its \(F\) statistic is the squared regression \(t\) statistic.

Code

anova_model <- aov(extra ~ group, data = sleep)

summary(anova_model)

            Df Sum Sq Mean Sq F value Pr(>F)  
group        1  12.48  12.482   3.463 0.0792 .
Residuals   18  64.89   3.605                 
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Code

tidy(anova_model)

# A tibble: 2 × 6
  term         df sumsq meansq statistic p.value
  <chr>     <dbl> <dbl>  <dbl>     <dbl>   <dbl>
1 group         1  12.5  12.5       3.46  0.0792
2 Residuals    18  64.9   3.60     NA    NA

\(t\)-test

The classical two-sample test below uses equal variances, matching the assumptions of the regression and ANOVA fitted above.

Code

t_test <- t.test(extra ~ group, var.equal = TRUE, data = sleep) 

t_test


    Two Sample t-test

data:  extra by group
t = -1.8608, df = 18, p-value = 0.07919
alternative hypothesis: true difference in means between group 1 and group 2 is not equal to 0
95 percent confidence interval:
 -3.363874  0.203874
sample estimates:
mean in group 1 mean in group 2 
           0.75            2.33

Code

tidy(t_test)

# A tibble: 1 × 10
  estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
     <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
1    -1.58      0.75      2.33     -1.86  0.0792        18    -3.36     0.204
# ℹ 2 more variables: method <chr>, alternative <chr>

The same result

The signs of the two \(t\) statistics can differ because the methods may subtract the group means in opposite directions. Their squares, and more importantly their p-values, are the same.

Code

regression_result <- tidy(linear_model)
regression_result <- regression_result[regression_result$term == "group2", ]

anova_result <- tidy(anova_model)
anova_result <- anova_result[anova_result$term == "group", ]

t_test_result <- tidy(t_test)

comparison <- tibble(
  method = c("Regression", "ANOVA", "Two-sample t-test"),
  statistic = c(
    regression_result$statistic^2,
    anova_result$statistic,
    t_test_result$statistic^2
  ),
  statistic_type = c("t²", "F", "t²"),
  p_value = c(
    regression_result$p.value,
    anova_result$p.value,
    t_test_result$p.value
  )
)

comparison

# A tibble: 3 × 4
  method            statistic statistic_type p_value
  <chr>                 <dbl> <chr>            <dbl>
1 Regression             3.46 t²              0.0792
2 ANOVA                  3.46 F               0.0792
3 Two-sample t-test      3.46 t²              0.0792

This is the useful reminder: choosing among these methods is often about how we want to express the question and extend the model, not about three entirely different statistical ideas.