Decision Making

In the previous ANOVA exercises, we found that data suggested rejecting the null hypothesis. To remind you of the two hypotheses,
N₀ – All means are equal
N_A – Not all means are equal
So, we at least rejected the proposition that all means are equal because the p-value was lower than the chosen significance level of 0.05 (or the F value was outside the critical F value corresponding to the 0.05 level). But we have no idea which of the pairs of means had the most significant difference.

Tukey method can create confidence intervals for all pair-wise differences while controlling the family error rate to whatever we specify.

Family error rate

You know what is an error rate. It is the probability that the null hypothesis is correct when you reject it when the p-value is less than the significance level. At a significance level is 0.05, there is a 5% chance of getting your outcome when the null hypothesis is correct. The situation is called a false positive.

The p-value we obtained for the material testing problem was 0.03, but it was for the entire family of four vendor groups (each with ten samples). This is the experiment-wise or family-wise error rate. Since our significance level for the F-test was kept at 0.05, we can regard the family error rate to be 5%.

Four groups, six comparisons

Since we had four groups (factors) of samples, each representing one vendor, we have six possible comparisons. They are:

Family-wise error rate, 0.05, is the grand union of all pair-wise error rates. If the pair-wise error is alpha, family-wise error = (1 – (1-alpha)^C), where C is the number of comparisons. If you substitute alpha = 0.05 and C = 4, you get the family-wise error as 0.26. Obviously, 26% is too high a significance level.

Tukey method preserves the family-wise error rate to what we specify, say, 0.05, and therefore the pair-wise error rates could be about 0.0085.

By keeping all these points in mind, let’s perform the Tukey’s method on our dataset using R.

TukeyHSD(res.aov)

Which leads to the following output:

Tukey multiple comparisons of means
    95% family-wise confidence level

Fit: aov(formula = Strength ~ factor(Sample), data = AN_data)

$`factor(Sample)`
                         diff        lwr       upr     p adj
Vendor 2-Vendor 1 -2.26479763 -4.7917948 0.2621995 0.0924842
Vendor 3-Vendor 1 -0.51997359 -3.0469707 2.0070236 0.9448076
Vendor 4-Vendor 1 -2.36456760 -4.8915647 0.1624295 0.0736423
Vendor 3-Vendor 2  1.74482404 -0.7821731 4.2718212 0.2632257
Vendor 4-Vendor 2 -0.09976996 -2.6267671 2.4272272 0.9995613
Vendor 4-Vendor 3 -1.84459400 -4.3715911 0.6824031 0.2197059

In the next post, we will do a complete exercise of ANOVA including the Post Hoc test.

Hypothesis Testing: An Intuitive Guide: Jim Frost

We have seen how F-statistics work to test the hypotheses in one-way ANOVA. We also know the definition of F as a ratio between two variances. Variances measure how spread the data is around the mean, estimated as the sum of squared deviation divided by the degrees of freedom. If you forgot, you get the standard deviation if you take the square root of the variance.

F = Between groups variance / Within-group variance

F-tests use F-distribution

Recall how we used the t-distribution or binomial distribution to determine the probability where the null hypothesis was true. F-distribution, too, has a characteristic shape and is based on two parameters – the degrees of freedom 1 and 2, the ones used in the numerator and denominator, respectively.

In the case of the material strength problem we have been working out in the past two posts (four groups with ten samples each leading to df1 =4-1 = 3 and df2 = 4 x (10-1) = 36), the F-distribution appear in the following form.

One way to understand the above plot is to imagine you are repeating the sampling several times (keeping for vendors and taking ten samples each so that df1 and df2 remain the same), and the null hypothesis is true. You calculate the F values each time. Finally, if you plot the frequency of those F values, you get a plot similar to the one above.

Let’s do the ANOVA step by step. We use the F-statistic to accept or reject the null hypothesis by comparing it with the critical F value. Once you get the F-value, you can calculate the p-value based on a significance level.
The definition of F-statistic is

F = Between groups variance / Within-group variance

Between groups variance

Here, you are estimating the variation of the group statistic from the global statistic. In other words, you determine the means of each group and the global mean (of all data or the mean of means). The estimate the difference, square, add up and divide by the degree of freedom like you do standard variance.

Recall the previous example (strength of materials by four vendors). So you have four groups, each containing ten samples. First, estimate four means and the global mean. They are:

The numerator (mean squares of factor) is calculated by dividing the sum square of factor with the degrees of freedom, i.e., 43.62/3 = 14.54.

Within-group variance

Here, you add up all the variations inside the groups. Add them up and then divide by the sum of the degrees of freedom of each group.

The denominator (mean squares of error) is calculated by dividing the sum within group squares for error with the total degrees of freedom, i.e., 158.466/36 = 4.402.

F – Statistics = 14.54 / 4.402 = 3.30

The 3.30 is then compared with the critical F-value corresponding to a set significance level, 0.05, in the present case. You can either look up at the F distribution table or use the R function.

qf(0.05, 3,36, lower.tail=FALSE)

The critical value is 2.87. Since the F-statistics in our case is larger than 2.87, we reject the null hypothesis. The p-value turned out to be 0.031.

pf(3.303, 3, 36, lower.tail = FALSE)

We know how to use a 1-sample t-test to perform a hypothesis test on the mean of a single group and a 2-sample t-test to compare two groups. The scope of t-tests ends here as two is the limit. What happens when you have three groups of data? If the number is two or more, you will use the analysis of variance or ANOVA. As we have done before, we will do an ANOVA using R programming.

Comparing four vendors

ANOVA requires an independent variable (categorical factor) and a dependent variable (continuous). The following table will tell you what I meant. The data used in the analysis is taken from https://statisticsbyjim.com/. The strength of certain materials from four vendors is available, and we are determining if there is a statistically significant difference between the mean strengths. Here is how a few selected entries appear (out of 40).

Note that the categorical factor, vendor names (1, 2, 3 and 4), divided the continuous data into four groups. Before performing ANOVA, we plot the data and check how they look.

par(bg = "antiquewhite")
boxplot(AN_data$Strength ~ factor(AN_data$Vendor), xlab = "Vendor", ylab = "Strength of Material")

The null and alternate hypotheses are:

N₀ = four mean strengths are equal
N_A = four mean strengths are not equal

Doing ANOVA is pretty easy; the following commands will do the job.

sum_aov <- aov(AN_data$Strength ~ factor(AN_data$Vendor))
summary(sum_aov)

                      Df Sum Sq Mean Sq F value Pr(>F)  
factor(AN_data$Vendor)  3  43.62  14.540   3.303 0.0311 *
Residuals              36 158.47   4.402                 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

We are testing the statistical significance by the F-test. And here is the most important thing, the p-value is 0.03, less than the 0.05 we chose. So the null hypothesis is rejected. We will see what they all mean, in the next post.

Hypothesis Testing: An Intuitive Guide: Jim Frost