Here is an example from Dr Vickers’s book, ‘What is a p-value anyway?’ about issues related to investigators running more analyses hoping to get statistical significance. A well-known type is a sub-group analysis. Note the following data on cancer drugs.
New.Drug | Old.Drug | |
Recurred | 150 | 190 |
Cancer free | 850 | 810 |
Run a Fisher’s Exact Test, and you get a p-value of 0.02, which is statistically significant that the new drug is more effective.
p-value = 0.02016
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
0.5904410 0.9576516
sample estimates:
odds ratio
0.752434
Now, you do two sub-groups:
MEN | New.Drug | Old.Drug |
Recurred | 80 | 100 |
Cancer free | 420 | 400 |
WOMEN | New.Drug | Old.Drug |
Recurred | 70 | 90 |
Cancer free | 430 | 410 |
Run the test for the first sub-group (men): p-value = 0.12, and for the second (women), the p-value = 0.1; the new drug work for people, but not for men or for women!
Reference
What is a p-value anyway? 34 Stories to Help You Actually Understand Statistics: Andrew Vickers