Troubles with Sub-Group Analysis

Here is an example from Dr Vickers’s book, ‘What is a p-value anyway?’ about issues related to investigators running more analyses hoping to get statistical significance. A well-known type is a sub-group analysis. Note the following data on cancer drugs.

Run a Fisher’s Exact Test, and you get a p-value of 0.02, which is statistically significant that the new drug is more effective.

p-value = 0.02016
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
 0.5904410 0.9576516
sample estimates:
odds ratio 
  0.752434

Now, you do two sub-groups:

Run the test for the first sub-group (men): p-value = 0.12, and for the second (women), the p-value = 0.1; the new drug work for people, but not for men or for women!

Reference

What is a p-value anyway? 34 Stories to Help You Actually Understand Statistics:  Andrew Vickers