statistical tools for data analysis true value

Typically, multiple comparison procedures control for the family-wise error rate (FWER) instead, which is the probability of having one or more false positives out of all the hypothesis tests conducted. This method is too liberal, since we do not want to have such a great number of false positives. For example, if we were doing a genomewide study looking at differential gene expression between tumor tissue and healthy tissue, and we tested 1000 genes and controlled the FPR, on average 50 truly null genes will be called significant. This method poses a problem when we are conducting a large number of hypothesis tests. So if we control the FPR at an alpha of 0.05, we guarantee than the percentage of false positives (null features called significant) out of all hypothesis tests is 5% or less. The false positive rate (FPR), or per comparison error rate (PCER), is the expected number of false positives out of all hypothesis tests conducted. The more features you have, the higher the chances of a null feature being called significant. When we are conducting multiple comparisons (I will call each test a “feature”), we have an increased probability of false positives. In other words, we want the probability of a type I error, or a false positive, to be less than 5%. When we set our alpha to 0.05, we are saying that we want the probability that a null finding will be called significant to be less than 5%.

We usually like to keep this probability under 5%. Since this is a small probability, we reject the null hypothesis and say that the means are significantly different. If we had a p-value of 0.03, for example, that would mean that if our null hypothesis is true, there would be a 3% chance of obtaining our observed test statistic or a more extreme. When conducting hypothesis tests, for example to see whether two means are significantly different, we calculate a p-value, which is the probability of obtaining a test statistic that is as or more extreme than the observed one, assuming the null hypothesis is true. In order to be able to identify as many significant comparisons as possible while still maintaining a low false positive rate, the False Discovery Rate (FDR) and its analog the q-value are utilized.

Use of the traditional Bonferroni method to correct for multiple comparisons is too conservative, since guarding against the occurrence of false positives will lead to many missed findings. When analyzing results from genomewide studies, often thousands of hypothesis tests are conducted simultaneously. This page briefly describes the False Discovery Rate (FDR) and provides an annotated resource list.

Statistical tools for data analysis true value