Statistical Significance: What exactly does it mean?

What does it mean to be statistically significant?

Significance, as it applies to statistics, is often misinterpreted or misunderstood. Unlike the traditional meaning of significance which implies that something is important, statistical significance suggests that a relationship between variables is not due to random chance or dumb luck. A statistically significant finding may be important or unimportant. However, it simply means that we assume that the relationship between variables actually exists in reality, and is not happening due to chance or error.

For example, let's say that a team of researchers from ABC University release a report from their recent study which suggests that men are significantly more likely than women to suffer from depression. The statistically significant relationship, as observed in this study, implies that the results did not happen due to random chance or error. In other words, this study argues that men are more likely than women to suffer from depression in general, not just within this particular study.

How is statistical significance determined?

Testing for statistical significance begins with a null hypothesis (i.e., we assume there is no relationship between the variables being tested). Researchers then use statistical tools to determine a p-value or the percent likelihood that a result happened by chance. Typically, the standard cut-off point for a p-value in social science research is .05. In other words, p-values lower than .05 (.04, .01, .001) would imply that the relationship between variables is statistically significant. A p-value of .05 assumes that there is a 5% chance that the relationship between variables is due to chance. A p-value of .01 would imply that there is a 1% chance that the relationship occurred by chance. In either of these cases, we would reject the null hypothesis and conclude that there is a statistically significant relationship between the variables. Furthermore, we believe this relationship is not being caused by chance or error.

P-values larger than .05 (.051, .06, .1) would not be considered statistically significant since the chances that the results happened by chance is larger. For instance, a p-value of .1 assumes that there is a 10% chance that the relationship between variables is happening by chance. In this case we would accept the null hypothesis and conclude that the relationship between the variables being tested is not statistically significant.

Let's use another example. Imagine that we conduct a study to find out if men or women have higher IQ scores. Our null hypothesis would be to assume that there is no relationship between gender and IQ scores. In other words, we expect that there are no differences between men and women and their overall IQ scores. Upon analyzing our data, however, we discover that the average IQ for men is 100, and the average for women is 106. Since our results suggest that women are more likely to have a higher IQ score, we must rely on our p-value to tell us whether or not these results happened by chance or luck.

After doing our statistical calculation, we find that our p-value is equal to .061. Since .061 is above our threshold of .05, we conclude that the relationship between men and women and their IQ scores is not statistically significant. Even though women scored higher than men, our p-value tells us that the difference between scores may be the result of chance or luck. In this case, we would accept the null hypothesis and say that there is no relationship between gender and IQ scores. 

How should significance be interpreted?

Just as correlation does not imply causation, statistical significance does not mean that the relationship we are observing will happen 100% of the time. When interpreting significance, it's important for the reader to examine other aspects of the research such as how large or how small the sample is, the type of sample collected (i.e., Is it a random sample or something else?), and how the questions are asked or presented. Just because a study yields significant results does not guarantee that the research was designed and executed carefully. There's plenty of research out there that can be misleading or interpreted incorrectly.