Statistical significance is a value, called p-value, which indicates the probability that a given result will occur, provided that a certain statement (called the null hypothesis) is true. If the p-value is small enough, the experimenter can safely say that the null hypothesis is false.
Steps
Step 1. Determine the experiment you want to perform and the data you want to know
In this example, we will assume that you have purchased a wooden board from a lumber yard. The seller claims that the board is 8 feet in size (let's denote this as L = 8). You think the seller is cheating, and you believe the length of the wooden board is actually less than 8 feet (L <8). This is what is called an alternative hypothesis H.TO.
Step 2. State your null hypothesis
In order to prove that L = 8, given the data we have collected. Therefore, we will state that our null hypothesis states that the length of the wooden plank is greater than or equal to 8 feet, or H0: L> = 8.
Step 3. Determine how unusual your data must be before it is considered significant
Many statesmen believe that a 95% certainty that the null hypothesis is false is a minimum requirement for obtaining statistical significance (given a p-value of 0.05). This is the level of significance. A higher level of significance (and therefore a lower p-value) indicates that the results are even more significant. Note that a 95% significance level means that 1 in 20 times you conduct the experiment is wrong.
Step 4. Collect the data
Most of us who would use the tape measure would find that the length of the board is less than 8 feet, and would ask the dealer for a new wooden board. However, science requires far more significant proof than a single measurement. Since the manufacturing process is imperfect, and even if the average length was 8 feet, most of the boards are slightly longer or shorter than that length. To deal with this, we need to make several measurements and use those results to determine our p-value.
Step 5. Calculate the average of your data
We will denote this mean with μ.
- Add up all your measurements.
-
Divide by the number of measurements taken (n).
Assess Statistical Significance Step 6 Step 6. Calculate the standard deviation of the sample
We will denote the standard deviation with s.
- Subtract the mean μ from all your measurements.
- Square the resulting values.
- Add the values.
- Divide by n-1.
-
Calculate the square root of the result.
Assess Statistical Significance Step 7 Step 7. Convert your mean to a standard normal value (Z result)
We will denote this value with Z.
- Subtract the H value0 (8) from your mean μ.
-
Divide the result by the sample standard deviation s.
Assess Statistical Significance Step 8 Step 8. Compare this Z value to the Z value of your significance level
This comes from a standard distribution table. Determining this fundamental value is beyond the intent of this article, but if your Z is less than -1.645, then you can assume that the board is less than 8 feet in length and a significance level greater than 95%. This is called "denial of the null hypothesis", and it means that the calculated μ is statistically significant (since it is different from the declared length). If your Z value is not less than -1,645, you can't reject H.0. In this case, note that you have not proved that H.0 it's true. You simply don't have enough information to say it's false.
Assess Statistical Significance Step 9 Step 9. Consider a further case study
Doing another study with further measurements or with a more accurate measurement tool will help increase the significance level of your conclusion.
Advice
Statistics is a vast and complex field of study; take an advanced undergraduate (or higher) statistical inference course to improve your understanding of statistical significance
Warnings
- This analysis is specific to the example given, and will vary based on your hypothesis.
- We have developed a number of hypotheses that have not been discussed. A statistics course will help you understand them.
