Significance test

A significance test helps you assess if differences observed between subgroups in your sample are statistically significant, or if they're simply caused by chance or variations in the sample.

For example, your report data shows participants in the 25-34 age group react more positively to a new website design than the 35-44 age group. But is the difference significant enough that you need to pay attention to it? Applying a significance test to your data will help you answer this question.

Significance test workflow for Analytics

  1. Add nested columns to a crosstab report
  2. Apply a significance test
  3. Examine the significance test results

Significance test limitations

As a best practice, before you apply a significance test to your report, consider the following:
  • Subgroups based on Multiple Choice questions cannot be used for significance tests. T-tests cannot be calculated for overlapping subgroups and creating subgroup definitions with Multiple Choice questions may result in overlapping subgroups.
  • Subgroups that have a sample size of less than 30 are excluded from significance tests. Significance test results may be unreliable when the sample sizes are small. To ensure the reliability of the significance tests, we recommend that each subgroup contains a sample size that is greater than 100.
  • Answer values that have 0% or 100% of responses are excluded from significance tests. These answer values make comparing across all answers unfeasible. For example, if you are comparing responses from Canadian provinces and have 0% responses from Saskatchewan, you cannot compare Saskatchewan to the other provinces. Although you can still compare the remaining provinces to each other, we recommend that you wait for additional responses before running significance tests.