Confidence level

The confidence level is a number that acts as a threshold for a "surprising" P-value and as a cut-off for determining when two groups have a difference in proportions.

The P-value is a numeric probability of seeing the difference in proportions under the assumption there isn't a real difference in the proportions between the two groups. With a confidence level of 95%, any P-value less than or equal to 0.05 is considered surprising. A small P-value means it is unlikely to occur if the variations between group proportions was random. If there is a difference because of random chance it would be surprising enough to believe it is wrong. So, it can be concluded there is a true difference in the proportions between two groups.

Example
In an example where the confidence level is set to 95%, that means any P-value that is less than 0.05 (1 - 0.95 = 0.05), is considered "surprising", and we would reject the assumption there is no difference between proportions and instead say there is an actual difference.

The confidence level sets what is the acceptable probability of getting a false positive, when there is no real difference between groups.

Example
Using the same example where the confidence level is 95%, it indicates we can accepting false positives 5% of the time in a situation where there is no real difference between two group proportion.