Some of the hypothesis test pages will end by constructing a confidence interval. This page is aimed at developing an intuition for what confidence intervals represent. We are still working with the box model here: the box is our population, and a confidence interval is how we estimate what is in the box without having to look at every ticket.
So first, what is a confidence interval? A confidence interval is a range of values calculated from sample data that is used to estimate an
unknown population parameter. When we create a confidence interval, we specify a confidence level as a percentage, such as 95%.
This might seem a bit vague, so it is best explained through an example. Let’s consider that we have sample data, and we want to estimate the
population mean. This is a realistic scenario. In real life, the population mean is usually unknown, and we want to use the sample to gauge what
the population mean is likely to be. If we were to calculate a 95% confidence interval using our sample, we would end up with a range of values -
a plausible range for the population mean, produced by a procedure that works 95% of the time.
But what does the 95% mean in this context? It does not mean there is a 95% chance that the true mean falls within this particular interval.
Once the interval is calculated, the true mean either is or is not in the interval - it’s not a matter of probability.
The key is to be clear about what is random here. The population mean is a fixed number. It is unknown to us, but it does not move. What changes
from one study to the next is the sample, and therefore the interval. Every time we take a new sample, we get a new interval sitting in a
slightly different place. Some of those intervals land over the population mean, and some of them miss it.
The 95% describes how often the intervals land well. If we were to repeat this entire process of taking many random samples from the population
and calculating a 95% confidence interval from each one, then about 95% of those intervals would contain the true population mean. The 95% is
therefore a property of the procedure - its long-run success rate - rather than a statement about the one interval sitting in front of us.
We use this idea to quantify uncertainty and to make more informed judgments based on limited sample data. The simulation below makes this
concrete: it generates 100 intervals at once, so we can see for ourselves how many of them capture the population mean.
To illustrate confidence intervals, there is a simulation to the right. Do not worry about the method that has been used to generate the
confidence intervals, but know that they are 95% confidence intervals.
Your Task: Press ‘simulate’ to generate 100 confidence intervals.
Here is a breakdown of what we see in the graph:
What we should see is that around 95% of the confidence intervals are green, meaning they contain the true population mean.
There will be some variability however, especially as we are only looking at 100 intervals here. Press simulate a few times to see how
the intervals change as new data is encountered.
There is also a running total to the right which shows you how many confidence intervals that you have generated so far did NOT contain
the true population mean.
Note: In reality, we would only ever have one sample, and so we would only ever construct one confidence interval - a single line out of the 100 in this graph. We would also have no way of knowing whether our line was a green one or a red one, because we would not know the true population mean. Generating 100 intervals is a teaching device. It lets us step outside the problem and watch how the procedure behaves across many samples, which is exactly what the 95% is describing. It is not something we would ever do with real data.
You might be wondering, what are the different features that can change the size of a confidence interval? There are lots! Toggle the sections below to learn more.
In the example above, the confidence level was 95%. We said that this means if we were to repeat the process of taking many
random samples from the population and calculating a 95% confidence interval, then about 95% of those intervals would contain
the true population mean.
The confidence level is what decides how wide we make each interval. If we lower it to 80%, we are asking for a procedure that
only needs to capture the true population mean about 80% of the time. That is a weaker demand, so each interval does not need to
be as wide, and the 80% intervals come out smaller than the 95% ones.
If we raise it to 99%, we are demanding that about 99% of the intervals contain the true population mean. To succeed that often,
each interval has to cast a wider net. Hence, the 99% confidence intervals will be larger than the 95% ones.
Key point: In general, increasing the confidence level will increase the confidence
interval size.
The whole point of a confidence interval constructed from a sample mean is that we want to have an indication of where the
population mean is. This is often desired in statistics because it is costly to collect data from everyone in a population.
The more data we collect, the more reliable our sample mean becomes. For example, consider that we wanted to know whether people
preferred Coke or Pepsi. If we only interviewed 10 people, our result would be fragile - a couple of unusual answers would swing
the sample mean noticeably, and interviewing a different 10 people could easily give us quite a different answer. If we
interviewed 1000 people, no single unusual answer counts for much, because it is averaged out against many ordinary ones. The
sample mean settles down, and varies far less from one sample to the next.
Because a larger sample gives us a more stable sample mean, we do not need to hedge as much when we build an interval around it.
There is less uncertainty about where the population mean is, and hence we would expect the confidence interval to be
smaller.
One thing worth being clear about: what matters here is the size of the sample itself, not the fraction of the population that it
covers. A sample of 1000 people tells us roughly as much about a city of 100,000 as it does about a country of 25 million. This is
why national polls can work with only a couple of thousand respondents.
Key point: In general, increasing the sample size will decrease the confidence interval
size.
We will not focus on this too much here, but different hypothesis test types will have different-sized confidence intervals. For example, a confidence interval for a t-test will typically be larger than that of a z-test (because t-curves have fatter tails).
Your Task: For a given normal distribution (we are taking samples from the specified distribution):
Note: Here, each confidence interval is taken from sampling from the specified normal distribution. This is for learning purposes only. In reality, you would not know the true population mean or population standard deviation (otherwise, what would be the point of constructing an interval in the first place).