A Guide to Confidence Intervals

When you look at a data point, you might wonder how confident you can be in the information presented. There’s always some uncertainty in estimates, but an important concept that can help clarify the extent of uncertainty in data is precision.

• Precision is about how much a data point changes when the measurement is repeated. This change can be due to natural variation (sometimes called “random error”) in the sample of the population that is being collected (see more about using samples). Data points are of course most useful when they are precise. For instance, if you calculate a metric about a population based on a sample, but every time you take a different sample you get a slightly different number, it helps to have some guidance about how much you can rely on the results.

This is where confidence intervals come in: they help us interpret estimates and understand how precise they are. Confidence intervals use information on random error to explain the range of possible values in the population.

What Are Confidence Intervals?

For every data point, confidence intervals provide two additional numbers that mark a lower value and an upper value, between which the real number likely falls. Because we can never be 100% certain when we take a measurement, confidence intervals are commonly presented at levels of 90%, 95%, or 99%. For example, for an estimated data point of 40%, the 90% confidence interval might range from 34% to 46% (typically written as (34%,46%).

This means that if we repeated the measurement using a different sample or distribution of events such as death, then 90% of the time the value would fall between 34%-46%. This quantifies the level of confidence (e.g., 90%) we have that the true value falls in that range. A more narrow interval means the estimate is more precise. A wider interval means it is less precise.

There are different ways to calculate confidence intervals, depending on how a measure is defined and calculated. More information about how we calculate confidence intervals for each of the Dashboard’s metrics can be found in our technical documentation.

Where can I find confidence intervals on the City Health Dashboard?

On the City Health Dashboard, confidence intervals for census tract-level measures can be found on the Metric Detail page, in the “Table” tab, displayed in the “Error Margin” column.

Confidence intervals for city-level data are on the Compare Cities page, where they are displayed in parentheses next to the estimate for each demographic group.

Confidence intervals are also available in downloaded data in the columns labeled “lci” for lower confidence interval, and “uci” for upper confidence interval.

Confidence Intervals in Practice

Confidence intervals can be helpful measures of precision when interpreting data. Take the estimate for children in poverty in two New York City (NYC) neighborhoods: in one Midtown neighborhood (Census Tract 010100), the children in poverty rate in 2023 is 24.5%. However, the confidence interval is 0% - 100%, which means the true value could be as low as 0% or as high as 100%. This is not very precise, so we might want to verify this estimate with community members or other data sources before using it for decision making.

In contrast, in the Lower East Side neighborhood (Census Tract 000202), the children in poverty rate is 31.9%, and the confidence interval is 17.2% - 46.5%. This is much more precise, and therefore more useful for making decisions about where to send resources or in guiding interventions.

Confidence intervals can also help you compare two values to determine if they are likely to be truly different. In the case of children in poverty, it is possible that the Midtown and Lower East Side values are not very different – even though 24.5% is lower than 31.9%, the confidence intervals overlap, so the true values for each of these neighborhoods could actually be the same. If the confidence intervals of two values do not overlap at all, it is likely that the true values are different.

For example, near NYC’s Union Square (Census Tract 004800), the children in poverty rate is 2.7%, with a confidence interval of 0% - 7.3%. Because this does not overlap at all with the confidence interval of the Lower East Side children in poverty rate, we can be reasonably confident that the rate of children in poverty is lower near Union Square than in the Lower East Side.

Best Practices with Confidence Intervals

While in a perfect world all estimates would be precise, this is not always the case. If you see estimates with large confidence intervals, here are some best practices to help you better understand and interpret the data:

• Look at other years of data. Looking at trends over time can show you if the specific estimate is stable across multiple years or if it fluctuates substantially. Estimates that show large fluctuations year-to-year and have large confidence intervals should be interpreted with caution, as the changes can be due to random chance rather than reflecting true changes over time.

• Look for data interpretation guidance. It can also be useful to review guidance when presenting data points with large confidence intervals. The Dashboard does this by flagging estimates with large confidence intervals and encouraging users to interpret the estimate with caution

• Ask for community feedback. Community-based organizations and neighborhood residents can provide crucial context to national-level data, clarifying interpretation and decision-making.

Data are helpful for understanding more about the world, yet they do not always capture the world perfectly. Confidence intervals give more information about a data point to help us understand what it represents and how we can use it to inform action. We hope you take a look at the confidence intervals on the Dashboard to guide your data exploration!