As part of our annual data review process, the American Community Survey (ACS) program calculates and publishes quality measures for the ACS data including response rates, coverage rates, item allocation rates, sample sizes and number of interviews. These quality measures are available on our quality measures website for years 2000 to the present. In addition to informing data users about the overall quality of the data, these measures also serve as input to our assessment of whether the data meet the Census Bureau’s Statistical Quality Standard F1: Releasing Information Products as described at Statistical Quality Standards.

Requirement F1-6 stipulates that the coefficient of variation, defined as the standard error divided by the estimate, for a majority of the key estimates is less than 0.30. To address this requirement, the ACS assesses the sampling error for 17 “key” estimates. These key estimates were selected based on the following criteria:

1. Occur in about 10 percent of the population at the national level.
2. Are used for critical social policy purposes at subnational levels.
3. Are included in our Data Profiles.
4. Cover a spectrum of topic areas including both population and housing characteristics.

For each estimate, the median census tract-level CV is calculated and compared to the benchmark of 0.30. Tract is used as the level of geography because the ACS is uniquely designed to provide information for these small areas for a number of applications.

In a typical year, 12 to 13 of the 17 key estimates pass the threshold and the quality standard is met. However, in 2020, the ACS faced numerous data collection challenges as the result of the COVID-19 pandemic as described in the blog Adapting the American Community Survey Amid COVID-19. As a result of those challenges, the ACS collected only two-thirds of the responses it usually collects in a survey year. This reduced number of interviews in the 2020 portion of the 2016-2020 5-year estimates caused an increase in the CVs by approximately 15-20% in relative terms (e.g., a CV of 0.25 would increase to 0.29-0.30). We would expect to see similar relative increases in the published margins of error as well.

As a result, only 8 of the 17 key estimates passed the threshold with 9 of the 17 estimates having a median tract-level CV that exceeded 0.30. Most of these “new” estimates that failed previously had a CV in the range of 0.25-0.30 and this year fell in the range of 0.30-0.35. Thus the 2016-2020 ACS 5-year data fails the quality standard requirement based on this criterion.

We are, however, continuing to publish the data under the waiver process described on the Statistical Quality Standards webpage. The reason for this is that we believe that there is a critical need for the ACS data as it is the only source of data for small geographic areas and that the margins of error published with each estimate allow data users to make informed decisions regarding the reliability of the estimate. While users are always encouraged to make use of the margins of error in their decision making, we further stress to use caution when using estimates with high margins of error.

We anticipate similar impacts to the margins of error in future 5-year products that contain the 2020 data.