Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
Topics
Data & Maps
Surveys & Programs
Resource Library
Using Administrative Data to Evaluate Nonresponse Bias in the 2024 Current Population Survey Annual Social and Economic Supplement
Using Administrative Data to Evaluate Nonresponse Bias in the 2024 Current Population Survey Annual Social and Economic Supplement
In the latest release of its annual income report, Income in the United States: 2023, the U.S. Census Bureau tabulates the national income distribution using survey responses collected during February 2024 through April 2024 (with the majority of data collected in March) as part of the 2024 Current Population Survey Annual Social and Economic Supplement (CPS ASEC). However, as Research Matters blog posts from each of the last four years have explained (in 2020, 2021, 2022, and 2023), nonresponse in the CPS ASEC has biased income and poverty estimates.1 Nonresponse bias means there are differences between those who respond to a survey and those who do not respond, potentially leading to an estimate that is less representative of the population.
Figure 1 plots unweighted response rates to the monthly CPS through April 2024. While 82% of sampled U.S. households responded to the basic survey in March 2019, the response rate during March 2024 was 67%.
Figure 1. Unweighted Current Population Survey Monthly Response Rates: January 2010 to April 2024
Source: U.S. Bureau of Labor Statistics, <www.bls.gov/osmr/response-rates/home.htm>.
Lower response rates are not necessarily problematic if respondents and nonrespondents are roughly similar, such that the resulting sample of respondents remains representative of the nation. To investigate this possibility, we combined administrative records, 2010 Census data, and American Community Survey responses to determine whether households that did and did not respond to the CPS ASEC differed, and whether these differences ultimately biased the resulting income and poverty estimates. Figure 2 plots earnings differences between respondents and nonrespondents using IRS Form W-2 earnings records linked by address. It shows that in 2020, respondents began to have higher earnings than nonrespondents, whereas in earlier years — 2017 to 2019 — the gap was narrower. That phenomenon has repeated each year since 2020. Specifically, differences in median earnings between respondents and nonrespondents were statistically significant in each year. However, each of the years from 2020 to 2024 had differences between respondents and nonrespondents that were larger than the differences in each of the years 2017 to 2019, with the two exceptions that differences in 2024 and 2022 were not significantly different from the difference in 2017.2
Figure 2. Earnings Differences Between Respondents and Nonrespondents
Source: U.S. Census Bureau, Current Population Survey, 2017 to 2024 Annual Social and Economic Supplements (CPS ASEC) linked to IRS Form W-2 earnings data by address.
As in prior years, we measure how much these differences between respondents and nonrespondents ultimately bias the resulting income statistics. We measure this bias by estimating a nonresponse adjustment that leverages information from linked data sources that are not available in standard survey processing. This information includes linked administrative, decennial census, and survey data available for respondent and nonrespondent households. Since low-income households have been less likely to respond, this adjustment shifts weight away from higher-income respondent households toward lower-income respondent households.
Using alternative weights created from the alternative nonresponse adjustment, we calculate alternative income and poverty statistics and compare them to the official (survey-only) estimates. We find that estimates of median household income and poverty rates were not statistically different in 2017, 2018, or 2019. This indicates that in pre-pandemic years the normal nonresponse adjustment in the survey weights sufficiently addressed nonresponse bias for these statistics.
However, Figure 3 shows that since 2020, the alternative income estimates have diverged from those estimated using the CPS ASEC survey data alone. Figure 3 plots the ratios of these estimates across the household income distribution. The horizontal line at one indicates the ratio at which the alternative-weight estimate would equal the survey-only estimate at that point in the distribution. Before 2020 these ratios hovered near one, but since 2020 the alternative income estimates have been about 2% to 3% lower than the survey-only estimates.
Figure 3. Comparison of Nonresponse-Adjusted Income to Survey-Only Income Over Time
Source: U.S. Census Bureau, Current Population Survey, 2017 to 2024 Annual Social and Economic Supplements (CPS ASEC) linked to administrative and survey data.
From 2020 to 2024, official poverty rates with the alternative weights were slightly — yet statistically significantly — higher than the survey estimates, by 0.5, 0.5, 0.3, 0.3, and 0.4 percentage points respectively. These differences contrast with the pre-pandemic years 2017 through 2019, which showed no statistically significant differences in poverty rates between the alternatively weighted estimates and the survey estimates.3
To summarize, since 2020, survey nonresponse has continued to bias income statistics upward by 2% to 3% and official poverty rates downward by a fraction of a percentage point.
In ongoing research, the Census Bureau continues to develop and apply new methods that better address nonresponse bias and other sources of nonsampling error. For example, the National Experimental Wellbeing Statistics combine data from a wide variety of sources to produce improved estimates of the income distribution. One component of that work involves applying the alternative weighting scheme discussed above. Whether current trends in response rates continue, this research promises to address well-known challenges with our surveys and advance the Census Bureau’s commitment to serve as the country’s leading provider of quality data about its people and economy.
Definitions and information on confidentiality protection, sampling error, and nonsampling error are available at www2.census.gov/programs-surveys/cps/techdocs/cpsmar24.pdf.
###
1 These blogs are available at <www.census.gov/newsroom/blogs/research-matters/2023/09/using-administrative-data-nonresponse-cps-asec.html>, <www.census.gov/newsroom/blogs/research-matters/2022/09/how-did-the-pandemic-affect-survey-response.html>, <www.census.gov/newsroom/blogs/research-matters/2021/09/pandemic-affect-survey-response.html>, and <www.census.gov/newsroom/blogs/research-matters/2020/09/pandemic-affect-survey-response.html>. More information about the design of the survey, including weighting for nonresponse, is available in Technical Paper 77 at <www2.census.gov/programs-surveys/cps/methodology/CPS-Tech-Paper-77.pdf> and in a summary of weighting in the CPS at <www.census.gov/programs-surveys/cps/technical-documentation/methodology/weighting.html>.
2 We refer to survey years in the text, tables, and figures to keep the references consistent and more clearly identify the 2020 to 2024 CPS ASEC as the ones affected by the nonresponse bias. These surveys ask about income received in the previous year, however. Statistics from the 2020 CPS ASEC, for example, describe income received during 2019.
3 The differences between the survey and alternative poverty rates in 2020, 2021, 2022, 2023, and 2024 are each statistically significant, but the differences among those differences are not statistically significant.
Page Last Revised - September 10, 2024
✕
Is this page helpful?
Yes
No
Yes
No✕
NO THANKS
255 characters maximum
255 characters maximum reached
255 characters maximum reached
✕
Thank you for your feedback.
Comments or suggestions?
Comments or suggestions?
