Test Objective

In late August through mid-December 2010, the Census Bureau conducted a field test of new and revised content in the 2010 American Community Survey (ACS) Content Test. The results of that testing will help determine the content to be incorporated into production ACS in 2013.

Research shows that respondents have difficulties remembering all the information read to them in a single, verbose question (Webster, 2006). If a question contains a long list of components or concepts for respondents to consider, respondents tend to focus on the last items in the list and forget the others when the list is presented orally. In the case of the ACS CATI (Computer Assisted Telephone Interview) /CAPI (Computer Assisted Personal Interview) Property Income question, respondents are asked if they received ―interest, dividends, net rental income, royalty income, or income from estates and trusts.‖ We believe respondents are focusing on reporting whether they received income from estates and trusts, royalty income, or net rental income and missing the reporting of interest and dividend income. We have anecdotal evidence to support this belief. A similar situation occurs with another income question. For example, while observing ACS interviews, we noted that respondents report having a wage/salary job but report having no ―wages, salary commissions, bonuses or tips‖ from that job. Because this ―end of the list‖ effect only seems to occur in telephone or personal interviews – not in mail mode responses – changes are being made to the CATI/CAPI questions only for this test. However, the analysis will study the impact on responses to the Property Income question across all modes, since we do not publish ACS data by mode. An investigation of the impact of the change will be done by mode for informational purposes only.

Summary

There were unfavorable results for the Property Income test question on the item missing data rates. Item missing data rates for amount were significantly higher for the test version than the control. There were more questions in the test version. Much of this difference in item missing data rates for the test version can be explained by more ―don’t know‖ responses to the net rental income and royalty income questions. No significant findings were found with item missing data rates by mode for amount. By mode, item missing data rates for recipiency were higher in CATI/CAPI combined and for CATI individually for the test version.

There were several positive results that ultimately gave reasoning to go forward with the recommendation for implementing the question change. The test version raised the estimate of the number of people receiving Property Income. Specifically by mode, recipiency estimates increased for CATI and CAPI individually and combined. The net difference rates for recipiency overall were also lowered.

Based on the test results, it was evident that there were several advantages to changing the Property Income question. It is recommended that the Property Income question be changed to that of the test version for CATI and CAPI.