Test Objective

• In January through March of 2006, the American Community Survey (ACS) conducted the first test of new and modified content since the ACS reached full implementation levels of data collection. The results of that testing will determine the content for the 2008 ACS.
• The focus of the proposed industry and occupation changes included in the 2006 ACS Content Test was to increase the amount of information provided to coders in order to produce a valid industry or occupation code. The item tested was the Type of Industry check box, an indicator of whether a respondent’s reported industry was manufacturing, wholesale trade, retail trade, or something else.

Methodology

• The Content Test compared two versions of the five survey items included in the industry and occupation question set. The control version replicated the current ACS Type of Industry question. In the control version, the Type of Industry question asked “Is this mainly – Mark (X) one box: “manufacturing, wholesale trade, retail trade, or other (agriculture, construction, service, government, etc.),” following the question “What kind of industry was this?” The test version modified this Type of Industry question by adding an additional check box for “service” after “retail trade” and by deleting the parenthetical reference to service within the “other” industry type. Additionally, the examples provided throughout the Industry and Occupation question series were modified slightly, although the effects of those changes were not statistically tested in isolation.

Research Questions and Results

• Question 1: Will the addition of a separate services industry type check box decrease the proportion of respondents who respond “other?” In addition, will this change impact the distribution of responses for the manufacturing, wholesale, and retail trade categories?
  Results: The proportion of responses showed a statistically significant decrease in responses to “other” and “retail trade” in the test version of the instrument.
• Question 2: Will the addition of the separate services industry type check box and refined examples change the high-level distribution of coded industries and occupations?
  Results: The distributions of high-level industry and occupation categories were not statistically different across instruments.
• Question 3: Will the combined changes reduce the number of cases that are not “codeable?”
  Results: The proportion of cases considered not “codeable” was not statistically different across instruments.
• Question 4: Will the combined changes result in a decrease in item missing data?
  Results: There were no statistically significant changes in responses to three of the industry and occupation survey items (Employer Name, Type of Business, and Most Important Activities). Nonresponse to the Type of Industry check box item showed a statistically significant decrease in the test version compared with the control version of the instrument. Nonresponse to the Type of Work item showed a statistically significant increase in the test version compared with the control version of the instrument.
• The findings reported above do not lend strong support for the test version of the Industry and Occupation question series. The unexplainable increase in nonresponse to the Type of Work item is troubling, and potentially damaging to data quality. Thus, it is recommended that the test version of the Industry and Occupation items not be adopted.