The mission of the decennial census is to count everyone living in the United States once, only once, and in the right place. Accurate counts are important because census results are used to allocate seats in the House of Representatives, to redistrict, and to distribute around $400 billion in federal funds each year. Fulfilling this mission is daunting in a country that is growing increasingly more diverse and complex.

Despite best efforts to count everyone, the U.S. Census Bureau’s own record of research shows persistent differential undercounts of some minority populations across decennial censuses, such as among African Americans and Hispanics. Prior ethnographic studies conducted during census data collection operations, but separate from those operations, have identified a range of factors affecting coverage and illuminated how and why they may affect enumeration in some populations. However, previous ethnographic studies have not included systematic observations of live interviews and respondents in an actual decennial census environment.

The present evaluation was conducted to address this gap, in addition to addressing the overall need to increase the understanding of the types and sources of these persistent miscounts in the census. In this report, the authors present findings from comparative ethnographic and partial validation research. The goal of this evaluation was to explore types and sources of possible census coverage error and identify the characteristics of households and of persons affected by them through observation of live, in-person 2010 Census interviews. We aimed to identify similarities and differences across race/ethnic groups to contribute to the discussion of why differential counts persist and to suggest improvements and new research for the next census.

To the authors’ knowledge, this is the first decennial census in which researchers have been able to conduct systematic ethnographic observations of enumeration methods and coverage by observing interviewers conduct live, in-person interviews with respondents within actual data collection operations in the decennial census environment. This enabled ethnographers to document factors within the interview that may have affected coverage, as well as to debrief respondents in real time.

This is a multi-stage, mixed methods evaluation. The first stage consisted of controlled comparisons of small-scale qualitative field observation studies across nine race/ethnic groups in which ethnographers accompanied interviewers to observe and audiotape (when permitted) live interviews and conduct immediate debriefings with respondents on coverage-related questions. The ethnographer observations were an independent assessment of where persons likely should be counted, reasons for any apparent miscounts, and other important enumeration issues. The second stage of this evaluation was a small-scale partial validation check of the 2010 Census by identifying inconsistencies in whether persons identified in the observed housing units were or likely should have been counted in the ethnographer-observed housing units on Census Day, April 1, 2010.

These stages were conducted on three personal-visit operations in decennial year 2010. Two of these operations were 2010 Census operations: Update Enumerate and Nonresponse Followup. The third operation was the independent Census Coverage Measurement Person Interview operation, part of the Census Coverage Measurement Survey, conducted some months later. The intent of observing both census and later Census Coverage Measurement operations in the same site was to facilitate the identification of wider factors in the sites over the time span of successive data collections that may have affected enumeration and coverage. The initial design specified that the same ethnographer would observe both the 2010 Census and later Census Coverage Measurement operations in the same site to document changes in the site and in different sets of respondents over time. However, that design was changed in consultation with Census Coverage Measurement statisticians to have different ethnographers in the census and Census Coverage Measurement observations in each site to avoid contaminating the independence of the Census Coverage Measurement from the census, which was a core requirement in the Census Coverage Measurement dual systems estimation methodology. The evaluation methods, procedures and research questions were the same in each of the three operations.

These observation studies were conducted in sites specially selected by the evaluation authors to focus primarily, but not exclusively, on certain race/ethnic groups: African Americans in Chicago; Alaska Natives in Kodiak Borough, Alaska; American Indians on two Southwest reservations (one in Nonresponse Followup); Asians (primarily Chinese) in the San Francisco Bay area (including Chinatown during the Nonresponse Followup operation); Native Hawaiians and Other Pacific Islanders on the Big Island of Hawaii; non-Hispanic Whites in the Kansas City, Missouri area in Jackson County; Hispanics in Dallas/Fort Worth, Texas; Middle Easterners in Wayne and Macomb Counties in Michigan (during Census Coverage Measurement only); and a final generalized site in Broward, County, Florida.

There were two sets of ethnographers per site. The first set observed 2010 Census operations in either the Update Enumerate operation in March and April or the Nonresponse Followup operation in mid- to late-May, 2010 during the height of those operations. Ethnographers for the Nonresponse Followup and Update Enumerate operations accompanied 69 enumerators and observed 363 interviews. The second set of ethnographers observed Census Coverage Measurement Person Interview operation in August, 2010 within specific pre-selected Census Coverage Measurement sample clusters in these sites. Ethnographers for Census Coverage Measurement accompanied 53 interviewers and observed 318 interviews in the Census Coverage Measurement Person Interview operation.

The second stage of this evaluation was a records check. The purpose of the records check was to partially validate where persons associated with the ethnographer-observed housing units on April 1, 2010 should be counted in the census. This was done by triangulating across multiple data sources to identify inconsistencies in who should be counted at the ethnographer-observed housing units and identify types and sources of possible coverage error. A records check inconsistency is defined as when the location of where a person associated with the observed housing unit on April 1, 2010 should be counted differs between two or more data sources. Census Bureau staff matched and compared the rosters from 1) the standard interviews; 2) the ethnographers’ assessments of where each person should be counted; 3) a special final localized 2010 Census unedited dataset created around each site for this evaluation and, for the Census Coverage Measurement observations, 4) the final localized 2010 Census Coverage Measurement dataset matched to the 2010 Census records around the sample clusters. It should be noted that searches of persons with possible coverage error were not done beyond these localized records check datasets and some of these persons who may have been omitted from the localized datasets could have been included in the 2010 Census elsewhere, while some other persons could have been duplicated elsewhere in the country. To distinguish the custom localized datasets used in this evaluation from the overall national 2010 Census Unedited Files, the custom files will be identified as “final localized 2010 Census unedited dataset” and “final localized Census Coverage Measurement (or CCM) dataset. Taking into account data from all of these sources, the research team coders made their own assessment of where each person should be counted.