Outsiders’ views of a person’s race or Hispanic origin can impact how she sees herself, how she reports her race and Hispanic origins, and her social and economic experiences. The way outsiders describe non-strangers in terms of their race and Hispanic origin may reveal popular assumptions about which race/Hispanic categories are salient for Americans, which kinds of people are seen as multiracial, and the types of cues people use when identifying another person’s race. We study patterns of observer identification using a unique, large, linked data source with two measures of a person’s race and Hispanic origin. One measure (from Census 2000 or the 2010 Census) was provided by a household respondent and the other (from the other census year) was provided by a census proxy reporter (e.g., a neighbor) who responded on behalf of a non-responsive household. We ask: Does an outsider’s report of a person’s race and Hispanic origin match a household report? We find that in about 90% of our 3.7 million (nonrepresentative) cases, proxy reports of a person’s race and Hispanic origin match responses given by the household in a different census year. Match rates are high for the largest groups: non-Hispanic whites, blacks, and Asians and for Hispanics, though proxies are not very able to replicate the race responses of Hispanics. Matches are much less common for people in smaller groups (American Indian/Alaska Native, Pacific Islander, Some Other Race, and multiracial). We also ask: What predicts a matched response and what predicts a particular unmatched response? We find evidence of the persistence of hypodescent for blacks and hyperdescent for American Indians. Biracial Asian-whites and Pacific Islander-whites are more often seen by others as non-Hispanic white than as people of color. Proxy reporters tend to identify children as multiple race and elders as single race, whether they are or not. The race/Hispanic composition of the tract is more powerfully predictive of a particular unmatched response than are tract-level measures of socioeconomic status; unmatched responses are often consistent with the race/Hispanic characteristics of the neighborhood.