Revisions to the American Community Survey (ACS) questionnaire were fielded as part of a Content Test in 2022. One of the topics tested was the household roster questions. The complexity of household living arrangements in the United States is increasing (Cherlin 2010) and this adds to the challenge of rostering everyone in a household correctly. There is evidence that the roster section of the ACS can be confusing to some respondents (Ashenfelter et al. 2012). They may not know who in the household should be included in the roster, and who should not. To address these concerns, the roster questions were added to the content test for the first time. The proposed revisions were cognitively tested for the self-administered paper instrument as well as the interviewer-administered modes (RTI, 2021). The revisions appeared to reduce participant confusion. Similar changes to the sequence of roster screens were then programmed into the online self-administered instrument of the ACS (hereafter referred to as the online form) for usability testing. This report details the usability testing results of the online roster questions. The roster questions are on five screens. Participants are first asked to list everyone living or staying at the address. Then they are asked a series of questions to determine whether anyone was missed (two separate screens) or erroneously included (two or more screens depending on their responses).
Ten English-speaking participants and five Spanish-speaking participants were recruited for the usability study. Participants answered the household roster questions as they pertained to their actual household. Participants then answered the same roster questions based on a series of “vignettes” -- short hypothetical stories purpose-built to present respondents with living situations that can be ambiguous and difficult for respondents to report accurately. We asked them to imagine they were in such a situation to see how they would answer the questions. We also asked participants a series of satisfaction questions about their experience with the household roster questions. Approximately half of the participants saw the roster screens on their desktop/laptop display and half saw the roster screens on their smartphones. Usability testing uncovered issues with the roster series where the participants duplicated names on the roster and/or took additional time to navigate backwards in the online instrument to remove a name off the first question and put that same name on the 2nd or 3rd question in the series. Results from the Spanish-speaking participants highlight that there was also confusion on who to include on the original roster screen (the first screen). There was also some confusion with the last two screens in the series that aimed to identify if anyone was erroneously included in the household roster. Confusions arose with respect to counting college students or boarders (from the vignettes). Finally, there were some usability design issues identified with the layout of the screens. The team recommended additional wording changes to resolve some of the issues identified, modifying the layout of the screen including where the household roster was displayed, editing the rationale for asking the question and finally, conducting additional usability testing with additional participants. We also recommend analyzing the paradata from the upcoming field testing to determine if duplication of household roster names occurs or if the backing up and removing of a household name on one screen to only add the name to another roster screen occurs outside of this small sample of participants.