The March 2023 Census Test is an early decade test in preparation for the 2030 Decennial Census. The test compares response rates to different mailed notification panels to encourage response to an online survey and a paper questionnaire follow-up. To prepare for the test, the Center for Behavioral Science Methods (CBSM) at the Census Bureau carried out cognitive testing of the mailing materials in both English and Spanish. Up to four mailings were used in the 2023 Census Test. The mailing materials consisted of the following: an invitation letter inviting the respondent to respond to the 2023 Census Test (Letter 1) with a Frequently Asked Questions (FAQ) insert, a reminder letter (Letter 2), a reminder postcard, and a final reminder letter (Letter 4), which included a paper questionnaire, FAQ insert again, and a prepaid return envelope. The online survey website address (URL) and authentication code (Census ID) were present in each mailing. Some of the cognitive interviews were conducted virtually and others were conducted in person.
The research questions included:
• How did participants interact with a priority mail envelope, and did they read the FAQs?
• How did participants react to a message about using government and administrative records to answer questions if they do not respond to the survey?
• What items in the mailing materials stood out to participants as well as what did participants think was the main point was of the mailings?
The research questions for this study were answered by how participants interacted and responded to the different mailing materials. The test administrators noted spontaneous verbalizations, observed how participants reacted to and read the materials, and asked probing questions about the materials.
The results included feedback that most participants did not notice the government and administrative records message without first being directed to it by the test administrator. However, once they were directed to the message, many did not like the idea of using those administrative records data in place of collecting the information directly from the individual. Participants commented on the confusing messages about what happens next if they do not respond online. Participants noticed the required by law language and said they would likely respond because of it. Participants also said they would not have expected to receive mail from the government in a priority mail envelope, but they would be more likely to open it. Participants also noted the presence or absence of the Census ID, and mentioned that they appreciated the short, 10-minute length of the survey. Participants noted that certain bolding on the last letter helped them note what was important. Participants mentioned that the focus on “your” community was a positive component of the letters.
Our recommendations based on this round of testing included the following: (1) Modify the government records message. One option is to use the following sentence: We are depending on your responses to the census questionnaire. However, if you do not respond, we will use existing data from government records to fill in information for your household; (2) Change “if you cannot respond online/if you are unable to respond” to “if we do not receive your response online...”; (3) Include the word OR on the 4th letter to make it clear that respondents should either reply online or on the paper questionnaire and not do both; (4) Send materials in a priority mail envelope; (5) Use a box around the URL for the call to action and include the Census ID on the 4th letter; (6) Use 10-minute language in more mailings; (7) Bold the main points in other
mailings, like is done in the 4th Letter; (8) Remove the term "burden" from the FAQ; and (9) Keep the "your community" language and add it to additional mailings.