This assessment report documents the results and major findings from the operations of the 2010 Census Content and Forms Design Program. Only Public Use Forms and other materials produced within the purview of the Decennial Management Division Content and Language Branch are covered within the scope of this assessment. Materials produced outside of the Content and Language Branch, such as field procedures, manuals, and field evaluation questionnaires, are not within the scope of this assessment. The report aims to inform the Content and Forms Design Integrated Product Team, stakeholders, and decision-makers on the successes, impacts, and recommended changes and improvements for future censuses.   

The research questions in this assessment were answered by utilizing lessons learned documents, census reports and documents, and other tracking reports. The main research questions and results related to the Content and Forms Design Program are described below:

1. What was the process and outcome for determining final 2010 questionnaire content?
   
   

   The content determination activities started early in the decade through tests sponsored by the Content Research and Development Team and later through the Content and Forms Design Integrated Product Team in conjunction with the Mode Consistency Subgroup. Team members provided input and recommendations based on past test results and recommended 2010 Census questionnaire content for a baseline. Once the content was baselined, a formal Change Control process was established to handle questionnaire content and design changes.

   Materials used for the 2010 Census Mailout/Mailback operation were pre-tested and field tested extensively during the decade. Questionnaires used for field operations, such as Be Counted, Group Quarters Enumeration, and Enumeration at Transitory Locations, were not field tested due to the de-scoping of the 2008 Census Dress Rehearsal. Although these questionnaires were cognitively pre-tested in 2008, some recommendations from theses tests were not included in the final questionnaire due to timing issues.

   In general, content was finalized early enough to meet field operational and print production schedules, but many “non-substantive” content updates to English text did affect translation, forms design, and thorough review of the language questionnaires. The final 2010 Census printed materials produced under the Content and Forms Design Program met the needs of various operations and subject matter experts. However, some of the forms/questionnaires produced by other areas could have benefitted if they were developed and reviewed under the Content and Forms Design Program.

2. What was the review process and what were the outcomes? How well did the automated system (IBEAM) perform?
   
   

   A structured five-step review process was established for reviewing, adjudicating, updating, and finalizing the 2010 Census questionnaires and other supporting materials. Each review step resulted in updated draft forms that were validated against comments received before the next round of review. These steps were: 1) Initial review of drafts; 2) Content and Forms Design Integrated Product Team review that included review for data capture and response processing; 3) Initial quality review; 4) Final quality review; and 5) Technical table review.

   The reviewers of the 2010 Census forms were identified and most of them participated fully. It would be beneficial if the reviewers also included the “owners” of one or more of the content areas, not limited to demographic content, on the questionnaire and who participated during the content determination process.

   The review process generally enabled thorough review of the materials and produced print files that met the standard set forth in the print contracts we had. The process could be made more efficient by establishing a portal for form deliveries, and providing an electronic/automated mechanism for documenting reviewer comments and change history of each print item. A formal inspection confirming that the final print files required for the printing process were all present, valid, and correctly formatted, prior to delivery for print production, could have also helped eliminate forms being re-delivered due to design errors.

   The Information Base for Exchange Administration and Management system, used as the primary source to create the 2010 Census questionnaires, performed as designed and ensured content consistency across questionnaires for different operations. It eliminated the need for the Administrative and Customer Services Division’s forms designers to manually input questionnaire content into the forms design software. The system also notified the content users and the form designers the status of each questionnaire as well as helped to document changes in an organized fashion. Subject matter experts used the system to compile keying specifications and the processing staff used the system to generate record layouts for post data collection processing.

   Since the system did not have an integrated design tool, there were instances where content elements submitted from the IBEAM system (such as graphic and character spaces) were not received successfully by the forms design software. Additionally, the keying matrix report generated from the IBEAM system was not utilized as designed by the data capture contractor in an automated fashion.

   In general, the system performed well for creating consistent content for the English and Spanish 2010 Census questionnaires. It would be beneficial if the system could be used in managing content in languages other than English and Spanish, as well as other materials including letters, envelopes, and other public use forms. The system, as a content repository, should also integrate and include the content collaboration and forms review processes in its workflow.

3. What was the process and outcome for ensuring the 2010 paper questionnaires could be data captured and processed correctly?
   
   

   The 2010 Census paper questionnaires were designed using the 2008 Census Dress Rehearsal questionnaire as the base. Lockheed Martin, the 2010 Census data capture contractor, conducted data capture for the 2008 Dress Rehearsal questionnaires and participated in the testing and finalization of the 2010 Census questionnaires.

   To the extent possible, the 2010 Census questionnaires were drafted following the design guidelines provided in the “Forms Design Guidelines” issued by Lockheed Martin for successful data capture. Due to paper size limitation on the inclusion of additional content, it was extremely challenging to design a questionnaire that completely conformed to all of these guidelines. In addition, the guidelines did not include enough flexibility for designing all form types. For the 2020 Census, data capture of paper questionnaires should be included early in the content testing phase so that the data capture technology can be optimized for the designed forms. Additionally, the questionnaire content should be determined and modified in conjunction with the overall look of the questionnaire, as well as its implications for data capture and data quality.

4. Were requirements and issues communicated timely and sufficiently with stakeholders?
   
   

   While issues found during the forms design and finalization process were generally communicated with stakeholders in a timely and sufficient manner, and resolved without any impact to the succeeding operations, there were difficulties managing requirements that were undefined or came in late from field operational areas. Improvements are also needed for delivery of draft forms to the data capture contractor. For the 2020 Census, a formal procedure should be established for gathering and documenting requirements. These requirements should be incorporated in a database that can be used to facilitate content discussion and the forms review process among stakeholders.

5. How did the planned start and finish dates for each of the content and forms design activities/deliveries compare to the actual dates?
   
   

   The 2010 Census Content and Forms Design Program operations had the baseline start date of June 29, 2006. This start date was early enough to include all the planning and production activities from the 2008 Census Dress Rehearsal materials. The baseline finish date for the program was September 14, 2009. The first print file, the “Notice of Visit” sheet, was approved for print delivery on May 23, 2008. The last print file, the postcard with the Telephone Questionnaire Assistance telephone numbers in six languages, was delivered on January 22, 2010. Other than a few late deliveries due to operational changes (revision of the advance letter or addition of the Direct Mail postcard), the majority of the print files were delivered on time or ahead of schedule.