The American Community Survey Office (ACSO) is interested in redesigning the login screen of the online response instrument. Based on feedback from experts at the 2019 National Academics of Sciences annual meeting, an interdepartmental team of researchers created a new login screen design. We compared the current production design and the newly developed design to better understand the usability of the designs as well as to obtain cognitive feedback.  We tested both the English version and the version translated into Spanish.  Ten English-speaking and ten Spanish-speaking participants were recruited for the study. For each language, five saw the production design and five saw the new design. Approximately half saw the design on their desktop/laptop display and half saw the design on their smartphones.

Overall, the new login screen performed well. The metrics included efficiency (how long it took to log in), effectiveness (how accurately participants entered the User ID), preference (self-reported preference of login screen given a side-by-side comparison of both designs), and satisfaction (self-reported ease or difficulty of logging in).  Participants in both languages were more efficient (took less time) with the new design.  For the other metrics, results varied across the two languages.  For English-language speakers, the new design was more effective than the production design, meaning participants were able to log in without assistance more often.  This was not the case for the Spanish-language participants, where the production design performed slightly better than the new design.  More English-language participants preferred the new design; however, for the Spanish-language participants, there was a slight preference for the production design.  English-language participants were equally satisfied with both designs, while the Spanish-language participants were more satisfied with the production design.

Participants also provided cognitive feedback about their impressions of the login screen during specific probing questions as well as from spontaneous verbalized comments.  This feedback is included in the body of the report.  The team recommends moving forward with the new design, as it was most successful with English-speaking participants.  However, we are recommending some design alterations to address concerns raised with the Spanish-speaking participants.