U.S. flag

An official website of the United States government

Skip Header


Data Reliability Indicator Based on the Coefficient of Variation: Results from the Second Round of Testing

Written by:
SSM2010-04

Abstract

This study was the second round of usability testing for the Data Reliability Indicator for American Community Survey (ACS) data tables proposed by the sponsor team. Three prototype tables with a color-coded indicator based on an estimate’s coefficient of variation were compared to a baseline table that represented a past production version of an ACS data profile. One prototype had three levels of reliability and the other two had four levels of reliability. One version of the four-level table was labeled with the terms “excellent,” “good,” “fair,” and “poor.” The other was labeled with the terms “reliable,” ”mostly reliable,” “somewhat reliable,” and “unreliable.”

No differences were found in users’ accuracy and efficiency (time on task) across the four tested table designs. Overall, 19 of the 21 participants indicated that they preferred the prototype tables over the baseline tables, although their preference for either the three– (n=10) or four–level (n=9) tables or either of the two alternative wording options was fairly evenly split (nine preferred the “good” wording, ten preferred the “reliable” wording, and two had no preference). However, there were some significant differences in satisfaction scores between the tables. There is evidence that satisfaction as measured by the QUIS instrument was significantly higher for both the baseline and the four–level tables than for the three–level table.

An analysis of how likely participants were to mention the margin of error (MOE) or report it along with the estimate showed no significant differences by table design. Eye-tracking heat maps showed that participants looked at the MOE column on the table fewer times for the three–level condition.

Participants did use the reliability indicator and frequently said that they would report the message contained in the reliability column along with the estimate (12% indicator reporting for the three–level condition and 43% reporting for each of the four–level conditions). Participants in both the four–level “good” table and the four–level “reliable” table conditions were significantly more likely to report the message from the color–coded reliability indicator than the three-level indicator in a post–hoc test. Also, participants in both the four–level “good” table and the four–level “reliable” table conditions were significantly more likely to explicitly report the message from the color–coded reliability indicator along with the estimate than participants in the three-level indicator condition. There was no significant difference between the two four–level tables themselves on this variable.

More detailed results and potential usability issues are discussed.

Page Last Revised - October 8, 2021
Is this page helpful?
Thumbs Up Image Yes Thumbs Down Image No
NO THANKS
255 characters maximum 255 characters maximum reached
Thank you for your feedback.
Comments or suggestions?

Top

Back to Header