The Census Bureau produces a vast array of statistical information used by governments and businesses to inform decisions that affect everyone’s lives. It designs the surveys, collects the data, processes the completed questionnaires, and produces the statistical information. These statistics touch every aspect of Americans’ lives – health, crime, income, education, labor force participation, housing conditions, consumer expenditures, and a host of others.

Over the past two years the senior leadership of the Census Bureau has been engaged in a series of organizational change initiatives designed to improve its ability to supply the country credible and cost-efficient economic and social statistics. The rationale for this is simple:

1. The difficulties of measuring the busy, diverse, and independent American society and economy are increasing every year (that is, it costs more money to do the same things the Census Bureau has done for years)

2. The demands by American business, state, local, and community leaders for statistics on their populations are continually increasing

3. New technologies are being invented almost daily that can be used to make it more convenient for the American public to participate in surveys

4. New digital data resources are being created from Federal-state-local government programs, private sector transactions, and internet-related activities

5. Near-term Federal government budgets are likely to be flat or declining

Combining these five observations leads to a profound conclusion: the current Census Bureau survey and census methods are unsustainable. Changes must occur in the acquisition of data and construction of statistical information for the Census Bureau to succeed.

Auxiliary data sources

The world is now producing large amounts of data without active participation of persons (e.g., data from Internet searches, credit card transactions, retail scanners, and social media). There also are more and more digital administrative data (e.g., tax records, social security records, Medicare/Medicaid records, food stamp records, HUD records). Some of these data are not directly linked to the populations we study; some have item missing data problems; none offer a real replacement for our surveys, but many will be useful as auxiliary data sources.

While few of our surveys have used such data, many of our surveys are discovering that multiple modes of data collection (e.g., paper forms, internet, telephone interviews, face-to-face interviews), employed in one survey, can address some of their participation problems within current budgets (with administrative records considered a “mode”). Indeed, there is a consensus among survey methodologists that multi-mode surveys will be a key component of the future of statistical information. This consensus arises because of evidence of great diversity across people in whether and how they wish to supply answers to our questions. With mixed-mode surveys we can offer the cheapest mode to the willing respondents and reserve the more expensive modes for those less easily measured.

At the forefront of adapting survey methods

Over the decades the Census Bureau has been at the forefront of adapting its survey methods to changes in society. We have done this well; it is a proud history. We moved first to probability sampling with its powerful abilities for measures of uncertainty in estimates. Computers first enhanced surveys at the Census Bureau in data processing and tabulation. This was followed by computerization enhancing the estimation of sampling errors on the tabulated statistics and editing of the digital processed data. With the storage of data records on large files, we then enhanced sample designs by analysis of frame data and direct selection of digitized frame records. Finally, computerization entered the data collection step directly with computer assisted telephone interviewing computer assisted personal interviewing, and finally web survey software. During these years we were also developing enhanced methods of record-matching and estimation. Thus, at this point, almost all the steps of the survey process utilize computer assistance.

Further, and most recently we have acquired unique combinations of administrative records useful for statistical estimation, while researching their use under very strict confidentiality constraints. We have developed remarkably efficient record matching procedures. These allow us to consider novel combinations of those records with survey records, to reduce overall costs.

In short, we have many of the ingredients to adapt to the new changing environment. In my judgment, we need to assemble them to work in a new, coordinated way.

Our near-term future will call on us to reduce costs while continuing to produce the same data products. Simultaneously, there will be alternative statistics arising from other organizations that offer timelier, but less accurate estimates from new data sources. We cannot react to this future quickly, immediately changing our designs. Nonetheless, we must respond to this changing environment and we need to act now. Not doing this threatens our relevance for the future.

I’ll speak more about this in future blog posts.