The Future of Producing Social and Economic Statistical Information, Part II

In my last post, I reviewed five observations. Because of changes in American society, 1) the Census Bureau’s methods of data collection are costing more money to produce the same statistical information, but 2) the demands are increasing for more statistical information from businesses, governments, and the public, and 3) there are new data collection technologies that are being invented constantly, 4) there are new sources of digital data from Federal program agencies, the internet, and economic transactions, but 5) in the medium run the Census Bureau is not likely to have more fiscal resources to take advantage of these. My conclusion: the current methods used in the Census Bureau are unsustainable in the medium run.

These observations suggest a way forward for this agency. In some areas, we have unique resources to achieve success; for others, we will need to work together in new ways.

1. The Census Bureau’s future must actively employ multiple modes of data collection from the American public and businesses. Some people prefer to talk to someone they can see face-to-face; others want to talk to someone over the telephone; others want to use the internet to answer survey questions at whatever hour of the day they wish; still others want us to use answers they’ve already provided to another agency. We need to adapt to these diverse desires. We need to make survey response as convenient as possible.
2. The resulting data sets may have more missing data, connected to the weaknesses of specific modes of data collection. For example, respondents using paper questionnaires sometimes fail to answer some questions that they would answer when asked by an interviewer. Each mode of data collection has strengths and weaknesses. We must be ready to use one mode to fill in gaps of another. When multiple modes are used in a single survey we can use each mode to bolster the weaknesses of another.
3. Prior to contacting our sample units, we will be ignorant about their mode preferences; we must be able to switch across modes in real-time during the data collection phase to produce timely estimates. Our lists of addresses provide little information about who lives in them. We know nothing about their preferences about completing our surveys. We will learn about their preferences only after sending them requests. To save taxpayer money, we need to switch to another mode when there emerges evidence that one mode is not effective. The faster we do this, the better.
4. Despite the multi-mode approach, we will not be able to gain self-reports from all sample members; statistical models will be used to produce accurate estimates when mode-switches to fill in missing data are judged cost-inefficient. To save taxpayer money, we need real-time data-based decisions about when it is more effective to cease efforts to measure a sample unit and instead use statistical models to account for the case in the final estimate.
5. Our final statistical estimates from these “swiss cheese” data sets must rely on new statistical modeling techniques to repair differences in item missing data and measurement properties across modes. Modern statistical modeling can improve the quality of statistical information. The Census Bureau can incorporate such tools to address the weakness of failing to measure all sample cases in our demographic and economic surveys.

These new designs thus will build on practices that are appearing within the Bureau already.

This future will require some changes in our key work processes. All the changes share the common theme of developing new design, data collection, and analysis methods to improve participation rates and efficiencies of surveys and censuses.

All of these appear to be within our reach, given pockets of developments that have occurred in different Bureau programs recently. We need to consolidate these practices for the benefit of all the programs as they move to mixed-mode designs. One area needing direct and quick attention is the collection and analysis of cost data so that wise tradeoff decisions about mode switches can be made. Another development area is management information systems permitting the real-time administration of mixed-mode surveys.

I have some ideas on steps that we might mount to address these issues. I’ll talk about them in a later post.