Federal statistics have a rather schizophrenic view of survey inference. The preferred approach for inferences about descriptive population quantities from large surveys is the so-called “design” or “randomization” based approach, where population values are treated as fixed and uncertainty is based on probabilistic selection of the sample. This approach is widely attributed to the famous 1934 paper by Jerzy Neyman, and the Census Bureau was a pioneer in putting it into practice, led by Morris Hansen and others.

The design-based approach does not work well for situations where the survey information is limited and the so-called “direct” estimates it produces are noisy, like small area estimation. It also falls down for problems such as missing data where response cannot be considered random. An alternative is the modeling paradigm, which bases inference on a statistical model for the population values. It is also widely practiced at the Bureau. Indeed, economists and other social scientists are trained as modelers and are often somewhat mystified by the design-based approach. This leads to controversies over such matters as how and when design weights need to be included in the analysis.

I favor the approach known as “calibrated Bayes,” where all inferences are based on Bayesian models, but models need to be chosen that have good repeated sampling properties. To me everything is modeling, but some models make limited assumptions and lead to answers similar to “direct” design-based approaches, others make stronger modeling assumptions to allow useful estimates for situations where direct estimates are too noisy. I have argued that this approach is more unified than the existing paradigm and provides a valuable way forward for official statistics. See “Calibrated Bayes, an Alternative Inferential Paradigm for Official Statistics” for more details.

This may seem a rather abstruse topic, but it’s fun to think about, and fundamental since it underlies nearly everything we do. What’s your view?

Roderick Little, Associate Director for Research and Methodology and Chief Scientist