When competing small area models are proposed for a particular set of estimates, they should be evaluated and compared not only by large-sample theory but also by their small-sample properties on real or realistic data. As an alternative to evaluations on data simulated directly from simple parametric models, we evaluate models by a simulation study tailored more closely to the source data that will be used in production. We use the 2007-2011 American Community Survey (ACS) 5-year unit-level sample data as a universe, which is likely to account for relationships among variables and other complexities that may not be reflected in a purely model-based artificial population. We then sample the population repeatedly with a design that mimics the ACS sampling design. In that sense this is a “design-based” simulation, although the simulation’s response mode and unit nonresponse behavior are model-based, and item nonresponse is not yet implemented. This simulation framework allows for comparison of different statistical inference approaches, with no method being inherently favored over others. Possible future improvements and potential drawbacks of this approach are also discussed.