• Survey Design
• Data Collection
• Compilation of Data
• History of Survey Program

Back to top     

The target population for the Manufacturers’ Unfilled Orders (M3UFO) Survey was defined as all domestic companies with five or more employees that have one or more manufacturing establishments classified in any of the 2017 North American Industry Classification System (NAICS) industries that make up the 41 M3 industry categories believed to maintain unfilled orders. Six of these industry categories are each broken into two categories, a defense component and a nondefense component. Therefore, the M3UFO survey  covered 47 M3 industry categories. These 47 industry categories encompass the following major groups in manufacturing:

• Primary Metals
• Fabricated Metal Products
• Machinery
• Computers and Electronic Products
• Electrical Equipment, Appliances, and Components
• Transportation Equipment
• Furniture and Related Products
• Miscellaneous Products

Back to top

The sampling frame for the most recent full M3UFO survey, the 2019/2018 M3UFO survey, was constructed using information extracted from the Census Bureau’s 2018 Business Register and updated with measure-of-size (shipments) data from the 2017 Economic Census. This sample frame represented roughly 70,000 manufacturing companies located in the United States having five or more employees and with establishments classified in any of the 47 M3 industry categories described above.

Back to top

The sampling unit for the M3UFO survey is the company. Sample estimates are produced for the M3UFO survey from a sample of roughly 6,000 companies.

Back to top

The initial sampling frame for the M3UFO survey was comprised of company by M3 industry category records.  Companies with activity in multiple industry categories have multiple records in this initial frame.  Each company by industry category record represents the combined activity of all establishments classified within that industry category.

The sample for the M3UFO survey is selected using a probability-proportional-to-size (pps) sample design based on the assigned measure of size.  Sampling is controlled at the M3 industry category level and each company is assigned a probability of selection based on its respective measure of size for each industry in which it has activity.  Therefore, each company by industry category record in the initial frame is assigned a probability of selection that is commensurate with its relative importance (based on value of shipments) within the respective industry category.  Consequently, a company with activity in multiple industry categories is assigned multiple probabilities of selection.  The final probability of selection assigned for each company is the maximum of these industry-based selection probabilities.  We also impose a minimum probability rule, which results in a maximum sample weight (inverse of the minimum probability).  This minimum probability rule reduces the risk of exceedingly large sample weights that can adversely impact sample estimates and variances.  For the M3UFO survey, the minimum probability is 0.01, resulting in a maximum sample weight of 100.  There are also some companies specified by the M3UFO survey analysts to be included in the sample as arbitrary certainties.  These companies are flagged up front and included in the sample with weights of 1.

The sample is allocated across the 41 unique M3 industry categories that are in scope of the M3UFO survey based on industry category priorities.  Twenty-three of the 41 industry categories are designated as high-priority industries, so sampling is controlled under tighter CV constraints for these industries.  Ultimately, the sample selection process for the M3UFO survey results in a final sample of 6,000 companies.

Back to top

Every 5 years (implemented two years after each Economic Census).

Back to top

Since the M3UFO survey sample is redesigned every five years, there is a need for sample maintenance in the intervening years.  Each survey cycle, companies are lost through sample attrition, so something needs to be done in order to maintain the desired total sample size of 6,000 companies.  Therefore, in each intervening year, a birth sample is selected in order to accurately reflect the universe for a given survey year and to offset the effects of this sample attrition each survey year.  The target number of companies selected in each annual birth sample is determined by the company attrition rate from the previous survey cycle.  Similar to the full sample selected every five years, each birth sample is allocated across the 41 unique M3 industry categories included in the M3UFO survey based on the respective company attrition rates in each industry category.  This ensures that respective industry category sample sizes are maintained, while also maintaining the total sample size of 6,000 companies.

Back to top

Back to top  

Unfilled orders and value of shipments for current and prior years are collected.

The survey questions can be found here.

Back to top

Unfilled orders and value of shipments (company must report both data items in order to be counted as a respondent).

Back to top

Mandatory

Back to top

The Census Bureau mailed the M3UFO survey form each year in April, requesting that the respondents return the form within 30 days. The first follow-up letter was mailed 30 days after the initial mailing in an attempt to obtain data from companies that did not respond. Thirty days after the follow-up letter, a telephone follow-up was conducted of the largest nonrespondents.

Back to top

Company (data reported for each M3 industry category in which company has activity).

Back to top

N/A

Back to top

Back to top    

Standard edits, macro diagnostics, faux estimation

A series of standard edit failure clauses are run throughout the year to determine outlier units.  These outlier units are manually reviewed by analyst(s) and resolved manually if deemed necessary.

Each industry category’s annual shipments totals and unfilled orders totals are reviewed on the macro level after data collection closeout.  Problem units within each industry category are manually reviewed by analyst(s) and corrective actions are taken if necessary.

Faux estimation is run after the completion of the standard edits review and the macro level diagnostics.  It compiles the annual shipments to unfilled orders ratio for each industry category while outputting the units with the greatest impact per industry category.  Analyst(s) review the output to determine if corrective action and/or industry justification is needed for clearance and dissemination. 

Back to top

Nonresponse is defined as the inability to obtain requested data from an eligible survey unit. Two types of nonresponse are often distinguished. Unit nonresponse is the inability to obtain any of the substantive measurements about a unit. In most cases of unit nonresponse, the Census Bureau was unable to obtain any information from the survey unit after several attempts to elicit a response. Item nonresponse occurs either when a question is unanswered or unusable.

Back to top

In some instances, total receipts from the Business Register are used to impute for annual value of shipments for missing/incorrectly reported values.  Analysts use annual reports (10-Ks) that are treated as other sources of reported data.

Back to top

N/A

Back to top

Company (data tabbed in each M3 industry category for which data reported by company).

Back to top

Unfilled orders estimates are produced for each of the 41 unique in-scope M3 industry categories using a ratio estimator.  For six of these industry categories, the unfilled orders estimates are broken into defense and nondefense components based on allocations obtained from actual M3 data.  Therefore, unfilled orders estimates are actually computed for 47 M3 industry categories.  Horvitz-Thompson (HT) weighted estimates of unfilled orders and value of shipments are derived for each industry category using only companies in the M3UFO sample that have reported data for both data items, provided that shipments data are greater than zero.  The sample weight assigned to each company is the inverse of its probability of selection.

Ratio estimates of unfilled orders are produced for each industry category by constructing a ratio of the weighted estimate for unfilled orders to the weighted estimate for value of shipments (UO/VS) from the M3UFO survey and applying this ratio to the corresponding Annual Survey of Manufactures (ASM) or Census of Manufactures total value of shipments estimate.  So, the unfilled orders estimates for each industry category produced from the M3UFO survey are derived using the following formula:

[(HT estimate of unfilled orders) / (HT estimate of value of shipments)] * ASM or Census total value of shipments estimate

Similar to the M3 monthly link-relative estimation procedure, when a large company reports an unusual ratio of unfilled orders to shipments for a given industry category and strongly influences the overall industry ratio, the company’s data are excluded from the UO/VS ratio for that industry.  However, the company’s unfilled orders data are included in the final unfilled orders estimate for the industry category.

Back to top

The sampling error of an estimate based on a sample survey is the difference between the estimate and the result that would be obtained from a complete census conducted under the same survey conditions. This error occurs because characteristics differ among sampling units in the population and only a subset of the population is measured in a sample survey. The particular sample used in this survey is one of a large number of samples of the same size that could have been selected using the same sample design. Because each unit in the sampling frame had a known probability of being selected into the sample, it was possible to estimate the sampling variability of the survey estimates.

Common measures of the variability among these estimates are the sampling variance, the standard error, and the coefficient of variation (CV), which is also referred to as the relative standard error (RSE). The sampling variance is defined as the squared difference, averaged over all possible samples of the same size and design, between the estimator and its average value. The standard error is the square root of the sampling variance. The CV expresses the standard error as a percentage of the estimate to which it refers. For example, an estimate of 200 units that has an estimated standard error of 10 units has an estimated CV of 5 percent. The sampling variance, standard error, and CV of an estimate can be estimated from the selected sample because the sample was selected using probability sampling. Note that measures of sampling variability, such as the standard error and CV, are estimated from the sample and are also subject to sampling variability. It is also important to note that the standard error and CV only measure sampling variability. They do not measure any systematic biases in the estimates.

The Census Bureau recommends that individuals using these estimates incorporate sampling error information into their analyses, as this could affect the conclusions drawn from the estimates.

The variance estimates for the M3UFO survey incorporate the variance attributable to the unfilled orders and value of shipments estimates produced from the M3UFO survey, as well as the variance attributable to the ASM total value of shipments, which is also part of the final unfilled orders estimate.  Rosén’s variance is computed for the components of the UO/VS ratio derived from the M3UFO survey, which incorporates the variance for the unfilled orders estimate, the variance for the value of shipments estimate, as well as the corresponding covariance between these two data items.  Rosén’s variance is combined with the ASM variance to calculate the final variance estimate using the Taylor Series approximation.

Back to top

The sample estimate and an estimate of its standard error allow us to construct interval estimates with prescribed confidence that the interval includes the average result of all possible samples with the same size and design. To illustrate, if all possible samples were surveyed under essentially the same conditions, and an estimate and its standard error were calculated from each sample, then:

1. Approximately 68 percent of the intervals from one standard error below the estimate to one standard error above the estimate would include the average estimate derived from all possible samples.
2. Approximately 90 percent of the intervals from 1.645 standard errors below the estimate to 1.645 standard errors above the estimate would include the average estimate derived from all possible samples.

In the example above, the margin of error (MOE) associated with the 90 percent confidence interval is the product of 1.645 and the estimated standard error.

Back to top

Nonsampling error encompasses all factors other than sampling error that contribute to the total error associated with an estimate. This error may also be present in censuses and other nonsurvey programs. Nonsampling error arises from many sources: inability to obtain information on all units in the sample; response errors; differences in the interpretation of the questions; mismatches between sampling units and reporting units, requested data and data available or accessible in respondents' records, or with regard to reference periods; mistakes in coding or keying the data obtained; and other errors of collection, response, coverage, and processing.

Although no direct measurement of nonsampling error was obtained, precautionary steps were taken in all phases of the collection, processing, and tabulation of the data in an effort to minimize its influence. Precise estimation of the magnitude of nonsampling errors would require special experiments or access to independent data and, consequently, the magnitudes are often unavailable.

The Census Bureau recommends that individuals using these estimates factor in this information when assessing their analyses of these data, as nonsampling error could affect the conclusions drawn from the estimates.

Two measures or indicators of nonsampling error are calculated for the M3UFO survey. The first measure is the unit response rate (URR). The URR is computed as the number of respondents (R) divided by the number of companies eligible for data collection (E) plus the number of companies for which eligibility cannot be determined (U). This rate is then multiplied by 100 to yield the percentage. So, the formula for calculating the URR is as follows:

URR = [R/(E+U)] * 100

The URR for the 2018 M3UFO survey year was roughly 80%.

The second measure or indicator of nonsampling error is the coverage rate. The coverage rate is calculated by dividing the total weighted measure of size (value of shipments) for M3UFO respondent companies by the total weighted value of shipments for all active companies in the M3UFO sample. Again, this rate is multiplied by 100 to yield the percentage. The coverage rate for the 2018 M3UFO survey year was almost 89%.

Back to top

Benchmarking adjusts the level of a given series to the levels (referred to as benchmarks) from a less frequent data source that is considered to be of better quality, while attempting to minimize revisions to the period-to-period changes from the more frequent series.  In this way, we produce consistent time series and attempt to reduce the effects of sampling and nonsampling errors in the original, more frequent series.

Unfilled orders data are not collected in the ASM or the Census of Manufactures, so in order to obtain more accurate monthly M3 estimates of unfilled orders, which are also used in deriving M3 estimates of new orders, the M3UFO survey is conducted annually to serve as the source for benchmarking monthly M3 unfilled orders data.  The M3 benchmarking process occurs on an annual basis.

Back to top

Disclosure is the release of data that reveals information or permits deduction of information about a particular survey unit through the release of either tables or microdata. Disclosure avoidance is the process used to protect each survey unit’s identity and data from disclosure. Using disclosure avoidance procedures, the Census Bureau modifies or removes the characteristics that put information at risk of disclosure. Although it may appear that a table shows information about a specific survey unit, the Census Bureau has taken steps to disguise or suppress a unit’s data that may be “at risk” of disclosure while making sure the results are still useful.

As stated above, the M3UFO estimator for unfilled orders is a ratio estimate derived by constructing a ratio of weighted unfilled orders data to weighted value of shipments data from the M3UFO survey and applying this ratio to the corresponding ASM total value of shipments estimate. Disclosure avoidance is done for the M3UFO survey using cell suppression. Cell suppression is used for the ASM as well, so all of the ASM total value of shipments estimates incorporated in the M3UFO estimates for unfilled orders have also been subjected to cell suppression.

Cell suppression is a disclosure avoidance technique that protects the confidentiality of individual survey units by withholding cell values from release and replacing the cell value with a symbol, usually a “D”. If the suppressed cell value were known, it would allow one to estimate an individual survey unit’s too closely.

The cells that must be protected are called primary suppressions.

To make sure the cell values of the primary suppressions cannot be closely estimated by using other published cell values, additional cells may also be suppressed. These additional suppressed cells are called complementary suppressions.

The process of suppression does not usually change the higher-level totals. Values for cells that are not suppressed remain unchanged. Before the Census Bureau releases data, computer programs and analysts ensure primary and complementary suppressions have been correctly applied.

For more information on disclosure avoidance practices, see FCSM Statistical Policy Working Paper 22.

The Census Bureau has reviewed this data product for unauthorized disclosure of confidential information and has approved the disclosure avoidance practices applied.  (Approval ID:  CBDRB-FY21-185)

Back to top

As mentioned above, the monthly M3 survey benchmarks shipments and inventories data to the ASM or the Census of Manufactures. However, we cannot benchmark monthly M3 unfilled orders data to the ASM or the Economic Census since neither of these surveys collects unfilled orders data. Therefore, the M3UFO survey is conducted annually to provide these annual unfilled orders benchmarks.

Historically, the M3UFO survey has not always been conducted on an annual basis. There were many years in which we did not have the resources to conduct such an annual survey, so prior collections of annual unfilled orders data were only conducted when the budget allowed. Therefore, we were only able to conduct the M3UFO survey for survey years 1986 and 1999, before establishing the annual collection of unfilled orders data we now have in the current M3UFO survey.

The M3 staff conducted a pilot version of the annual M3UFO survey in 2008 (collecting 2006 and 2007 data), but with limited resources. At that time, forms were keyed by hand in a database and very few edits were applied. Results from this pilot survey were never published because estimates were deemed to be unreliable. The first actual production version of the current annual M3UFO survey was conducted in 2010. This survey collected 2008 and 2009 shipments and unfilled orders data. We derived unfilled orders to shipments ratios obtained from data collected on the M3UFO survey and applied these ratios to the respective ASM shipments data to produce the desired annual unfilled orders benchmark estimates. Annual unfilled orders benchmarks are produced from the M3UFO survey for 47 industry categories encompassing the following major groups in manufacturing:

• Primary Metals
• Fabricated Metal Products
• Machinery
• Computers and Electronic Products
• Electrical Equipment, Appliances, and Components
• Transportation Equipment
• Furniture and Related Products
• Miscellaneous Products

The first M3 Benchmark report to include unfilled orders estimates produced using these 2008 and 2009 M3UFO survey data was released on May 13, 2011. Since then, the M3UFO survey estimates have continued to be published in the M3 Benchmark report in May of the year following collection. The monthly M3 estimates are based on a relatively small panel of domestic manufacturers and reflect primarily the month-to-month changes of large companies. There is a clear need for periodic benchmarking of these M3 estimates to reflect the manufacturing universe. The Economic Census or the ASM provides annual benchmarks for the shipments and inventories data in the monthly M3 survey. The Manufacturers’ Unfilled Orders (M3UFO) Survey provides the annual benchmarks for the unfilled orders data.

A sample redesign for the M3UFO survey is conducted every five (5) years. The last sample redesign of the M3UFO survey was conducted in 2020 for the collection of 2019 and 2018 data.
 

Back to top