Several chapters of The Vanishing Automobile mentioned problems with the 1997 Natural Resources Inventory (see pp. 156-157, 408, and 438-439). After a 1999 release of the inventory with great fanfare as proof that urban sprawl was accelerating, the federal government withdrew it in early 2000 for correction of data errors.
The U.S. Department of Agriculture (USDA) has now released a revised report in pdf format. Individual tables can also be downloaded in Excel format. Although the revision is dated December 2000, it wasn't actually released until January, 2001.
The report explains that the revision was needed due to a computer programming error. "Sample points that changed into developed land were affected in a systematic manner (not randomly), so that the corrections lowered all estimates of 1997 developed land" (p. 1). In short, the previous report systematically overestimated the amount of developed land.
The revised data indicate that the original release overestimated developed lands by more than 7 million acres. The new data reduce the amount of developed land in 1997, but they also reduce the estimates of developed land for previous years as well. Based on this, USDA still claims that the pace of development has accelerated in recent years.
"In the 5-year period between 1992 and 1997," says a report summary, "the pace of development (2.2 million acres a year) was more than 1-1/2 times that of the previous 10-year period, 1982-92 (1.4 million acres a year)." Yet there are significant reasons to doubt the inventory's reliability on this point.
To summarize, 1997 NRI estimates of changes in land use are suspect for several reasons.
The Natural Resources Inventory (NRI) does not measure every acre in America. Instead, it is based on 800,000 sample plots distributed on non-federal lands in nearly every state. (While USDA says that it sampled every state, it reports no data for Alaska.)
With nearly 1.5 billion acres of non-federal land in the U.S. (outside of Alaska), each of the 800,000 inventory plots represents about 1,865 acres. This is called the plot blow-up factor: If a plot lands in a city, it represents roughly 1,865 acres of urban land; if it lands in a corn field, it represents roughly 1,865 acres of crop land.
This is oversimplified. The Natural Resources Inventory uses a stratified sample, which means that lands are divided into strata and enough data are collected for each stratum to be reliable. This means that the plot blow-up factors for each stratum (such as, perhaps, croplands and developed lands) will be different. It was an error in these different blow-up factors that forced USDA to revised the 1997 data. But we can use 1,865 acres per plot as a rough estimate.
A simple rule of thumb in a sampling inventory is that more plots mean more reliable results. For example, data for the U.S. as a whole are more reliable than for individual states. The NRI says that there are about 327 million acres of cultivated cropland in the U.S. The USDA estimates that this is likely to be correct within plus or minus 1.8 million acres or 0.6 percent (appendix table 1; for statisticians, this is the margin of error or standard error at the 95 percent confidence level).
However, Rhode Island is estimated to have only 4,600 acres of cultivated cropland, and the margin of error for this number is 67 percent. In other words, Rhode Island could have anywhere from 1,500 to 7,700 acres of cultivated cropland. Apparently, only a handful of plots in Rhode Island were cultivated cropland, which isn't enough to be very reliable.
Although there are more than three times as many acres of cultivated cropland as developed lands, the report claims that the margins of error for estimates of developed land are only a little greater than for cultivated cropland -- 0.9 percent for the nation overall, and no more than 20 percent in any specific state. Perhaps USDA collected a few more plots per acre of developed strata in order to keep data more reliable.
USDA has conducted the Natural Resources Inventory (NRI) every five years since 1982. People actually visited every single plot for the 1982 inventory. The 1987 and 1992 inventories visited only a sampling of plots and relied on aerial photos for the rest. Though some plots were revisited in 1995, the 1997 inventory relied almost entirely on aerial photos.
"Careful consideration was given to assure that 1997 NRI data elements were consistent with definitions, categories, and concepts from previous inventories," says the NRI report. "The same sample used for the 1992 NRI was used for 1997 data collection. This enables analysis of trends extending over 15 years (1982, 1987, 1992, and 1997)" (p. 3)
However, the report warns that "All comparisons for two points of time should be made using the new 1997 NRI database. Comparisons made using data published for the 1982, 1987, or 1992 NRI may produce erroneous results, because of changes in statistical estimation protocols, and because all data collected prior to 1997 were simultaneously reviewed (edited) as 1997 NRI data were collected" (pp. 4-5).
In other words, the 1997 report recalculated the numbers for 1982, 1987, and 1992. No doubt USDA inventory specialists attempted to be as accurate as possible in all four inventories. But the 1997 workers threw out all the numbers calculated in previous years.
Curiously, this allows us to make an alternative estimate of the reliability of the data. I have 1987 inventory data (published in the 1990 through 1994 editions of Statistical Abstracts of the United States) and 1992 inventory data (which was on line until it was removed when the 1997 data were posted). Since all of the inventories present numbers from the previous inventory years, we can compare how much land the 1987 inventory said was developed that year with the amount the 1992 and 1997 inventories said was developed in 1987. Since these data are not in the current NRI report, I have posted them in your choice of Excel or tab-delimited format.
The table below shows these numbers for the U.S. as a whole and for Minnesota, as an example of one state. For the U.S., the amount of developed land in 1987 as reported in 1992 was 9 percent greater than as reported in 1987. Although USDA claims that its estimate of developed land in 1997 is reliable within plus or minus 0.9 percent, the actual variation in USDA's own estimates for 1997 is ten times that amount.
Amount of Developed Land in 1987
Total U.S. Minnesota
1987 77,285 2,136
1992 84,569 2,309
1997 81,686 1,941
1997r 79,086 1,843
Margin of error 0.9% 5.4%
Variation 9.4% 25.3%
The USDA claims that the margins of error for the revised 1997 (1997r) data are low, but the variations between the highest and lowest estimates of the amount of land that was developed in 1987 are much greater than those claimed margins of error.
USDA's estimate of the reliability of the amount of developed land in Minnesota is plus or minus 5.4 percent. Yet USDA's 1992 estimate of the amount of developed land in 1987 is 25 percent greater than its revised 1997 estimate. In most states, the variation of USDA's own estimates of land developed in 1987 is two to six times greater than the margin of error USDA claims for land developed in 1997.
Why should we accept USDA's 1997 estimates of reliability when USDA's previous estimates using similar statistical techniques varied so much more? Given that earlier inventories relied more on field checks while the 1997 inventory relied almost completely on aerial photos, the latter inventory is likely to be less, not more, reliable.
The revised 1997 NRI claims that the amount of developed land in the U.S. increased by 24 percent between 1987 and 1997. But more than a third of this increase is represented by variations in USDA's past estimates of the amount of land that was developed in 1987. The actual increase in developed land may have been only 16 percent or even less.
In Minnesota, the amount of developed land supposedly increased by 19 percent from 1987 to 1997. But the variation of estimates of land that was developed in 1987 is 25 percent, more than the supposed increase. With a variation that large, estimates of the changes in developed land are meaningless.
USDA tried to measure enough plots to get a high level of statistical accuracy within each stratum. But few plots in each stratum changed uses in the past ten years. So the reliability of data on changes in land use is much lower. The 1997 revised report admits this in appendix 1.
For the U.S. as a whole, the report estimates that the margin of error for the change in cultivated croplands is 8.8 percent, or nearly 15 times greater than the error for cultivated croplands as a whole. In fourteen different states, the margin of error for the change in cultivated croplands is more than 100 percent. In Vermont and North Dakota, it is more than 300 percent; in Maine and Montana, more than 500 percent. In Wyoming, it is 2,743 percent!
Aside from these problems, the revised inventory still does not answer questions we had about the initial 1997 report, including:
These questions were already raised in The Vanishing Auto, but the revised report does nothing to answer them.