My article on comparing American Community Survey (ACS) estimates over time was published earlier this month. It will take another 6 to 12 months before it’s slotted into a actual volume and issue, but until then it’s available online via the link in this citation:
Donnelly, F. P. (2026). The Feasibility of Comparing 5-Year American Community Survey Estimates Over Time. Journal of the American Planning Association, 1–14. https://doi.org/10.1080/01944363.2026.2709860
Unfortunately it’s not open access; if you don’t have a subscription, you can read the article in the publisher’s e-pub platform; this link only works for the first 100 clicks, then it’s kaput. I’ll eventually post the pre-published version of the article on this website, as I can freely share it. Much of the summary data from the research and the scripts I wrote for processing and analyzing the data are available on GitHub.
Background
The idea for the paper sprang from my work helping graduate students compare estimates over time when making maps in GIS. I wrote a post about these early experiences, where I described the fairly involved process we had to take to prepare and assess the estimates, only to discover that statistically they were not comparable (you couldn’t discern actual change from sample variability), or if they were comparable they were too imprecise to be useful. I wondered, how feasible is it to compare estimates over time – is going through all of these steps ever worthwhile?
To answer this question, I chose 25 detailed tables that captured a broad range of variables and represented basic, fundamental tables that many researchers would likely use. These tables included over 300 individual variables. For example, table B01001 included 49 variables: total population, total population that was male and that was female, and then 5-year age cohorts by sex (male population aged 0-4, male population aged 5-9, etc). I chose 19 geographic summary levels that represented the most essential geographies, from the nation down to census block groups. I compared estimates for these variables from the 2010-2014 to the 2015-2019 ACS for approximately 400,000 individual geographic areas; I chose these two 5-year periods as they do not overlap, and they are drawn from the same geographic vintage (2010) so I would not have to account for geographic boundary changes that occur with each decennial census.
ACS TABLES INCLUDED IN THE STUDY
| Table ID | Table Name | Variables | Observations (M) | % Total |
| B01001 | Sex by Age | 49 | 19.9 | 18.0% |
| B01002 | Median Age by Sex | 3 | 1.2 | 1.1% |
| B02001 | Race | 10 | 4.1 | 3.7% |
| B03002 | Hispanic or Latino Origin by Race | 21 | 8.5 | 7.7% |
| B05002 | Place of Birth by Nativity and Citizenship* | 15† | 2.8 | 2.5% |
| B07204 | Geographical Mobility in the Past Year* | 19† | 3.5 | 3.2% |
| B08006 | Workers by Means of Transportation to Work* | 17† | 3.1 | 2.8% |
| B11001 | Household Type (Including Living Alone) | 9 | 3.7 | 3.3% |
| B13002 | Women Who Had a Birth in the Past 12 Months* | 19 | 3.5 | 3.2% |
| B14001 | School Enrollment by Level of School* | 10 | 1.8 | 1.7% |
| B15002 | Sex by Educational Attainment | 35 | 14.2 | 12.9% |
| C17002 | Ratio of Income to Poverty Level | 8 | 3.2 | 2.9% |
| B19001 | Household Income (Dollars) | 17 | 6.9 | 6.2% |
| B19013 | Median Household Income (Dollars) | 1 | 0.4 | 0.4% |
| B19083 | Gini Index of Income Inequality* | 1 | 0.2 | 0.2% |
| B23025 | Employment Status | 7 | 2.8 | 2.6% |
| C24030 | Sex by Industry for Employed Civilians | 29† | 11.8 | 10.6% |
| B25002 | Occupancy Status | 3 | 1.2 | 1.1% |
| B25003 | Tenure | 3 | 1.2 | 1.1% |
| B25010 | Average Household Size of Occupied Housing Units | 3 | 1.2 | 1.1% |
| B25063 | Gross Rent | 24† | 9.7 | 8.8% |
| B25064 | Median Gross Rent (Dollars) | 1 | 0.4 | 0.4% |
| B25070 | Gross Rent as a Percentage of Household Income | 11 | 4.5 | 4.0% |
| B25077 | Median Home Value (Dollars) | 1 | 0.4 | 0.4% |
| B26001 | Group Quarters Population* | 1 | 0.2 | 0.2% |
* Data for block groups and tribal block groups is not published for this table
† Value represents a subset of the total variables in the table; some variables were excluded
I opted against randomly sampling the variables, which would have allowed me to statistically analyze the results and make “firmer” assumptions. The primary audience for the paper is a practitioner-based one, for whom it’s more meaningful to have complete results for all geographies, for all or most variables in tables that people commonly use. The variables I chose also represent relatively broad swaths of the population. A random sample would have yielded many variables for small populations from disparate tables, which would be of limited interest.
I calculated change and percent change for each pair of estimates from the two time periods. I applied the test for significant difference, to determine if the estimates from both time periods were truly different from another, or if any difference was likely the result of sampling variation. If the estimates were significantly different, I calculated the coefficient of variation, to determine how reliable the estimates were. I summarized the results by variable, table, and geographic summary level. Ultimately, I had a dataset with 110.5 million individual data points that represented change for a variable between two time periods for a given place, of which 107.5 million were distinct (i.e. variables included in more than one table were only counted once).
Results
The results were surprising. Approximately 90% of the time, estimates from the two time periods were not statistically different from each other, and thus could not be reliably compared to compute change over time. The estimates that were significantly different were of low precision approximately 90% of the time (a CV value higher than 30). In short, it’s usually not worth investing the effort that’s involved with comparing two, consecutive 5-year period ACS estimates over time.
That being said, there are exceptions. Not surprisingly, the larger a population was, or the larger the sample size, the more likely it was that the estimates would be comparable and reliable. For the nation as a whole, estimates were comparable about 94% of the time, and of those that were comparable, estimates were of reliable precision about 98% of the time (a CV value less than 30). Both of these values fall to 73% for states, to 28% and 34% for counties, and 12% and 6% for census tracts. When considering all estimates regardless of geography, variables that represented large segments of the population were also more comparable and reliable. For example, estimates for the total population, median home value, population not enrolled in school, and workers 16 years and older were comparable 25% to 30% of the time and reliable 22% to 33% of the time.
There were some interesting divergences. Estimates for workers who take various forms of public transit to work ranked low in terms of being statistically comparable, but of the estimates that were comparable, they tended to be the most reliable. While this population is small or zero in many parts of the country, it is large and highly clustered in dense urban areas, so when the values are comparable they are of high precision. So if you were doing research in New York City or Chicago, you would have good estimates to work with (but not so if you were looking at all tracts / places / ZCTAs in New York State or Illinois).
Beyond the relationship between population and sample size, areas or populations that experienced a large degree of growth or decline were also more likely to be comparable and reliable, as these large shifts outweighed any sample noise. These two time periods experienced dramatic growth in home value (adjusted for inflation) and decline in unemployment. Median home value was consistently ranked as one of the most comparable and reliable estimates, while the unemployed labor force was ranked as one of the most comparable (despite the fact that this population is relatively small). The map in the header of this post depicts census tracts in and around San Antonio, TX. Rapidly growing exurban tracts tended to have a higher number of reliable estimates (CV values less than 30).
As a data librarian, I was equally interested in the means to the end, as a large part of my mission is to support students and faculty with this process. During the data collection and analysis phase, I wrote a post that provided tips on working with big(ger) datasets when your computing power is limited to your laptop. I relied on a PostgreSQL database to efficiently store the data and generate the calculations, and I had to be thoughtful in how to use Python so that I didn’t run out of memory, eschewing popular tools that make things easy in favor of approaches that were more efficient, and using SQL and Python in tandem to do the work that each was best suited for.
While the ACS isn’t well suited for short-term historical comparisons, it remains a valuable and indispensable dataset. Its strength is that it gives us a detailed snapshot of our present circumstances, at a fine degree of geographic and temporal resolution that is unmatched by any other US government dataset. We should not confuse shortcomings with the data with calls to curtail or eliminate it. The current administration’s attempts to sabotage the federal statistical system are detrimental to our country’s economic and social well being. As data practitioners we need to support and protect our nation’s datasets, while collaborating with agencies to improve them.
You must be logged in to post a comment.