Homes in the construction pipeline should be readily countable, unlike, for example, grains of sand in Salisbury Beach. But go try and figure out how many homes, exactly, Massachusetts municipalities permitted last year, and last decade. The numbers slip through your fingers like sand.

We don’t have a system in place to count them thoroughly. We lack concrete numbers on most dimensions of residential projects that are under permit review or under construction statewide.

The problem lies, first, in a system of self-reporting by city and town governments; some numbers get lost in their digital and paper files. Data reporting is a lift, so the ask has been left light, which is the second problem. The result is data with too many holes and too little detail.

Homebuilding is mission-central for home-short Massachusetts. Good data on home production could help us solve a lot of big problems, not just related to housing, but also transportation, public health, social mobility, and environmental sustainability, among other things. Fortunately, we can get good data, with some up-front work. Data reporting can be made automatic within the permit process.

For decades, our main system for estimating homebuilding rates has been the US Census Building Permits Survey (BPS). All 351 cities and towns of Massachusetts are supposed to report regularly their residential building permits to the Census. This year, we gained better production data from a new dataset, the US Census Address Count. Planning agencies and state agencies have supplemented these numbers with further data solicitations. For example, this year the Commonwealth’s housing office has been surveying cities and towns to count permits granted for accessory dwelling units specifically.

It sounds like a lot of data. But, for just one moment, let’s get lost together in the details.

How many homes did Malden build?

Back in 2018, I called and emailed municipal planners and building inspectors across 100 cities and towns of Greater Boston, asking for their multifamily permit tallies, covering the years 2015 through 2017. Malden’s redevelopment authority let me know the following nine projects got special permits:

2015: One project with 174 units

2016: Four projects with 119, 85, 69, and 86 units

2017: Four projects with 85, 247, 73, and 22 units

These were special permits, not building permits which are what the Census BPS tracks. Applicants that gain special permits still have to apply for building permits. Before construction, every home needs a building permit. Offhand, I don’t know if all nine projects ever got building permits. I didn’t ask. Maybe some projects never got built.

BUT, the Census BPS data tells us that for the entire decade, from 2010 to 2019, Malden granted a total of zero building permits for multifamily housing, defined as buildings containing three or more homes. The Census credits Malden with only 82 dwelling units—all in single-family and duplex homes—for the entire decade. This is certainly very wrong.

Go check out the excellent restaurants in Malden’s historic downtown and you’ll see the large new multifamily buildings winking back at the Census. Beyond its downtown, Malden also saw the large Rowe’s Quarry property built out with homes, among other industrial-to-residential redevelopments. How many homes does this all add up to? Well, the data aren’t handy, but... thousands?

Malden was the most extreme case of underreporting that I found. The data, overall, are not THAT far off. Still, many municipalities underreported permits; some overreported.

Let me pause here in my analysis of datasets to point out that the Census undercount does not mean “Aha, we've built enough homes. Malden’s got us covered.” Other indicators tell us there’s a serious region-wide shortage, such as low vacancy rates, high home prices, and a net domestic outmigration while the region’s jobs and economy were growing. Better data on the homebuilding pipeline would help us manage housing policy and catch up with demand.

Census Building Data Is Not Enough

One response would be to achieve better reporting of building permits to the Census. A good thing to do. But even if its data were perfect, the US Census BPS only offers blunt metrics—annual counts of building permits by city or town—not the detailed information we need to manage housing policy.

Here are some things BPS doesn’t give us that we’d like to know:

• Addresses of developments. If you’d like to find out how much new development is transit-oriented, for example, BPS cannot help you.
• Time from application to building permit. BPS doesn’t tell you how long permitting took, or whether discretionary permits or variances were involved. BPS data won’t help you evaluate policies aimed at reducing the costs of permitting processes.
• Status of construction and occupancy of permitted projects. If we want to know the universe of projects approved but potentially failing on finance, or getting held up by appeals, BPS data doesn’t help.
• Net number of units permitted. BPS doesn’t tell us about teardowns. Boston’s suburbs have been permitting construction of thousands of single-family homes. How many of those replaced existing homes? We don’t know.
• Bedroom counts, affordability restrictions, and other policy-relevant details about projects.
• Adaptive reuse. BPS doesn’t count new dwelling units added to existing buildings, for example old mills in Ludlow or Lowell (or in Millville, Millbury, Milford, or Millis). Mill redevelopments alone account for thousands of homes we want to know about.
• Connections to public sewer systems or other critical infrastructure. Officials investing huge public sums on city-sustaining underground conduits should have good data.
• Counts of permitted accessory dwelling units, or units in projects permitted under new MBTA Communities zoning, or in projects that receive “Comprehensive Permits” under the state’s Chapter 40B law, which lets some projects bypass local zoning. Such information is needed for evaluation of the state’s legalization of ADUs, MBTA Communities zoning law, and Chapter 40B.

Enter stage left: The Census Address Count, a new dataset the Census Bureau has released to help us understand actual rates of homebuilding, not just permitting. The dataset derives from US Postal Service address lists. New residential addresses mean new homes. Unlike the permit data, the addresses offer a way to track net numbers of new homes. When a single-family home is replaced with a new home, the permit count registers +1, while the address count doesn’t tick up. This dataset also captures new homes whose permit data got buried in files, untallied, like Malden’s apartments. And it shows production at the block level, not just townwide or citywide, like the Census BPS counts.

The new address data is helpful, for sure, but it comes to us on the tail end of the permitting pipeline that stretches down a long timeline. There’s still some imprecision in backing out construction levels from address listings. The unit counts do not distinguish the types of housing, for example single-family homes or multifamily buildings. And we don’t have enough detail to match up permitting datasets with postal address datasets. We should be able to access richer, timelier data to manage something as fundamental as housing.

MAPC Steps Up to Fill In

Enter stage right: The Metropolitan Area Planning Council (MAPC), a regional planning agency, digging for data and showing what’s possible. MAPC’s MassBuilds statewide building data map is an inspiration, a reminder that the universe of building data is not infinite. Try clicking around it, to see what I mean: https://www.massbuilds.com/map. If MAPC can get this far tracking permits, with limited resources and the help of interns, we know it is possible to assemble a more comprehensive picture.

MassBuilds includes detailed information on buildings either recently constructed or in the pipeline—their addresses, height, parking spaces, status, year completed, number of dwelling units, affordable units, retail, etc. The map’s data for the inner-core communities are in the best shape; entries get patchier further out from Boston. The website notes that the map is for buildings with 10 or more units in suburban and rural areas, and 20 or more units in urban areas, but the map does include some smaller projects.

MAPC is also throwing its arms around the data challenge in another way, to meet goals set by the Metro Mayors Coalition, which it staffs. Leaders of metro inner-core communities committed collectively in 2018 to gaining 185,000 new homes by 2030. MAPC analyst (housing data hero) Brandon Stanaway made it his mission to track actual progress toward the goal, not estimated progress. He’s been making calls and showing up at city halls to nail down exact numbers. It isn’t enough, he explains, to learn that special permits were granted; he wants to know the dates of building permits, and a lot more. He proves the task is possible.

This Is Where the State Comes In

Governor Maura Healey has announced that housing is her top priority. She formed the Executive Office of Housing and Livable Communities (EOHLC) to elevate housing management in her administration. And EOHLC has hired Tim Reardon, housing data guru formerly at MAPC, to run a housing data team for the state agency. EOHLC has also been tracking developments permitted under new MBTA Communities zoning, as well as new ADUs permitted since the state legalized ADUs statewide. The commitment, the staff, and the need are now aligned to launch a comprehensive system to manage homebuilding stats. And the technology is ready too.

Currently, our state-level datasets on homebuilding are collected through non-automated self-reporting and/or solicitation—separate tasks from the local processes of permitting home construction. This is true for the Census BPS, MassBuilds website, Brandon’s canvassing of Metro Mayors communities, and the EOHLC lists of ADU and MBTA-C projects. We should integrate the data systems into the permitting processes themselves, so data reporting is both required and automatic.

This brings us to the question of how municipalities now manage their building data.

Most municipalities use online permitting platforms. Plainville (a town between Wrentham and Attleboro), for example, just announced its launch of the OpenGov portal where applicants can submit building, zoning, and utilities permits, pay fees, and upload documents. OpenGov is the most common platform. Other platforms include PermitEyes, CitizenServe, and Accela. Many communities use Excel spreadsheets for permit tracking. A minority of communities still rely on paper files only. The thing is, even for communities using the same online platform, there is no standardization of information input about projects, like permit and construction status, number of units, number of parking spaces, etc. The Commonwealth right now has no way to pull metadata from the individual portals.

A superior approach would be for EOHLC to undertake the following steps:

1. Create standardized forms for data input (while allowing communities to tailor some aspects to match unique local requirements),
2. Get all communities on the same platform,
3. Gain direct access to the data contained therein,
4. Train municipal staff in the use of the systems, and
5. Audit the data.

I could be convinced we don’t need all communities on the same platform; the alternative would be to launch standard data fields across all platforms in use.

People who work with Massachusetts municipal governments may be laughing at me now. Three hundred fifty-one separate municipalities all have their own ways of doing things. That's a lot of unique bureaucratic systems to reform. Some municipal officials are still reeling from the local politics of recent state-required zoning reform. Moreover, many towns are so small they simply don’t have the staff or tech to manage online permitting.

I’m not naive, this is hard. But, first of all, this is the work of good government. This is the work of making sure everyone has safe, stable housing in livable neighborhoods.

Second, this is not the level of HARD that getting as-of-right zoning for multifamily housing adopted in MBTA-served communities has been, and that was baseline successful. Municipal data on homebuilding is public information; we aren’t dealing in medical privacy or trade secrets here. The number of construction projects in Massachusetts is middle school math. The data just need to be organized, managed, audited, and made accessible. It takes leadership, staffing, requirements, and budget allocation: things we’ve got, things we can mobilize.

Finally, for municipalities that are way too small to have the capacity to manage online permitting, the responsibility should be regionalized, overseen by entities with appropriate staffing and technology, such as regional planning agencies. In Berkshire and Franklin counties, there are already programs for cross-municipal inspections management.

But what about AI? Why invest in new data systems, when technology is evolving quickly? AI systems can’t yet manage this for us, and we shouldn’t sit on our hands waiting for AI to solve our pressing civic problems. Government needs to be nimble, which takes practice. We can start by mobilizing the best available tech to begin managing comprehensive housing data. As tech evolves, we can keep up as needed.

Data Management Is a Matter of Good Government

To summarize, Massachusetts should start collecting comprehensive data on the home construction pipeline to better manage the housing crisis. The data collection should be integrated into the permit process, and automated, to generate useful tallies.

The data would help us with things we care about, say public transit, the walkability of neighborhoods, safe sewage disposal, affordable construction, and production of enough housing for everyone to be well housed. Good data can also help us pivot as situations change; the world keeps changing.

For its top priority issue, the state should mobilize basic information systematically. It is not too hard to do.