In case you missed it, the case for energy efficiency died last week.

Actually, it didn't — but if you looked at some headlines around the country you might have thought it had. A study assessing low-income weatherization in Michigan found the program was not cost effective and returned fewer energy savings than anticipated, and the resulting coverage put efficiency advocates on the defensive.

A headline from Forbes: “Is Uncle Sam Baking The Books On Energy Efficiency?”

The Wall Street Journal: “Energy-Efficiency Programs ‘Nudge’ Consumers in the Wrong Direction”

The Washington Post: “Study raises doubts about whether improving your home to save energy is worth its cost”

The research, led by UC Berkely and the Energy Policy Institute at the University of Chicago, examined the federal Weatherization Assistance Program (WAP) operating in Michigan, where residents are given about $5,000 in weatherization upgrades. Those can include furnace replacement, new insulation and weather stripping, but the results of the study surprised many: While the upgrades did reduce energy consumption by 10% to 20% each month, the lifetime savings amounted to only about $2,400 – roughly half the cost of the upgrades.

And ultimately, the energy savings were about half of what was anticipated.

“I was actually hoping and expecting we would find closer to the expected savings than we did,” said Meredith Fowlie, an associate professor at the University of California at Berkeley and lead author of the study.

“I think the spirit of this study was more: We need to do something about climate change,” she said. “Most climate change strategies feature energy efficiency very prominently so it makes sense to get out there and start identifying what energy efficiency measures are working and what are not.”

But the study found WAP, at least in Michigan, was pretty ineffective. That led to some “polarizing” media headlines, Fowlie notes, and in turn efficiency groups like Natural Resources Defense Council questioned some of the study's findings and methodologies.

Much of this, says Fowlie, misses the point.

“This is not a litmus test for all residential efficiency programs, certainly. That's a misinterpretation,” she said. “I've seen a couple of articles that seem to insinuate that based on this, energy efficiency is a write-off. Clearly that's by no means the conclusion we intended to reach, and it's incorrect.”

The study, which encompassed about 30,000 homes, targets a small subset of income-eligible homes in a very cold state. You can't extrapolate that to all energy efficiency, or even all weatherization, Fowlie said. But she also cautions that trying to toss out the results because they contradict so many other studies “would also be incorrect.”

“This is one study looking at Michigan, looking at weatherization assistance, and I think there are aspects of weatherization that are truly unique and different from other residential energy efficiency programs,” she said. “At the same time, I think there are similarities. ... I think there are reasons to think that what we're finding here could well be indicative of what you'd find elsewhere.”

“I personally am not confident in extrapolating what we've found into other settings," Fowlie said. "I'd want to do more work."

What the study found

The study returned three main findings, including what seems to be a low interest in the WAP program. 

Researchers took 30,000 Michigan households eligible for the program, and then assigned about a quarter of them to a group that received outreach and help in applying for the weatherization. The other households were free to apply but were not assisted or contacted.

“Aggressive encouragement,” as the study puts it, raised participation from less than 1% to 6% – but the outreach, involving thousands of home visits, phone calls and folllow-up appointments, raised costs more than $1,000 per weatherized household.

And while the weatherization efforts did reduce energy consumption, providing “a substantial assist to participating low-income households in the form of reduced energy bills," the upfront investment was ultimately "about twice the realized energy savings.”

“Further, the model-projected savings are roughly 2.5 times the actual savings,” the study found.

The third looks at a potential explanation: That the modest energy savings were the result of customers raising the thermostat, the so-called “rebound” when energy use rises as a result of greater efficiency.

“The paper fails to find evidence of significantly higher indoor temperatures at weatherized homes,” researchers wrote, citing the study's use of measured indoor temperatures and thermostat set points that we conducted in the study. “Though the existence of the rebound effect has been the subject of much debate … our study is the first to provide a direct field test of this phenomenon for a broad spectrum of residential energy efficiency investments.”

Computing the costs and benefits

One of the main critiques of the study has been what costs it included, and how it calculates benefit. Many weatherization efforts are not directly aimed at energy savings, so what costs you include can be a matter of contention.

Martin Kushler, a senior fellow at The American Council for an Energy-Efficient Economy, calls the study “narrowly focused and arguably conceptually flawed" in a recent blog post. “States commonly exempt low-income programs from the usual cost-effectiveness tests,” he wrote, and also focus on non-energy benefits as well as energy use reductions.

WAP makes repairs beyond energy-focused upgrades, including replacing wiring and installing smoke and CO2 detectors, and the idea is that the societal, comfort and health benefits – which can almost equal the energy savings – will offset those times when energy upgrades are more expensive than their direct benefits, Kushler explained.

Over at NRDC, Merrian Borgenson questions how the costs were calculated and argues researchers erred in including including basic non-energy improvements as well as costs beyond the incremental furnace upgrade costs.

“This accounting makes a huge difference,” Borgenson writes. “For example, about 1/3 of all participants in the program got a new furnace at a cost of over $3,100 each. If you assume that the 'incremental' efficiency premium on the furnace is about $400 (meaning the less efficient furnace option is $2,700), the cost of the efficiency portions these projects is far less - about $3,100 on average instead of $5,000.”

Fowlie says that's correct, the costs include equipment and labor costs, as well as other fairly non-negotiable items like replacing the wall you tore down to install insulation, or air-quality upgrades after construction.

“We use the WAP program/standard as our guide here,” she wrote in an email. “We use the audit data that was used to evaluate cost effectiveness of these measures ex ante. More precisely, we use the cost estimates (and measure lifetimes) that are generated by the WAP audit tool and used in the cost-benefit tests that inform program implementation.”

Study finds need for more study

Fowlie sort of laughs when she says it – “like an Onion headline” – but ultimately the real finding may be that more research is needed. The WAP program in Michigan is a very specific subset, she said, but the research, if nothing else, highlights the need to take a close look at what works and what does not.

As energy efficiency grows, it will increasingly show up in policy debates and emissions reductions strategies, Fowlie notes, pointing to the federal government's Clean Power Plan. “Another reason to be concerned about making sure energy savings estimates from efficiency are accurate, is that they could potentially play a role in compliance and implementation plans.”

“There are lots of reasons to be concerned that our measures of energy efficiency are accurate,” she said, and getting that right at first will be key to assessing how much of the climate fight efficiency can shoulder on its own.