The E2e weatherization study: generating more heat than light
ACEEE, July 2, 2015. Image credit: geralt
Methodology
First, parts of this study use an experimental research design called a randomized controlled trial, where researchers define a pool of eligible customers and randomly assign some of the eligible customers to a treatment group and some to a control group. In this way the treatment and control groups are identical. This is a powerful research approach but also has an Achilles heel: you cannot generalize the findings to households and programs that are different from those that you studied, because different types of households and different programs can have characteristics that would change the findings. In this case, they looked at weatherization of low-income households in a few counties in Michigan, and the results cannot be generalized to other states, to other programs, or to other income groups.
To illustrate the point, a study by Oak Ridge National Laboratory is looking at 30,000 homes that participated in the low-income weatherization program across the country (and not just one state).Preliminary results from this study find that, relative to a control group, benefits are on average about 1.7 times the cost––a very different result from the E2e study. While there are some differences between E2e’s approach and the quasi-experimental approach used by Oak Ridge, the big difference is in geographic coverage.
Missed benefits
Second, many energy efficiency programs and investments serve multiple purposes. To compare all costs against only some benefits is to bias the analysis. In this case the paper in one place seeks to assign all the costs to energy savings and in another place puts all the costs on carbon reduction; nowhere do they look at other important benefits of the program. To fairly evaluate such programs and investments, either all of the benefits need to be quantified and compared to the full costs, or if only energy benefits are quantified, then only the incremental costs of the energy efficiency improvements should be counted. Given that the weatherization program being studied is deliberately intended to achieve multiple benefits beyond simple energy savings, a more comprehensive assessment of benefits should have been done.