Matt Golden, Open EE
January 4, 2017
Billing analysis, Bayesian statistics, randomized control trials, phone polls — with the advent of EEMetering, smart meters, and pay-for-performance, one of the most important questions we have to answer as an industry is how do we measure energy efficiency?
Turns out, it depends on the purpose.
While the art of evaluation for regulated programs is well established, the movement of energy efficiency towards pay-for-performance markets and procurement requires a rethinking of how efficiency is regulated, measured and transacted.
In traditional programs, where incentives are set and paid in advance, savings measurement is an after-the-fact regulatory tool that doesn’t directly affect program operations or incentives. This means that using ex-post adjustments based on survey results and quasi-experimental control groups to understand program effects makes sense for regulators who need to understand additionality and net savings for the purposes of passing regulatory cost tests.
However, as efficiency shifts towards pay-for-performance and resource procurement, the role of measurement is shifting as well. What, when, and how we measure savings directly affects the alignment of incentives, program operations, and market design. And yet, we cling to traditional EM&V methods that are akin to putting an old engine in a new car.
Flipping the Script on Measurement and Verification
This doesn’t mean that traditional EM&V is suddenly wrong, or that implementing pay-for-performance should mean giving up on methodological rigor. It simply means that traditional EM&V methods and processes weren’t designed to provide information to markets. Efficiency measurement in the era of procurement and pay-for-performance has to take a market-first approach.
Market information comes in two essential flavors: quantities and prices. Typical program design sets incentives based on upfront savings estimates and then uses EM&V to adjust actual savings after the fact.
In pay-for-performance and EE procurement, the script is flipped: utilities say how savings will be counted and what value they will place on those savings, and market actors figure out what business models, technologies, and market approaches work best to maximize their cash flow. This means that the way savings is quantified suddenly matters for the organizations delivering it.
In this setting, measurement plays a dual role. We can use measurement to both quantify the savings that are being purchased, as well as to value those savings. We also have an additional variable at our disposal: pricing. Savings that would normally be subject to quantity adjustments after the fact can instead be adjusted through changes in the price or locked in for the term of a contract that is similar to a power purchase agreement. This creates manageable risk for the market by providing a reliable cash flow that is not subject to after-the-fact adjustment uncertainty.
In this new landscape, different approaches to measuring energy efficiency can play complementary roles depending on context.
For Building Owners: Measurement and Verification (M&V)
M&V is designed to quantify site energy savings for building owners and investors in energy efficiency projects on individual buildings. This approach calculates weather-normalized bill savings and allows for adjustments for both routine and nonroutine factors that impact savings, such as changes in occupancy or operating hours.
M&V measures the effectiveness of a retrofit and helps communicate the value of energy savings to building owners. Typically, it relies on establishing a baseline from past consumption and comparing it to energy use after a retrofit. This approach is best enumerated through the Efficiency Valuation Organization IPMVP protocols and is spelled out in contracts between building owners and efficiency providers.
For Markets and Aggregators: Normalized Metered Savings (EEMetering)
EEMetered savings quantify normalized metered site-based savings based on monthly or interval meter data before and after an intervention for portfolios of buildings. Rather than accruing to the building owner as bill savings, the benefits of efficiency accrue as a demand resource to load serving entities.
The purpose of EEMetering is to allow those entities to quantify the gross site-level impacts from energy efficiency. An EEMeter is designed to provide a consistent measure of the results of energy efficiency projects at a portfolio level that can be used to send a payable price signal to market actors for the value of the savings.
The definition of a meter is an instrument that “automatically measures and records the quantity of something, as in gas, water, miles, or time, when it is activated.” An EEMeter is a standard weights and measures for efficiency savings that is transparent to all parties and can be replicated and verified to produce that same output regardless of who is doing the metering or where.
Just as a metered cubic foot of natural gas is the same on any house, and on any day of the year, wherever it is located, a unit of efficiency as measured by the EEMeter is constant.
However, the value of the natural gas is not fixed. In fact, PG&E tests the BTU intensity of natural gas on a weekly basis and adjusts the price per cubic foot accordingly.
Similarly, an EEMeter tracks weather-normalized metered savings, wherever you are located, on any house, and on any day of the year in exactly the same way using a transparent calculation that can be verified and replicated.
However, the value of that energy savings is not fixed. Regulators or utilities can use EM&V to test the “intensity” or “net” savings. The value of normalized EEMetered gross savings is then adjusted through pricing and forward contracts to reflect the net intensity of the resource being delivered.
For Regulators and Procurement: Evaluation, Measurement, and Verification (EM&V)
EM&V is an effort to identify net savings that can be attributed directly to a set of interventions for regulated programs, and to identify net effects on load for utilities. In order to identify “net” savings, evaluators must utilize control groups, phone polls, and other techniques to differentiate savings associated with exogenous factors, like economic conditions, impacts of code, and consumer behavior, from savings that are attributable to the program.
The purpose of EM&V is to monitor the cost effectiveness of ratepayer-funded programs and to meet regulatory requirements for claimable savings.
When To Use Control Groups
In a regulated EM&V context, control groups allow evaluators to estimate the energy savings that are attributable to isolated effects of a program. However, both M&V and EEMetering are designed to estimate gross site savings from a building baseline that reflect savings at the meter to a customer, counting all normalized metered savings.
Both M&V and EEMetering are designed to estimate gross site savings from a building baseline based on methods that are transparent and replicable. Control groups are important when estimating “net” impacts of efficiency, but distort actual gross site savings to the point where they won’t agree with savings at the meter or demand to the utility.
Control group adjustments to measurement, often after the fact, creates uncertainty for markets that are antithetical to investment. This is because using control groups requires either population-level customer energy data and quasi-experimental design to attempt to create a match-pair a control group that simply cannot be replicated or predicted by market participants.
From Evaluation to Valuation
When applied to metered efficiency as a utility-procurable resource, “evaluation” simply becomes “valuation.”
The utility has an interest in identifying savings that have a net impact on the utility’s load shape. This version of net savings is simpler than regulated net-to-gross ratios used for cost tests, and simply controls for any naturally occurring savings already accounted for in a utility’s load forecast to avoid double counting. Utilities care about what the net effect will be on their load shape.
If a utility were to procure a megawatt of energy savings, but 20% of what was being EEMetered was already accounted for in the load forecast (savings from things like code or efficient appliances that are already baked into projects), then only 800K kWh would actually impact the utility’s load shape. In a resource procurement paradigm where an EEMeter is used to track normalized site savings, utilities would discount the amount of savings being bid based on a forecasted naturally occurring effect, resulting in a higher per-unit price for efficiency that has a high percentage of naturally occurring savings, favoring efficiency that has greater attributable impacts.
A change in price has the same net effect on per-unit cost as an adjusting of measurement. The advantage of adjusting the price is that it can be set through contracts up front, allowing markets to engage and invest with manageable risk.
The Right Type of Measurement Depends on the Use Case
M&V and EEMetering do not replace the need for EM&V to comply with requirements to derive net-to-gross ratios for regulators, and control groups have an important place. At the same time, not every EE measurement problem requires an EM&V answer. Rather than trying to apply a one-size-fits-all solution to a complex question, it’s important to use the right measurement approach for each efficiency use case.
- If one is attempting to quantify savings on individual buildings, then M&V based on IPMVP makes perfect sense.
- If a utility or aggregator is quantifying normalized metered savings at a portfolio level, then an EEMetering approach is the way to go.
- If a regulatory entity requires a net-to-gross ratio to apply a cost-effectiveness test, or if a utility is valuing savings as a net resource, then EM&V and control groups will be necessary.
As an industry, we need to strive for more quantitative, replicable, and open methods across all three use cases, and keep in mind that there will never be a single solution that works in every case.