The Promise of Performance
Written by Carmen Best, Director of Policy and Emerging Markets, OpenEE, June 6, 2018
Performance is the buzzword of efficiency today.
Supporters seem to believe that the use of statistically-derived performance metrics may represent the greatest innovation in energy efficiency since the invention of the compact fluorescent light bulb. Pay-for-performance models tied to meter-based energy savings have already been implemented in California (D.18-01-004 p.43) and are being pursued in New York (p.35) and Oregon. The common thread between them is that that they all use meter-based savings as the transactional basis of performance.
As long as these approaches are based on consistent, transparent measurement and verification, and honest accountability, they have the potential to reduce risk across the whole delivery system, allowing efficiency to compete in the distributed energy resources market, and finally bring it to scale. This case for pay-for-performance efficiency is gaining momentum across the country, and the structures are being put into place to make it a reality.
While pay-for-performance efficiency holds enormous promise, there are also real risks if it’s not done right. Many variations of performance-based efficiency have been tried in the past, and many have failed. Getting it right this time around may represent the last chance for efficiency to survive as part of a competitive, distributed energy resource mix.
For those of us who want the new paradigm to succeed, we must remember the drivers of past failures. It is critical that our latest efforts not repeat the mistakes of the last generation.
If we go down this road, we must stick to our performance commitments. Agreement on methods is not enough; transparency and replicability are essential. Moreover, we must provide both sides of the market access to data to track their performance and be held accountable. Most of all, we must enforce the rules that we agree upon. If we find ourselves unable to hold underperformers accountable, we will once again have deemed energy efficiency an irrelevant resource.
Learning from history
As we dive into this new and exciting paradigm, I have also had some history on my mind.
I spent the better part of my career working with the California Public Utilities Commission as they navigated one of the worst performance debacles in recent energy efficiency history. While this experience is tied to a shareholder incentive mechanism, rather than a program implementer incentive, the Risk Reward Incentive Mechanism adopted in 2005 is still offering lessons for performance.
One promise of this policy was to “protect ratepayers by ensuring that the utilities bear risks related to the performance of their efficiency programs,” which sounds a lot like the current case for pay for performance. A review completed in 2014 found that the policy had been largely unsuccessful in meeting this goal. A variety of reasons accounted for this, but the biggest ones stemmed from controversies around evaluation results and lack of enforcement. The end result was an ugly, public battle that went on for 10 years. Given my proximity to this battle and my role in addressing the mistrust left in its wake, I have kept it at the top of my mind as I consider this new wave of performance-based regulatory opportunity.
I am still ready to go to the mat for performance as the solution for the future of energy efficiency as long as we keep three critical lessons in mind. First, agreement on methods is not enough–methods must be transparent and replicable for all parties. Second, rules have got to be clear and enforced. Finally, all parties must have access to information, analysis and knowledge of their own performance to allow for mutual accountability.
Agreement on methods is not enough.
The run up to the regulations adopting the Risk Reward Incentive Mechanism included the development of an evaluation framework and evaluation protocols for the implementation and execution of the methods to a prescribed level of rigor, as well as a 20 percent cushion on performance of total goals. Even with all of this, the dispute over the results and the application of methods was still one of the most contentious portions of the incentive mechanism when performance didn’t meet the minimum threshold. Accusations of “moving goal posts” won the short-term argument, the evaluation results held for the long term decision.
All actors in the system need to know how they will be measured and judged to assume responsibility and maintain accountability, and they need the ability to monitor progress as things roll out. Methods need to be consistent, but also transparent and replicable. To be replicable, the nitty-gritty execution of the method must be accessible. If there is too much room for interpretation, methods will be vulnerable to persistent attack, usually from the party that didn’t get a favorable outcome. Energy efficiency requires upfront agreement, because there is no other ground truth.
Allowing energy efficiency savings calculations to be endlessly disputed make scaling the system difficult, impedes efforts to create and exchange a commonly understood unit of savings, and sacrifices the general sanity of all who are involved.
Citing a generic method is not enough. Publishing and following evaluation frameworks and protocols is not enough. The method and the execution of that method must be accessible at the design phase, foundational to deployment of the effort, and serve as the contractual basis for payment. There can be no shortcuts here. Having qualified evaluators on either “side” is not enough, as that only ensures the debate will be robust and last forever.
An agreed upon execution of a method, revealing all the decision points for calculation, needs to be the foundation for the performance agreement and the contractual execution.
Be clear about the rules and be ready to enforce them.
The failed incentive mechanism in California did have clear rules, but at the end of the day the Commission was not ready to enforce them. While multiple versions of “why” still abound, the Commission later committed to enforcing it through legal settlement.
The new pay-for-performance paradigm will require rules at various levels of the system. Identifying who is responsible for setting up which rules and is as important as enforcing the rules themselves. And we must provide clarity as to why a particular rule is necessary
In order to design successful business models and dissuade bad actors, all parties need a clear set of rules that are aligned with the intended outcomes. Regulators and utilities will need to find the right balance of performance criteria and specific rules to manage contracts and conduct oversight. Utilities will have the challenge of balancing regulatory guidance, state and federal laws, and their own strategies – all without overburdening the system. Regulators will need to create stable rules while also offering a process that allows market actors to iterate and improve, making room for creative solutions to emerge and be tested so that the market can evolve toward the best models.
Depending on performance requires enforcement. Enforcing the rules builds accountability both by ensuring a level playing field, and also by reinforcing intent of the rule. It builds confidence in the results, and it rewards those who can deliver based on the agreed-upon parameters. Performance based policies or contracts will not drive accountability, and they will not survive, in an uncertain enforcement environment.
Access to information, analysis, and performance allows for mutual accountability.
One challenge in the risk reward incentive mechanism process was that the results of the evaluation were not available to actors (implementers, utilities, or other stakeholders) until about three years after deployment. Given that the financial incentives were entirely dependent on the results, staff could not discuss evaluation plans with utilities outside of the formal proceeding. Several warnings, and preliminary reports were shot over the bow regarding the risk of performance based on metrics surrounding compact fluorescent lightbulbs, but mid course corrections did not happen.
Adopting agreed-upon techniques for monitoring meter-based savings offer regulators, utilities, implementers, aggregators, and customers an opportunity to have a common understanding of what “savings” means and monitor them in near real time. It is not a stretch to also monitor the effects of interventions alongside natural changes in usage. Providing ubiquitous access to the data and information on which performance is based means that actors can confidently exchange their services based on these results. They can also take corrective action to improve performance along the way in order to deliver on their promises. As trust in savings performance builds, balancing payments with other performance criteria may be prudent. Time, location, and load shifting values can create new incentives and reinforce value of kWh and therm savings up and down the market chain based on what’s delivered.
Validation and verification by third parties doesn’t have to be done behind closed doors. A clear and transparent audit can be focused on review of approved methods and agreed upon execution. Instead of saddling regulators with the full burden of re-creating a savings profile in a “crime scene investigation style” where gaps in information and timely return of results consistently hinder value, having a transparent approach allows evaluators to tackle interesting questions that they never get to in typical research plans because of a lack of data or budget. All actors can contribute to delivering the value of information on performance because it will matter to all of them.
The promise of performance is real. The potential pitfalls are also real. With the lessons learned from the failed incentive mechanism in California and many others from past performance initiatives in mind, I believe we can proceed with confidence in realizing the many possibilities that performance has to offer today.