5 Benefits of Real-time Evaluation

An excerpt from Gary Epstein’s paper for ECEEE Summer Study 2015.

For many energy efficiency and demand management programs, determination of immediate results can support the effort toward strategic planning and monitoring for infrastructure planning and emission reduction and mitigation. Impact and process evaluations on the traditional time scales (i.e., after the program cycle is complete) are not particularly useful to the directors of efficiency efforts or to the planners who need to determine whether programs will have procured the energy efficiency impacts and demand reductions as specified in various directives. A postprogram impact evaluation will not alert planners in time, and a post-program process evaluation will not provide actionable findings that immediately improve the program operation.

In the end, the most useful output of evaluations will be real-time feedback to assess the actual progress toward the goals and provide suggestions to make the program more effective and progressive. Such a real-time evaluation exercise readily flows from the more dynamic pre/post evaluation protocol discussed in the prior section of this paper. Taking the next steps, the real-time evaluation would have the following characteristics:

  1. Integrated real-time evaluations – Real-time evaluations will build upon program pre/post measurement and verification (M&V) activities described above, incorporating both program impacts and processes. Such efforts will decrease duplicative effort and will be more cost-effective overall than separate post-program evaluations, providing near immediate results to project and program implementers, evaluation managers, and regulators.
  2. Dynamic reporting of site, project, and overall program impacts – The real-time evaluation will provide ongoing data to the program staff and regulators to support reporting and planning. This timely information will be valuable in targeting program efforts to the most productive measures and customer types and can help infrastructure planning teams with up-to-date information for forecasting, as well as energy-focused efficiency reductions.
  3. Assured quality control and immediate programmatic and project improvements – Through dynamic reporting of pre- and post-M&V of newly installed projects, much data can be ascertained on the merits of a wide variety of technologies, approaches for implementation in various sectors, and assumptions on operating characteristics that drive technology success. This information can be used to guide future projects – even ones that are near ready for implementation planning – in enabling better installations that are more likely to achieve their projected savings. In addition, programs can learn from the immediate insights, enabling improved targeting of appropriate sectors and technologies for the most significant and successful program and project impacts.
  4. Real-time dashboard and database of program results – The mechanisms for real-time reporting and information dissemination are crucial for the success of real-time evaluation efforts. Typical strategies include graphic reporting user interfaces (on-screen dashboards), with associated backbone databases, that are continually updated to reflect results on specific projects and aggregated results. The authors have developed and applied such structures in some on-going real-time evaluation efforts to great success. Recent efforts for CECONY, National Grid, and NYSERDA have proven highly successful and being expanded to capture more details of the installed efficiency projects.
  5. Progressive move toward incorporation of real-time metering with immediate reporting of impacts – The ultimate stage in real-time evaluations involve use of actual, real-time metering system reporting, with cloudbased meter result distribution and storage and ultimate analysis and aggregation to stated database structures and reporting dashboards.

For a survey of protocols, more discussion of concurrent and integrated evaluation, and bonus discussion on the state of evaluation in Europe, the paper, The Case for Expanding Comprehensive US Evaluation Mandates and Considerations for the EU, provides more information on the topics.