Jon Maxwell and Brian McCowan for Zondits, June 8, 2015. Image credit: PixShark
“Thank you. Your evaluation has helped us greatly with improving our program.”
How often do evaluators hear that phrase? The answer is, of course, not very often. The reasons are many and varied, but the overriding reason is that evaluators, program administrators, regulators, and evaluation critics seem to have forgotten a basic premise: The primary purpose of program evaluation is to help improve programs.
Recent criticisms of evaluation efforts are not totally incorrect. In particular, impact evaluation results are often reported too late to have a significant effect on improving programs, and their quantified uncertainties can leave the results open to criticism from all sides. If the results of an impact evaluation come back poor, the evaluation is criticized as being flawed, too uncertain, or missing the forest for the trees (“Stop tearing down the programs; EE’s still better than generation!”). On the other hand, if they come back favorable, the evaluation self-evidently was a waste of money. And almost regardless of the outcome, one can expect the lament that the results were provided too late.
Unfortunately, the current reaction to evaluation issues both real and imagined threatens to throw the baby out with the bath water. Some critics claim that an enormous amount of money is collected by a cottage industry of evaluation consultants, but the truth is that nationwide, impact evaluation M&V costs an average of about 2% of program funding. This is a small expense relative to the technical and political value that evaluations provide. Independent third-party evaluation is the only way to judge program performance without potential bias from those with vested interests and/or socio-political goals both in favor of and against these programs. For the types of intervention that are funded by mandated taxes and considered unnecessary by some, the fact-based assessment offered by rigorous independent cost-effectiveness analysis is critical. Further, as utilities and ISOs rely more on efficiency and distributed generation to precisely meet local loads, delivered performance is more than just a scoring exercise‒it gives the engineering planners hard numbers to ensure that brownouts are not coming.
The solution is not to end evaluation efforts; it is to do them smarter and more quickly. New evaluation procedures and monitoring equipment are offering the opportunity to improve evaluations and the programs they serve, and innovative management techniques are heightening their impact. For evaluating some types of projects and programs, relying more on meter data analytics promises to provide useable results with less fieldwork. For complex projects and programs, new metering technologies can be used to quickly provide enhanced granular data that allows evaluators to more accurately pinpoint areas for improvement.
The revolutions of Big Data, wireless metering, and frankly even the Internet have yet to touch most evaluation M&V. Exciting technological solutions that will help evaluations serve programs better are in their nascent stages. Both engineers and econometricians are working on new techniques to deliver the highest value and speed feedback. Program management software is expanding to include dynamic performance monitoring. Mesh Wi-Fi networks promise to pull site M&V costs down. Evaluation metering hardware is available for real-time and permanent site installation. Evaluation designs that involve impact evaluation concurrent with project implementation and/or integrate impact and process evaluations are offered by our firm as the preferred evaluation methodology. This approach provides powerful feedback to administrators quickly and should become the norm rather than the exception.
Management as well as technology can solve problems. Don’t let perfection be the enemy of progress. Aggressive evaluation planning schedules with short and firm deadlines will deliver timely results that are more valuable than slightly more refined results delivered months or years later. As impact evaluation schedules accelerate, they enable another powerful management technique: integrated process and impact evaluation that combine data collection and create more acute and substantiated recommendations for program improvement.
There will always be some delay in delivering evaluation results. After all, whether it’s a concert, a theater production, or an efficiency program, one can’t assess performance until the performers take the stage. But if we accept the premise that the purpose of evaluation is to improve programs, and not simply to report results, we can – and must – turn years into months or maybe even days. With new hardware, software, and innovative procedures, we should work toward improving evaluations and focusing evaluation efforts on program improvement.
[bctt tweet=”New metering technologies enhance granular data & allow evaluators to pinpoint improvement areas. ” username=”ZonditsEE”]
Editor’s Note: This article was originally posted in June 2015 and has been revamped and updated for accurateness and comprehensiveness.
[mks_button size=”medium” title=”5 Benefits of Real-Time Evaluation” style=”rounded” url=”https://zondits.com/article/9587/5-benefits-of-real-time-evaluation” target=”_self” bg_color=”#81c341″ txt_color=”#FFFFFF” icon=”” icon_type=”” nofollow=”0″]