Losing Control: What Will Happen if Randomized Controlled Trials are Phased Out of Behavioral Program Evaluation?

Home energy reports and other behavioral conservation programs have gained significant traction in the utility industry as a supplement to traditional energy efficiency programs. The strongly preferred research design for evaluating these behavioral programs has been the randomized controlled trial (RCT), which typically relies on large control groups of non-participants. As these behavioral programs have grown from small pilots to full-scale programs that include a significant portion of the residential customer base, utilities and other stakeholders have started to question the future viability of the RCT, given that its large control groups of non-participants limit the potential for behavioral programs. Meanwhile, “Big Data” and innovative statistical methods from the fields of machine learning, Bayesian statistics and econometrics have also gained significant traction, especially in other industries such as the tech sector (Varian 2014). While these methodological alternatives show considerable promise for behavioral program evaluation, their statistical validity relative to the current “gold standard” – RCT – has yet to be tested rigorously. This paper leverages data from one of the largest home energy reports programs in order to compare the results of a large, multi-year RCT evaluation to the energy savings that evaluators would have estimated using alternative methods. Results from the analysis show that the alternative methods tested do not produce energy savings estimates that are similar to those of an RCT.