Most energy evaluation studies are too small to robustly detect the reductions they are looking for. How to fix? That depends on your appetite for false positives and false negatives.
On March 11th SERG’s Dr Ben Anderson presented a paper based on SERG’s experiences working on the Solent Achieving Value through Efficiency (SAVE) project at the (virtual) Energy Evaluation Europe: 2021 Conference. The conference, which was spread over 4 mornings to enable participants to join from as many time zones as possible, brought together engineers, researchers and policy/commercial stakeholders form across the energy evaluation sector.
Ben’s paper, which was co-authored with Tom Rushby, Abubakr Bahaj and Patrick James, highlighted the ongoing confusion in empirical energy efficiency and energy demand evaluation studies over the meaning, value and use of statistical significance and statistical power. This is compounded by confusion over how these concepts should be used both in designing studies and in deciding what can be inferred from them. As a consequence, sample sizes in most energy efficiency studies may be too low to provide adequate statistical power and so statistically robust conclusions cannot be drawn at conventional thresholds . The paper explores this problem via the design of a study focused on winter evening heat pump demand to demonstrate how sample sizes, effect sizes and confidence intervals matter.
Watch the presentation:
Further reading:
Anderson, Ben, Rushby, Tom, Bahaj, Abubakr and James, Patrick (2021) Ensuring statistics have power: sample sizes, effect sizes and confidence intervals (and how to use them) . Energy Evaluation Europe: 2021 Europe Conference: Accelerating the energy transition for all: Evaluation’s role in effective policy making, Online. 10 – 16 Mar 2021.
Anderson, B., Rushby, T., Bahaj, A., & James, P. (2020). Ensuring statistics have power: guidance for designing, reporting and acting on electricity demand reduction and behaviour change programs. Energy Research & Social Science, 59, 1-8. [101260]. https://doi.org/10.1016/j.erss.2019.101260