CAG will be presenting at a CECAN seminar for policy evaluators in Westminster, about methods used in a policy evaluation for BEIS. This seminar, held at lunchtime on 5thSeptember at 1 Victoria Street, London, will examine the application of process tracing methods within a realist evaluation.
The evaluation of the ‘Transitional Arrangements (TA) for Demand Side Response’ scheme was commissioned by the Department of Business, Energy and Industrial Strategy and has been led by CAG Consultants in partnership with Charles Michaelis (CECAN) on behalf of Adam Stiles at BEIS.
Our approach to process tracing has been informed by the work of Dr. Barbara Befani (CECAN). When evaluating the first TA scheme, we applied ‘process tracing with Bayesian updating’ to test the additionality of outcomes generated by the scheme – we described this as ‘contribution tracing’ to reflect the fact that we were testing for the contribution of the first TA to observed outcomes. This involved testing additionality hypotheses for each organisation participating in the first TA scheme, using a set of evidence tests for each hypothesis.
This process tracing contributed to our overall assessment of additionality, which used realist contribution analysis. However, the additionality hypotheses and evidence tests did not directly relate to the ‘Context-Mechanism-Outcome’ (C-M-O) configurations within our realist theory. When evaluating the second TA scheme, we attempted to use process tracing to test the individual ‘C-M-O’ hypotheses in our realist theory. This was challenging because of the large number of possible C-M-O configurations that could be tested. We have therefore applied process tracing in a more limited way when evaluating the second TA, using process tracing concepts (e.g. ‘hoop tests’, ‘straws in the wind’, ‘smoking guns’) to assess the weight of evidence supporting our C-M-O analysis for particular cases, but without assigning numerical probabilities or using Bayesian updating.
This seminar 
will discuss learning from these two approaches to undertaking process tracing within a realist evaluation.
Attendance at the seminar is free – if you are interested in attending, you can register via the CECAN website here.