Craig Riddell - Interpreting Experimental Evidence in the Presence of Post-Randomization Events: A Re-Assessment of the Self Sufficiency Project

Melbourne Institute Seminar

Melbourne Institute Seminar Room
Room 6.05, FBE Building
111 Barry St, Carlton

Map

More Information

Barbara Broadway

b.broadway@unimelb.edu.au

Title: Interpreting Experimental Evidence in the Presence of Post-Randomization Events: A Re-Assessment of the Self Sufficiency Project

Abstract: Randomized trials are the gold standard for credible evidence. Randomization ensures that treatment and control groups are statistically equivalent at the baseline. However, once treatment begins the characteristics of the experimental groups will generally diverge. Post-randomization events such as policy changes may exert different impacts on the two groups, altering the counterfactual and the interpretation of the experimental estimates. This paper argues that conclusions reached from a well-known welfare-to-work experiment – the Self-Sufficiency Project – need to be re-assessed because of policy changes during the demonstration. The treatment was a generous time-limited earnings supplement provided to those who left welfare for full-time work. The financial incentive resulted in large impacts during the supplement period. However, treatment-control differences subsequently faded and shortly after the supplement period there were no significant experimental impacts. The absence of lasting impacts reduced enthusiasm for this approach to welfare reform. However, important policy changes occurred during the demonstration. We provide credible evidence on what the treatment effects would have been in the absence of these changes. This re-assessment leads to significantly different conclusions; time-limited earnings supplements that ‘make work pay’ can have lasting effects on welfare use. Our study illustrates the importance of interpreting experimental findings with care in the presence of post-randomization events.

Presenter: Craig Riddell, University of British Columbia

The program coordinator of these seminars is Barbara Broadway. If you would like to subscribe to the Melbourne Institute Seminar Series email list, please contact Barbara.