Can We Determine What Works? A Meta-Analysis of Research Methods Used to Evaluate Behavioral Health Interventions in Criminal Justice Settings

Amy Blank, University of Pennsylvania
Wendy Pogorzelski, Rutgers University
Tina Maschi, Rutgers University - New Brunswick

Purpose: A meta-analysis was conducted to assess the quality and rigor of intervention and evaluation research in the areas of policing, courts, diversion, and re-entry that target individuals with behavioral health issues.

Method: Over 80 articles based on published intervention studies from the past decade were included in this meta-analysis. Content analysis was used to address the following questions: 1) What types of research methodologies are used? 2) What are the similarities and differences in research designs within and across content areas? 3) What are the challenges to isolating and measuring program outcomes?

Results: Overall, there was a lack of standardization in how design and methodological issues were reported across studies and content areas. There was wide variation in the amount of details that were reported in discussions of design and methodological decisions in each article. It was difficult to compare intervention studies across content areas due to variations in the developmental stage of the research and the disciplinary orientation of the researchers across content areas. Finally, when we reviewed the research for reported: use of equivalent comparison groups; the presence of adequate statistical power; and assessment of program effect size. Few studies were using equivalent comparison groups, and only two provided enough details about the effect size to determine if the study had the power necessary to detect an effect.

Implications: The methodological inconsistencies in these studies limit their applicability to policy debates about "what works". The current emphasis on "evidence-based practice" makes it imperative for researchers to critically review the strengths and weaknesses of intervention research. This will allow for the development of research strategies capable of answering policy questions related to what works in criminal justice interventions.

(Return to Program Resources)

Updated 05/20/2006