A Strategy for Evaluating the Impact of County-Level Programs: Communities That Care in Pennsylvania

D. Wayne Osgood, Pennsylvania State University
Amy L. Anderson, The Pennsylvania State University

ABSTRACT
Community level interventions have become increasingly popular in recent years, and evaluating these programs poses special methodological difficulties. Based on our evaluation of Communities that Care in Pennsylvania, we present a design and analysis strategy that addresses those difficulties and yields a strong estimate of program effectiveness. This strategy is applicable when the program is implemented in some proportion of a larger set of communities, as often happens when state government funds a limited number of counties to implement a program. The evaluation strategy is especially practical and cost effective when outcomes can be measured by routinely reported administrative data, such as county level crime data. The basic form of our analysis is Campbell and Stanley's multiple interrupted time series. Rosenbaum and Rubin's (1983) propensity socring method provides a useful approach for selecting a suitable group of comparison counties. The design is further strengthened by focusing the analysis on within-county change, before versus after the implementation of the program. We do so through a pooled-time-series regression model that controls for average time trends. We estimate this model using the Poisson version of hierarchical linear modeling, which also takes into account the limited accuracy of crime rates for small populations.

(Return to Program Resources)

Updated 05/20/2006