Evaluation Design – Step 3

Section 1:

Evaluation Purpose and Partners

Section 2:

Describing the Intervention

Section 3:

Evaluation Design

Section 4:

Methodology and Data Collection

Section 5:

Data Analysis

Section 6:

Reporting and Dissemination

Step 1

Step 2

Step 3

Step 4

Step 3: Specify the evaluation design to maximize causal inference and internal validity for each evaluation question
The next step is to create your evaluation design(s) with respect to one or more of your evaluation questions and the associated types of evaluation. As noted in the previous step, you want to maximize your ability to say that the pedestrian safety intervention caused the changes in outcomes observed. Although many pedestrian safety evaluations will not be able to show full causality, or attribution, your evaluation can show that the combination of intervention activities contributed to the observed outcome of interest. Causal inferences increase confidence in the internal validity of the relationship of the intervention to the outcomes observed. Rigorous evaluation designs help to minimize threats to internal validity.
Select your evaluation design
There are three basic categories of evaluation design: experimental, quasi-experimental, and non-experimental. The key characteristics of each design are shown in the following table:
View The Resource Table
SSA-Icon-Tools-resources
Tools and Resources

Evaluation Design Illustrations1
This table shows descriptions, examples, and graphic depictions of each evaluation design along with causality and strength of attribution.

Minimize threats to internal validity
Although you’ve learned about the many types of evaluation designs you and your evaluation partners can use, you’re probably wondering how to determine the evaluation design you should use. The short answer: Choose the design that can best provide the data you need to:
  1. Answer your evaluation questions;
  2. Complete your evaluation within the time allotted; and
  3. Utilize the resources at your disposal.
However, there are likely several designs that could provide these data, and you want to choose the design that will maximize your causal inference. Therefore, your next step is to consider the strengths and weaknesses of each evaluation design in minimizing threats to internal validity. Your final decision will need to strike a balance between getting the best possible data and making the most efficient use of available resources within your timeline. There are three basic conditions necessary to establish causality, including:
  • A relationship must be established between the intervention and the outcomes.
  • The intervention must precede the outcomes observed.
  • The outcomes cannot be attributed to other explanations.
With these conditions in mind, there are seven basic threats to internal validity and their relationship to evaluation designs:
View The Resource Table
SSA-Icon-Tools-resources
Tools and Resources


Evaluation Design: Strengths and Weaknesses2,3,4,5,6,7
This table shows strengths and weaknesses of each design, along with a description and example pedestrian safety intervention evaluations for each design.

Footnotes
  1. Adapted from: Measurement, Learning & Evaluation (MLE) Project (evaluation component of the Urban Reproductive Health Initiative). Retrieved from: https://www.urbanreproductivehealth.org/toolkits/measuring-success/types-evaluation-designs
  2. Adapted from: University of Albany Center for Problem-Oriented Policing. Assessing Responses to Problems: An Introductory Guide for Police Problem-Solver. Appendix D: Summary of Evaluation Designs’ Strengths and Weaknesses. Retrieved from: http://www.popcenter.org/tools/assessing_responses/7
  3. Zegeer, C., D Henderson, R Blomberg, L Marchetti, S Masten, Y Fan, L Sandt, A Brown, J Stutts, and L Thomas. Evaluation of the Miami-Dade Pedestrian Safety Demonstration Project. National Highway Traffic Safety Administration. Washington, DC, 2008.
  4. Pennsylvania Department of Transportation Bureau of Planning and Research. C. Strong, M. Kumar. Western Transportation Institute College of Engineering Montana State University. Safety Evaluation of Yield-to-Pedestrian Channelizing Devices: Final Report. 2006 Retrieved from: https://www.dot.state.pa.us/public/pdf/YTPCDFinalReport.pdf
  5. O’Connor, E., J. Bellamy, B. Spring. Evidence-Based Behavioral Practice Online Training Course. Critical Appraisal: Time Series Designs Retrieved from: http://ebbp.org/course_outlines/critical_appraisal/#C
  6. Huitema, RV Houten, H Manal. Time-series intervention analysis of pedestrian countdown timer effects. Accident Analysis & Prevention 2014. 72:23–31. Retrieved from: http://europepmc.org/abstract/med/25003967
  7. Compiled from Federal Highway Administration Office of Natural and Human Environment 2005. Pedestrian and Bicycle Data Collection in United States Communities: Quantifying Use, Surveying Users, and Documenting Facility Extent. Retrieved from: http://www.pedbikeinfo.org/pdf/PlanDesign_Tools_FHWACaseStudies.pdf
Next: Continue to Step 4
sec3step3_photo1
sec3step3_photo2