Methodology and Data Collection

Section 1:

Evaluation Purpose and Partners

Section 2:

Describing the Intervention

Section 3:

Evaluation Design

Section 4:

Methodology and Data Collection

Section 5:

Data Analysis

Section 6:

Reporting and Dissemination

Step 1

Step 2

This section provides steps to help you identify data sources and design data collection methods to answer your evaluation questions.

Step 1: Identify data sources or design data collection methods
There are two types of data that you can use in your evaluation: quantitative data and qualitative data. These data types can be used to provide insight to the evaluation questions using a variety of evaluation methods.
Quantitative data
Quantitative data are numerical data that can be counted (or quantified) and statistically analyzed. These data refer to specific assessment measures, such as sociodemographic information, geographic characteristics, knowledge and attitudes, perceptions and beliefs, behaviors, policy or environmental attributes, quality of implementation, or exposure to the intervention. The collection and analysis of quantitative data is intended to uncover numerical patterns and trends.
Common types of quantitative data and methods that can be used to evaluate outcomes corresponding to pedestrian safety interventions include:
View The Resource Table
SSA-Icon-Tools-resources
Tools and Resources
Pedestrian Safety Evaluation: Quantitative Data Collection Methods
Provides greater detail on specific quantitative data collection methods that can be used to inform pedestrian safety interventions.
SSA-Icon-Tips
Tip for Success
The DOT may collect pedestrian counts as part of the street auditing procedures. Working closely with the DOT partners can ensure that other variables that can inform the evaluation will be collected by during street auditors.
Often distrusting surveys requires a great deal of pre-planning and coordination with other partners to ensure that timing and survey distribution logistics are appropriate and in alignment with the intervention.
A survey should contain simply written questions that will help collect accurate and meaningful survey responses. For more tips on how to construct survey questions see “CDC Program Evaluation Tip Sheet: Constructing Survey Questions” and visit Creative Research Systems.
Qualitative data

Qualitative data are descriptive, non-numerical data that approximate or characterize – but do not measure – the attributes, characteristics, and properties of a thing or phenomenon. Qualitative data can provide contextual information that can convey the “how” and “why” of a phenomena or issue though the expressed thoughts, feelings, attitudes, and values of a person or group. Qualitative data describe whereas quantitative data define.

Examples of qualitative data collection methods that can be used to evaluate pedestrian safety interventions include:

Focus Groups
An interactive discussion between a homogenous sample of six to eight individuals. Focus groups are usually facilitated by a trained moderator who focuses on a specific set of topics to capture social trends and group perspectives.

Key Informant Interviews
A semi-structured, one-on-one conversation designed to gain insight on a given topic. The interviewer will guide the interviewee through a discussion of their own life experience, perspectives, and opinions to further understand or create new knowledge about a specific subject.

Observations
A research method aimed to systematically observe and record interactions, events, locations, etc. between individuals and their environment in a natural state. Observations provide allow evaluators to describe and understand people’s behavior in context.

Case Studies
An in-depth study of a documented event aimed to narrow down a board topic and offer a real-world application of a specific concept or theme. Case studies provide supplemental.

Document Analysis
A form of research that systematically reviews a document, policy brief, public record, etc. and then interprets the data to measure the impact of the file.

Inventories / Roadway Safety Audits and Reviews
A collection of all road-related information that defines and monitors the state-owned highway network, maintaining an inventory of the roadway features, conditions, and characteristics. Road Safety Audit (RSA) is the formal safety performance examination of an existing or future road or intersection by an independent, multidisciplinary team. It qualitatively estimates and reports on potential road safety issues and identifies opportunities for improvements in safety for all road users (FHWA).
SSA-Icon-Tips
Tip for Success
Although it is extremely valuable to review case studies presented by others online and through your network, you may find it even more valuable to create and review case studies from your own agencies. Review what you have done in the past and evaluate an intervention. Here is a resource to help you write a case study for your agency.
When conducting an environmental observational study on a specified area, you can engage your stakeholders and ask them to join you. This is a great way to continue collaboration and to get to know the environment and its functionality as best as you can.
Basic Differences between Quantitative and Qualitative Data
Below is a summary explaining the key differences between Qualitative and Quantitative Data: 1
View The Resource Table
Benefits of Combining Quantitative and Qualitative Data (Mixed method evaluations)

A mixed method evaluation is an evaluation approach in which both quantitative and qualitative data are collected, analyzed, and integrated to address one or more evaluation questions.2

Although quantitative and qualitative data have individual strengths, oftentimes both are needed to understand a complex problem and strengthen an evaluation. Quantitative data can provide statistically reliable numerical information, while qualitative data can provide richer and deeper insights into the phenomenon under study. Therefore, a mixed method evaluation provides an effective way to combine and cross-apply quantitative and qualitative data to present a reliable and valid set of evaluation results.

The table below details the strengths and limitations of quantitative only, qualitative only, and mixed method evaluation approaches.

View The Resource Table

As you create your evaluation plan and determine the specific data collection methods you will use, you must consider the following:

The purpose of the evaluation and the end users of the evaluation findings
Determine which evaluation and data collection methods would be most appropriate to answer the evaluation questions and yield credible results. Some data sources may allow you to collect both quantitative and qualitative data. For instance, a survey can have quantitative data (measures that are numeric – income, Likert scales) and qualitative data (measures that are non-numeric – race/ethnicity, open-ended questions). It is also important to consider data collection methods that will obtain the most buy-in from the primary audience or the end-users.

The availability of data and resources for the evaluation
Determine if you will have to collect your own data (primary data) or if you can utilize existing data sources (secondary data). The time and resources available to collect data should balance the strengths and limitations of each data source. Some methods are more costly to collect, analyze, and interpret.

Footnotes
  1. Nigatu, Tilahun. “Qualitative Analysis”. 2009. Presentation.
  2. (From John Creswell’s 2013 article: http://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1047&context=dberspeakers)
Next: Continue to Step 2
sec4step2_photo1
sec4step1_photo2