Section 1:
Evaluation Purpose and Partners
Section 2:
Describing the Intervention
Section 3:
Evaluation Design
Section 4:
Methodology and Data Collection
Section 5:
Data Analysis
Section 6:
Reporting and Dissemination
Section 3: Evaluation Design
Step 4: Determine sampling strategies to maximize external validity in the selected population or subpopulations for each evaluation question
To complement your evaluation design, you and your evaluation partners should determine your sampling plan. A sampling plan can help maximize the applicability of your evaluation findings to similar populations and sub-populations. The applicability of evaluation findings is commonly referred to as external validity and it can apply to people, places, and time periods.
Identify your sampling methods
In essence, your evaluation questions guide the selection of a sample of participants or data that represents your population or subpopulations of interest to ensure that the evaluation findings can be generalized to similar populations or subpopulations.
There are two main types of sampling methods:
- Probability, or random assignment of population or subpopulation members to intervention and control or comparison groups
- Non-probability, or non-random assignment
This graphic1 illustrates how the different types of probability and non-probability sampling methods relate to one another and the descriptions as well as relative strengths and weaknesses are described in the following table.
Tools and Resources
Sampling Methods: Comparing Strengths and Weaknesses2,3
This table shows descriptions of sampling methods along with strengths, weaknesses, and examples.
In creating your sampling plan, your partners should prioritize the sampling methods that best match your evaluation questions, design, and audiences as well as your partner’s’ skills and experience, timeline, and resources available to recruit participants and collect data.
Determining your sample size
Next, you and your evaluation partners should decide on the size of the sample or samples necessary to make causal inferences about your population or subpopulations of interest based on the evaluation findings from your sample.
The goal is to increase statistical power in order to increase confidence that the evaluation findings are detecting an intervention effect when the intervention effect truly exists.
Tools and Resources
powerandsamplesize.com
Free and open source online calculators.
Free and open source online calculators.
PowerUp!
provides convenient excel-based functions to determine minimum detectable effect size and minimum required sample size for various experimental and quasi-experimental designs.
provides convenient excel-based functions to determine minimum detectable effect size and minimum required sample size for various experimental and quasi-experimental designs.
PowerUpR
is R package version of PowerUp! and additionally includes functions to determine sample size for various multilevel randomized experiments with or without budgetary constraints.
is R package version of PowerUp! and additionally includes functions to determine sample size for various multilevel randomized experiments with or without budgetary constraints.
WebPower
Free online statistical power analysis.
Free online statistical power analysis.
SampSize
for Android and iOS iPhone and iPad.
for Android and iOS iPhone and iPad.
These power calculations should account for precision of the intervention effect estimates, systematic errors in the data collected or analyzed, and loss of participants in follow-up assessment over time (attrition).
Seek out referrals for statistical expertise from government agencies or academic institutions when more rigorous sampling and power techniques are needed.
Seek out referrals for statistical expertise from government agencies or academic institutions when more rigorous sampling and power techniques are needed.
Footnotes
- J. Dudovskiy 2011 Research-Methodology Website. Retrieved from: http://research-methodology.net/wp-content/uploads/2012/04/Sampling-in-primary-data-collection1.png
- http://study.com/academy/lesson/cluster-random-samples-definition-selection-examples.html
- J. Dudovskiy 2011 Research-Methodology Website. Retrieved from: http://research-methodology.net/sampling-in-primary-data-collection/multi-stage-sampling/
