Impact evaluation
| 
Impact Evaluation
What is it?
Impact evaluation is the systematic identification of the effects – positive or negative,
intended or not – on individual households, institutions, and the environment caused
by a given development activity such as a program or project. Impact evaluation helps
us better understand the extent to which activities reach the poor and the magnitude of
their effects on people’s welfare. Impact evaluations can range from large scale sample
surveys in which project populations and control groups are compared before and after,
and possibly at several points during program intervention; to small-scale rapid assessment
and participatory appraisals where estimates of impact are obtained from combining
group interviews, key informants, case studies and available secondary data.
What can we use it for?
- Measuring outcomes and impacts of an activity and distinguishing these from the
influence of other, external factors.
- Helping to clarify whether costs for an activity are justified.
- Informing decisions on whether to expand, modify or eliminate projects, programs
or policies.
- Drawing lessons for improving the design and management of future activities.
- Comparing the effectiveness of alternative interventions.
- Strengthening accountability for results.
ADVANTAGES:
- Provides estimates of the magnitude of outcomes and impacts for different demographic
groups, regions or over time.
- Provides answers to some of the most central development questions – to what
extent are we making a difference? What are the results on the ground? How can we do better?
- Systematic analysis and rigor can give managers and policy-makers added confidence
in decision-making.
DISADVANTAGES:
- Some approaches are very expensive and time-consuming, although faster and more
economical approaches are also used.
- Reduced utility when decision-makers need information quickly.
- Difficulties in identifying an appropriate counter-factual.
COST:
A number of World Bank impact evaluations have ranged from $200,000 - $900,000
depending on program size, complexity and data collection. Simpler and rapid impact
evaluations can be conducted for significantly less that $100,000 and in some cases for
as little as $10,000 - $20,000.
SKILLS REQUIRED:
Strong technical skills in social science research design, management, analysis and
reporting. Ideally, a balance of quantitative and qualitative research skills on the part of
the evaluation team.
TIME REQUIRED:
Can take up to 2 years or more. Rapid assessment evaluations can often be conducted
in less than 6 months.
EXAMPLES OF IMPACT EVALUATION DESIGNS
Randomized evaluation designs, involving the collection of information on project and
control groups at two or more points in time, provide the most rigorous statistical
analysis of project impacts and the contribution of other factors. But in practice it is
rarely possible to use these designs for reasons of cost, time, methodological or ethical
constraints. Thus most impact evaluations use less expensive and less rigorous evaluation
designs. The following table describes four approaches to impact evaluation
designs in development evaluation. The first is an example of a randomized evaluation
design; the second is a quasi-experimental design in which a "non-equivalent" control
group is selected to match as closely as possible the characteristics of the project population;
in the third example the project population is compared with a non-equivalent
control group after the project has been implemented; and the fourth is a rapid assessment
evaluation which combines group interviews, key informants, case studies and
secondary data. Each successive model sacrifices methodological rigor, in return from
which there are significant reductions in cost and time requirements.
| 
