What is it?
Impact evaluation is the systematic identification of the effects – positive or negative, intended or not – on individual households, institutions, and the environment caused by a given development activity such as a program or project. Impact evaluation helps us better understand the extent to which activities reach the poor and the magnitude of their effects on people’s welfare. Impact evaluations can range from large scale sample surveys in which project populations and control groups are compared before and after, and possibly at several points during program intervention; to small-scale rapid assessment and participatory appraisals where estimates of impact are obtained from combining group interviews, key informants, case studies and available secondary data.
What can we use it for?
- Measuring outcomes and impacts of an activity and distinguishing these from the
influence of other, external factors.
- Helping to clarify whether costs for an activity are justified.
- Informing decisions on whether to expand, modify or eliminate projects, programs
- Drawing lessons for improving the design and management of future activities.
- Comparing the effectiveness of alternative interventions.
- Strengthening accountability for results.
- Provides estimates of the magnitude of outcomes and impacts for different demographic
groups, regions or over time.
- Provides answers to some of the most central development questions – to what
extent are we making a difference? What are the results on the ground? How can we do better?
- Systematic analysis and rigor can give managers and policy-makers added confidence
- Some approaches are very expensive and time-consuming, although faster and more
economical approaches are also used.
- Reduced utility when decision-makers need information quickly.
- Difficulties in identifying an appropriate counter-factual.
A number of World Bank impact evaluations have ranged from $200,000 - $900,000 depending on program size, complexity and data collection. Simpler and rapid impact evaluations can be conducted for significantly less that $100,000 and in some cases for as little as $10,000 - $20,000.
Strong technical skills in social science research design, management, analysis and reporting. Ideally, a balance of quantitative and qualitative research skills on the part of the evaluation team.
Can take up to 2 years or more. Rapid assessment evaluations can often be conducted in less than 6 months.
EXAMPLES OF IMPACT EVALUATION DESIGNS
Randomized evaluation designs, involving the collection of information on project and control groups at two or more points in time, provide the most rigorous statistical analysis of project impacts and the contribution of other factors. But in practice it is rarely possible to use these designs for reasons of cost, time, methodological or ethical constraints. Thus most impact evaluations use less expensive and less rigorous evaluation designs. The following table describes four approaches to impact evaluation designs in development evaluation. The first is an example of a randomized evaluation design; the second is a quasi-experimental design in which a "non-equivalent" control group is selected to match as closely as possible the characteristics of the project population; in the third example the project population is compared with a non-equivalent control group after the project has been implemented; and the fourth is a rapid assessment evaluation which combines group interviews, key informants, case studies and secondary data. Each successive model sacrifices methodological rigor, in return from which there are significant reductions in cost and time requirements.