Impact Assessment of BDS and Training Programmes
Dear All,
Good evening and many thanks to all who contributed to the previous topic on sustaining BDS programmes. Great to come back again with another important related issue. This one perhaps will be a bit technical but we will all bring it down. It is on impact assessment of BDS or training programmes. I say this as we can agree that whatever we do, we need to show results. Impact, which should be measured. How can we measure impact of BDS interventions? What are the core indicators we can agree on to use in measuring impact of a standard youth enntrepnership programme? Number of new businesses created, number of jobs, increased incomes for existing business? What others can we suggest? What impact assessment methods have we used?. What is your take with impact assessment methodology of using control group (a similar group where there is no intervention introduced) and treatment groups (a similar group where the project initiates some interventions) in impact assessment of BDS and training programmes in order to compare results? How feasible is it? Let us share experiences, information and leads on how we can shape the thought process on this for the benefit of ourselves and our projects. Thanks. Jealous
Impact assessment in the light of youth entrepreneurship
Thank you very much for this wonderful and interesting topic, am joining late as usual but i think can still contribute on the intro before i could zero in into today's discussion. BDS approaches for me were interesting looking at what is obtaining on the ground in Zambia especially in the light of FSIP Programs from government, Cash transfer project that people in Eastern province are being given by DFID and HELP Age,constitutes one such initiative that corresponds to traditional approach. People are never consulted in the design, development and implementation of the program. This kind of initiative often, is not appreciated by the beneficiaries because often they do not owner the program. They just passive recipient of development or change and not drivers of that development , initiative or that change. As such sustainability of the project is compromised. Immediate the project funds are exhausted and the project closes, everything just falls to pieces. This is happening in Zambia and thus, though in other perspective impact is said to be there, in reality it is money invested in the drain. However, this is not to say market driven projects are not implemented there are and though not in favour of the developing interprenuers, let me not talk much it is a missed topic. Back on today's topic measuring impact especially in the light of the current enterprise in Zambia. Measuring impact is one important and integral part of any program that is implemented whose input is money(funds). Many donors and funders would want to know how they money was spent, and what was achieved in respect. However, measuring impact can only be done if measures are put in place before the program or business is designed. Indeed one needs to know the six element of effectiveness managent coined as COPIPO, You need to understand context where are you,( baseline and situation analysis), understand the planning , where do you want to be or go, develop management plans, work plans, monitoring plans and sustainability plan e.t.c. Input is another aspect, what do you need to get there,,,staff and financial resources,,is it enough, then one needs to look at Output, what did we produce in the implementation process. Then one need to look at what was the Bold textoutcome, what did we achieve in-terms of our objectives.
Now all these aspects are suppose to be put in place way before implementation then design indicators which should be able to capture all the above aspect.. You need to tell the owners of the money how mach impact was realised and this can only be realised through a robust Monitoring and evaluation plan that has provisions for evaluations ,,midterm, diagnostic impact or just the most important one and critical the end of project evaluation to which baseline survey should serve as comparing point to where the project or program is and impact achieved so far... Sorry i will add more later Regards
Dear Isaac.
Welcome to the forum. And thank you for the comprehensive discussion. I will repond to only parts of it. And you are picking it up from the first topics. I hope you have had chance to go through the discussions we have had so far. It is good that you are giving a case study of what happens locally andhighlighting what could have been docn ebetter, from the start. Thanks you for sharing your COPIPO framework - I hope others learnt from it and can take something from it. And, as you say, a robust M & E system is needed. Many thanks again. Jealous
Dear Jealous,
Thanks for this interesting topic again.
Apart from The DCED recommended universal indicators on private sector development. Generally the following key indicators are common for the propose of the BDS impact assessment. Most of them are already reflected in your discussion question above.
• Number of BDS providers recruited,
• Number of support actions identified
• Number of trainers and facilitators trained,
• Number of courses held,
• Number of trainees participating in the training
• Number of transactions and amount of the transactions
• Type of BDS services provided/offered
Methodology: In our case before any interventions, we use to develop result chain logic/intervention logic (with clear indicators on Inputs, outputs ,outcomes and impacts).Control group mechanism is one of the best approach to compare the results
Dear Ekanath,
Good evening and thank you for expanding on these. And the reference to the DCED guidelines, which is useful for everyone participating in the discussions. I urge everyone to google their website. In terms of the indicators, we can have another entire discussion there, so that we are able to go beyond the output / activity level indicators to more outcome oriented ones. Those that measure the results of what has been done (outputs), for instance number of new businesses started, increased sales and incomes of the business oweners, number of SMEs and amount of credit accessed, number of new jobs. these will better show results.
I am happy you share the notion that control growup methodology is the best. I agree as it helps us to answer to the question of attribution. In other words, did the intervention contributed to the results? Would the same result not have been achieved without your intervention. This is really key. We need to exchange more on this. Jealous
Dear Jealous
Thanks for your elaborative response. Yes impact assessment on BDS interventions is one of the most debatable issues. Another issue is time frame. According to the DCED standards, the standard time frame for impact assessment is 2 years . another critical issue is attribution which you have raised / questioned in your respond. I totally agree with you that actual result attribution is another challenge. Without our particular interventions there might be many other interventions and supports, therefore while attributing the results/ impacts we have to consider the element of “coping in” and “crowding in “aspects. As per the DCED guideline/ standards, if it is possible we need to distinguish the direct and indirect results in our result chain logic.
Dear all, Please visit the webpage www.enterprise-development.org for more DCED guideline and result measurement standards.
With best regards,
Ekanath
Thank you very much for you response
For me indicators to measure impact should be well framed. They are supposed to depict the results that one needs to achieve. As i pointed out the key word is what did we achieve and not what did we produce in terms of products or services, this is a very key guiding principle. Now what do we need to monitor as in indicators we can frame for us to ensure that we do not miss our mandate. For me if we are to look at number of courses held, number of transactions transacted and number of BDS providers recruited, then we are going to miss the point. These indicators are giving as output and they are therefore only used for monitoring purposes. Impact is a product of evaluation and that is more inclined to outcomes. If we look at number of trained entrepreneurs that are running there businesses, Owning companies ,Number of Enterprise using the market approach, number of people trained who are able .....e.t.c These type of will help us to know what we have achieved in the light of impact,9reducing poverty,creation employment among other social economic dynamics.
On the Other hand logical frame approach is a very important tool that one can use to monitor and evaluate their programs it is been used extensively noways and it is a donor requirement in most cases so forks it is only better to know it and use it. Logical framework will tell you what risk assumptions to expect,frequency of data collection , under which objective is that activity being carried out, Goal , output and outcome expected. It is simply a beautify monitoring and evaluating tool as it even give provision for the person responsible and the cost involved. I should however, make a mention here that care should be exercises especially when crafting these two tools, indicators and logical framework as it may not be able to detect any deviation from the normal implementation of a program. You may expend money on programs that does not address the objective of that program. However, a robust monitoring mechanism will within no time detect a problem at any given time and help you to make timely strategic management intervention. regards
Dear Isaac,
Thank you very much for your respond. I really appreciate your ideas that we should not focus our impact assessment around outputs only and we should design more impact focused interventions.
Yes I do agree with you about the importance of Logical framework approach which is more elaborative frame work for result measurements. But I do prefer to use result chain logic or intervention logic . The result chain logic/intervention logics are much more easy frameworks to connect inputs- outputs - outcome and impacts logics .
With best regards,
Ekanath
Dear Ekanath,
Thank you very much for the your response and clarification on what you use and especially the one which is seemingly easier for you to track. Impact measurement is just in a way like any other activity that a program may be mandated to implement. The key thing is for one to own the process and be able to use it effectively,enable you to use your result efficiently and ultimately ensures your organization's project sustainability. This is only also again possible if we take keen interest in formulating frameworks that we can use effectively. I have been working as a consultant on evaluating management effectiveness of programs. At the project design stage good and robust frameworks are adopted some are even way above the organization M&E capacity needs but because people what to please donor they adopt and later labor to adapt. This kind of frameworks beautiful as they may look they do not ultimately help the organization to attain its desired goal. This is waste of time and money. Designing a simple one we understand and be able to offer a clear cut way to tracking progress in results, it is actual thousand times progress than adopting one that one can not use. In life academic exercise do not apply only what one is comfortable with. Sorry Ekinath, this is not to say yours is simple iam just trying to stress the need a for simple and one comfortable models. Good and robust M&E system calls for the adhering to the adaptive management principles, able to monitor results, adapt, analyst ,learn ,communicate and adapt best practices. All these are taken care of in my COPIPO framework for project assessment and impact diagnosing. Regards
Hi Jealous and Colleaques,
The impact of BDS to Youth Enterpreneur has to do with youth iniatives. This is to start with planning to the exercution of the project.That is from start to end. In undertaking a project, participants need to see the needs on ground as need assessement, after that, plan on how it will be carry out. These involve inputs, outputs, outcomes and impacts e.g, the number of people, the type of services to be provided,the people to do the training - facilitators, budgets- that is the costs etc
Dear Ubandoma,
Great. Normally all this will be contained in the logical framework. I have seen project documents which have logical frameworks that are not solid, especially on the indicators. I have also seen programme managers that make whast I would say 'sloppy' use of the LOGFRAME - they dont follow it, they divert to do other this. It will become difficult when it comes to impact measurement when there were no baselines and no solid indicators. This is important. Jealous
Dear Jealous
First of all sorry for joining the discussion late. I have had a tight schedule having been on a mission with the African Development Bank to Central and Southern provinces. In the places I was staying Internet connectivity was a real challenge and even my MTN dongle did not come to my rescue due to poor mobile network.
Anyway good to be back regarding the first discussion point: How can we measure impact of BDS interventions?
Monitoring is the routine tracking and reporting of high priority information about aprogramme or project, its inputs and intended results. Monitoring information has to be used at all levels of the system for self-assessment and tracking the progress and decision making. It is therefore important for everyone involved to be clear about which indicators are being monitored as well as the information to be collected and how.
Evaluation is an assessment, as systematic and impartial as possible, or project, programme, strategy, policy, theme, sector, operational institutional performance. Evaluation aims at determining the relevance or appropriateness, effectiveness, efficiency, impact and
sustainability of the interventions and contributions to the organizations. The evaluation
process and methods should be decided during the initial planning process. This requires
prior planning and may require the use of external resource persons / experts. Evaluation
should be based on a set of indicators developed during the initial planning process.
Responding to the question: What are the core indicators we can agree on to use in measuring impact of a standard youth enntrepnership programme? The indicators are:
- Number of new businesses created;
- number of jobs created;
- increased incomes for existing business;
- number of youths starting businesses from urban high density areas
At my organisation we have used a number of impact assessment methods. Before discussing these it is useful to note that "Social Impact Assessment can be defined in terms of efforts to assess or estimate, in advance, the social consequences that are likely to follow specific policy actions (including programs and the adoption of new policies), and specific government actions. It is a process that provides a framework for prioritizing, gathering, analyzing, and incorporating social information and participation into the design and delivery of developmental interventions. It ensures that development interventions: (i) are informed and take into account the key relevant social issues; and (ii) incorporate a participation strategy for involving a wide range of stakeholders" (Misra, 2004:1).
We have used the following impact assessment methods at work:
- Outcome indicators for outcome level results/objectives. Outcome indicators assess
progress against specified outcomes/objectives. For example in 2007 my organisation aranged some business development services that included skills training, entrepreneurship training and book-keeping to small scale women miners. An M & E visit was conducted in 2009 to assess the impact of the training and what interventions were required if any.
Hi All
Monitoring and Evaluation is a useful method for measuring impact assessment. In my organization measuring impact assessment of youth entrepreneurship programmes is usually done by monitoring beneficiaries. Core Indicators should reflect the underlying reasons for designing and implementing the programme and so the logical framework which reveals details on specific objectives and goals of the programme, expected output, planed activities, targets, etc, serves as a guide to determining appropriate indicators. However, common indicators are: Numbers of new businesses created, Level of production and volume of service of various trainees, quality of products or services, no jobs of created,etc.
Dear Jealous and Colleagues,
Showing the results or on the issue of impact assessment of BDS or training programmes, they can be measured through monitoring and evaluation. Some of the indicators we can measure are individual level impacts: Acquisition of new knowledge and skills, Application of knowledge and skills, improved skills, knowledge and capabilities of workers, increased motivation of trainees and Control over decisions related to business/ enterprise.
Helo! I am very happy about this discussions, especially about the impact assessment. As said ealier Iam from the SIYB Master Training and one of the requirements for my certification is for me to do a complete BDS impact assessment. All this info shared will be useful for me. I cannot thank you enough.
wjmbongwe
good day all
Impact assessmentis achieved through monitoring and evaluation. The perpose of M&E is to inditify what is working well so that it can continued or improved, or what id not working well and should be stopped. How can we measure impact of BDS inteventions? have to assesse the sucess of the the programme in obtaining pre-determined objectives or goal. the intention is to determine the relevance, efficiance, effectiveness and sutainabilty of of the programme to the organisation. in addition to the aready given indicators , the standard indicators to use in your enterprenuership programme include, among others, % of tenders allocated Youth funded projects, number of Youth Involved income generatimg activites and number of Youth trained in basic business management etc.
Good day colleuges, Impact assessment of BDS is feasible in the sense that anything that has input, output and expected outcome components must be monitored and evaluated in order to ensure that the project is on course and cachieving the desired results.The indicators that may be used include the growth of the project; the number of additional jobs created; the regular returns on investment, etc.
hello all,
Impact assessment should be at the level of jobs created and incomes generated. Howeer this should be done through a logical process of agreeing on the strating points/baselines, the inputs (including BDS) and outcomes for training, the knowledge and skills acquired which are then used to create jobs and incomes. It must be pointed out that its not as straightforward as it solunds because there are a number of factors that need to be considered eg the environmment, politically, economically etc. For example in a high inflation environment, training and BDS might not have the impact it has in other environments.--Smauye 13:34, 15 April 2011 (UTC)
The impact of BDS interventions can be measured by looking at the extent to which the initial objectives have been met. This is possible if a baseline study was conducted prior to the interventions.
The core indicators to use in measuring the impact of a standard youth entrepreneurship programme include number of jobs created and income levels.
Of all the youth entrepreneurship programmes I have been involved with, none had an impact assessment conducted.
