Evaluation of eLearning for Effective Practice Guidebook/Class Projects/Katie and Lisa - project page
Contents
- 1 Katie and Lisa - Evaluation Plan
Katie and Lisa - Evaluation Plan
[Katie and Lisa - This is an excellent plan and you have given the reader a comrehensive overview of what is at stake. All the components fitting together perfectly - purpose, questions and decisions fit well with the data collection methods. It is very good that you have combined your projects because you will have more data overall. The matrix makes it easy to visualise how the data collection methods fit with each of the evaluation questions you have chosen. Both the survey and interview questions are very good and should produce some very useful information. The methods section is well explained. Remember triangulation is necessary to ensure validity and reliability of the data so that different perspectives are obtained. You have a good list of references. Great work! --Bronwynh 11:33, 4 November 2010 (UTC)]
Introduction
'The main concerns participants had at the end of the project about ICTs in teaching and learning were...a heightened awareness that the need for professional development in the area was likely to continue after the end of the project, and a lack of time in which to undertake such development.'
Ham, Gilmore, Kachelhoffer, Morrow, Moeau & Wenmoth, 2002.
This evaluation plan describes a needs analysis which will be conducted with teaching staff at the conclusion of a 3 year ICT cluster contract, from this it has become evident that changes should be made to the delivery of professional development sessions. A needs analysis has been chosen as it will provide information to help make the decision as to what professional development needs to be put in place for teachers to continue to teach ICT effectively. The evaluation includes decisions, background and key questions. The questions have been divided up to allow a range of staff to be surveyed and/or interviewed. These will explore the principal concerns of staff and allow the report facilitators to ascertain the needs and requirements of each school. Both qualitative and quantitative forms of data (a mixed methods approach) will be gathered.
Report evaluators - Katie Gibbs and Lisa Reid
Background
Both primary schools are nearing completion of the 3 year ICT contract. The contract run by the Ministry of Education since 1999 aims to improve 'Confidence and Capability: Increasing information literacy skills and competencies' (Ministry of Education,2006). Schools are grouped into clusters during the time of the contract. The grouping allows staff to work with other schools not always similar to their own (e.g.different decile and access to resources). As part of the contract lead teachers within each school have been involved in training with external facilitators and a number of applications have been introduced to them and in turn to be introduced to other staff. How the applications are introduced is often dependent on the school but research carried out by the Ministry of Education (Ham et al., 2002) had found that the 'trickle down' model worked best. At present this is carried out in teams that are formed from teachers of similar levels and have a team leader who is not necessarily a lead teacher for ICT. These teams plan together.
At the conclusion of the contract a number of issues could arise: lead teachers in most cases are paid extra only during the period of the contract and now at the end would the same level of support for other staff be provided; contact between clusters schools may not be maintained; also the need to encourage sustainability of learning if new staff are employed after the contract period. Both schools need to evaluate their present situation and decide the way forward. A needs analysis will provide the necessary direction.
Purpose
The purpose of this evaluation is to explore and determine the professional development which needs to be put in place for teachers to teach ICT effectively. This involves finding out the methods of learning staff prefer including their learning environment, how confident they are, and what eLearning practices are available to motivate staff and ensure sustainability.
The evaluation will focus on two questions, based on two eLearning Guidelines (Milne & Dimock, 2006). A sub-set of questions have been developed to ensure that the eLearning quidelines are answered and researched adequately.
- TD11 Should staff use a team approach to develop the ICT skills they require, and to teach ICT applications confidently?
• How do staff prefer to learn?
• What are the most appropriate learning methods to deliver eLearning applications to teams?
• How can trainers ensure collaborative learning is taking place in teams?
- TO10 Is good eLearning practice available to staff in a way they can adopt and adapt in their own work?
• What further professional development is available to staff to enable them to engage in eLearning?
• What motivates staff to engage in eLearning?
• What on-going/extra support is available or required by new users of eLearning?
Limitations
There are a number of limitations associated with this evaluation. Both evaluators were not involved in the ICT contract so are compiling the report as observers. As with any survey the staff who are involved, how they complete and interpret the questions can threaten the validity of the design. As said by Reeves & Hedberg (2003) you need "to remain aware of the competing pressures and demands on each stakeholder as his or her own interest and biases. Managers and workers have different perspectives on the needs addressed by training" (p.122).
Some of the data will be partly gathered using an online survey (questionnaire) because of time limitations, but as stated by Reeves & Hedberg (2003) using "a questionnaire as a last resort, as it requires time to prepare, it may not be welcome, and the response rate will almost certainly be lower than that of any other method" (p.122). Face-to-face interviews rely on goodwill of the individual staff, that and issues of time may limit the number completed.
As both of the report evaluators work full time and are required to work around their full time commitments, completing this assignment within the given time frame could also be seen as a limitation.
Audiences
The primary audience will be the evaluators and the course facilitator. The secondary audience will the teachers involved in the survey, leadership team and if required the Board of Trustees.
Decisions
The information from this evaluation will be used to inform each school about recommendations for the professional development which needs to be put in place for the individual and role played by teams. Solutions for professional development which is sustainable will be investigated. If the report is adopted by the leadership team at each school it will influence any future decision making in teachers' effective practice of ICT.
Methods
The evaluation will use a needs analysis approach. This will allow the report facilitators to ascertain the needs and requirements of each school as well as exploring other possibilities. Both qualitative and quantitative forms of data (a mixed methods approach) will be gathered as it will allow the facilitators to gain an in-depth understanding of the ways staff prefer to learn. All teaching staff (approximately 40) will be invited to answer the survey questions. The lead teachers, team leaders and Assistant Principal (in-charge of ICT) will be invited to answer an additional questions in a face-to-face interview. The aim of collecting the data is to talk to key people and create a picture that allows the next stage in ICT professional development to be formed. The survey will be used to gather baseline information, and the interviews to build on survey responses where possible. The final documentation will be presented to management.
It is important to use the triangulation data collection method to collect relevant information, and using a mix of methods ensures triangulation. Triangulation refers to when more than two methods is used in a study with a view to double (or triple) checking results[1]. This research will use different data sources (surveys and interviews) involving different informants (lead-teachers and individual teachers) that complement each other.
- Two interviews will be carried out with lead teachers.
- Three interviews with individual teachers will be carried out face-to-face.
- An online survey consisting of questions will be used to gather information from the present teaching staff and lead teachers, approximately 40 surveyed between the two schools.
Questions to be used for the interview and survey are listed in Appendices 1 & 2.
Data collection matrix
Questions |
Survey |
Interviews lead teachers |
Interviews Individual teachers |
TD11 Should staff use a team approach to develop and teach the applications [course]? |
X |
X |
X |
How do staff prefer to learn about ICT? |
X |
X |
X |
What are the most appropriate learning methods to deliver eLearning applications to teams? |
X |
X |
X |
How can you assure collaborative learning is taking place in teams? |
|
X |
|
TO10 Is good e-learning practice available to staff in a way they can adopt and adapt in their own work? |
X |
X |
X |
What further professional development is available to staff to engage in eLearning? |
X |
X |
X |
What motivates staff to engage in eLearning? |
X |
X |
X |
What on-going/extra support is available to new users of eLearning? |
|
X |
|
Logistics and Timeline
Dates: | Events: |
---|---|
23 August to 19 September |
|
20 to 26 September |
|
11 to 31 October |
|
1 to 28 November |
|
29 November to 3 December |
|
Budget
60 hours @ $50 per hour x 2 facilitators = $6000 + some additional costs for survey participants. These costs are unknown but will involve the payment of chocolate on the completion of survey.
References
Gratton, C. & Jones, I. (2004). Research methods for sport studies. New York: Routledge.
Ham, V & Gilmore, A., Kachelhoffer, A., Morrow, D., Moeau, P. & Wenmoth, D.(2002). What makes for effective teacher Professional Development in ICT? Retrieved 14th September, 2010 from: http://www.educationcounts.govt.nz/publications/ict/5807
Milne, J., & Dimock, E. (2006). eLearning Guidelines. Guidelines for the support of eLearning in New Zealand. Retrieved 17 September, 2010 from: http://elg.massey.ac.nz/Guidelines-questions.pdf
Ministry of Education (2006). ICT Strategic Framework for Education. At: http://www.minedu.govt.nz/NZEducation/EducationPolicies/Schools/Initiatives/ICTInSchools/ICTStrategicFrameworkEducation.aspx
Reeves, T., C. & J. G. Hedberg (2003). Interactive Learning Systems Evaluation. Englewood Cliffs: Educational Technology Publications.
Appendix 1 - Survey Questions for Teaching staff
The survey will be emailed to staff using an online survey.
Statement on opening page: - Dear staff, This survey is aimed at taking less than 10 minutes to complete. Its purpose is to look at what ICT professional development needs to be put into place as we come to the end of the ICT cluster contract. I appreciate the time you have taken to share your ideas and thoughts. Kind regards...(signed by individual evaluators).
Data Collection: | Questions: |
---|---|
How do staff prefer to learn about ICT? |
How do you prefer to access professional development (PD) on eLearning? Multiple answer questions - choose as many as appropriate
Comment - please explain further (Comment box)
Learning about ICT is challenging.
I prefer to learn about ICT in a group or team.
ICT is easier to learn when I have the support of others.
I prefer to teach myself how to use ICT.
|
What are the most appropriate learning methods to deliver eLearning applications to teams? |
|
What further professional development is available to staff to engage in eLearning? |
Comment - please explain further (Each will have a comment box)
|
What motivates staff to engage in eLearning? |
What motivates you to engage in eLearning? I am motivated to engage in eLearning so I can ... Choose as many as appropriate
Comment - please explain further (Comment box) |
Appendix 2 - Interview Questions for Lead teachers/Team Leaders
This is to be a face-to-face interview. Not all lead teachers are team leaders so some adjustments may be required.
Data Collection: | Questions: |
---|---|
How do staff prefer to learn? |
|
What are the most appropriate learning methods to deliver eLearning applications to teams? |
Prompt: should they set up a professional blog and learn by becoming familiar with doing it themselves?
Prompt - group support, hands on training |
How can you assure collaborative learning is taking place in teams? |
Prompts - Share ideas, feedback, observe other teams |
What further professional development is available to staff to engage in eLearning? |
Prompt - As a whole staff, in team meetings or buddy groups |
What motivates staff to engage in eLearning? |
Prompts - Self taught, given tasks to complete etc
Prompts - as a team and/or as a whole |
What on-going/extra support is available to new users of eLearning? |
Prompts - Is what is in place working?, adequate?, can it be sustained? |