Evaluation of eLearning for Effective Practice Guidebook/Class Projects/Katie and Lisa - project page

Katie and Lisa - Evaluation Plan

[Katie and Lisa - This is an excellent plan and you have given the reader a comrehensive overview of what is at stake. All the components fitting together perfectly - purpose, questions and decisions fit well with the data collection methods. It is very good that you have combined your projects because you will have more data overall. The matrix makes it easy to visualise how the data collection methods fit with each of the evaluation questions you have chosen. Both the survey and interview questions are very good and should produce some very useful information. The methods section is well explained. Remember triangulation is necessary to ensure validity and reliability of the data so that different perspectives are obtained. You have a good list of references. Great work! --Bronwynh 11:33, 4 November 2010 (UTC)]

Introduction

'The main concerns participants had at the end of the project about ICTs in teaching and learning were...a heightened awareness that the need for professional development in the area was likely to continue after the end of the project, and a lack of time in which to undertake such development.'
Ham, Gilmore, Kachelhoffer, Morrow, Moeau & Wenmoth, 2002.

This evaluation plan describes a needs analysis which will be conducted with teaching staff at the conclusion of a 3 year ICT cluster contract, from this it has become evident that changes should be made to the delivery of professional development sessions. A needs analysis has been chosen as it will provide information to help make the decision as to what professional development needs to be put in place for teachers to continue to teach ICT effectively. The evaluation includes decisions, background and key questions. The questions have been divided up to allow a range of staff to be surveyed and/or interviewed. These will explore the principal concerns of staff and allow the report facilitators to ascertain the needs and requirements of each school. Both qualitative and quantitative forms of data (a mixed methods approach) will be gathered.

Report evaluators - Katie Gibbs and Lisa Reid

Background

Both primary schools are nearing completion of the 3 year ICT contract. The contract run by the Ministry of Education since 1999 aims to improve 'Confidence and Capability: Increasing information literacy skills and competencies' (Ministry of Education,2006). Schools are grouped into clusters during the time of the contract. The grouping allows staff to work with other schools not always similar to their own (e.g.different decile and access to resources). As part of the contract lead teachers within each school have been involved in training with external facilitators and a number of applications have been introduced to them and in turn to be introduced to other staff. How the applications are introduced is often dependent on the school but research carried out by the Ministry of Education (Ham et al., 2002) had found that the 'trickle down' model worked best. At present this is carried out in teams that are formed from teachers of similar levels and have a team leader who is not necessarily a lead teacher for ICT. These teams plan together.

At the conclusion of the contract a number of issues could arise: lead teachers in most cases are paid extra only during the period of the contract and now at the end would the same level of support for other staff be provided; contact between clusters schools may not be maintained; also the need to encourage sustainability of learning if new staff are employed after the contract period. Both schools need to evaluate their present situation and decide the way forward. A needs analysis will provide the necessary direction.

Purpose

The purpose of this evaluation is to explore and determine the professional development which needs to be put in place for teachers to teach ICT effectively. This involves finding out the methods of learning staff prefer including their learning environment, how confident they are, and what eLearning practices are available to motivate staff and ensure sustainability.

The evaluation will focus on two questions, based on two eLearning Guidelines (Milne & Dimock, 2006). A sub-set of questions have been developed to ensure that the eLearning quidelines are answered and researched adequately.

• TD11 Should staff use a team approach to develop the ICT skills they require, and to teach ICT applications confidently?

• How do staff prefer to learn?

• What are the most appropriate learning methods to deliver eLearning applications to teams?

• How can trainers ensure collaborative learning is taking place in teams?

• TO10 Is good eLearning practice available to staff in a way they can adopt and adapt in their own work?

• What further professional development is available to staff to enable them to engage in eLearning?

• What motivates staff to engage in eLearning?

• What on-going/extra support is available or required by new users of eLearning?

Limitations

There are a number of limitations associated with this evaluation. Both evaluators were not involved in the ICT contract so are compiling the report as observers. As with any survey the staff who are involved, how they complete and interpret the questions can threaten the validity of the design. As said by Reeves & Hedberg (2003) you need "to remain aware of the competing pressures and demands on each stakeholder as his or her own interest and biases. Managers and workers have different perspectives on the needs addressed by training" (p.122).

Some of the data will be partly gathered using an online survey (questionnaire) because of time limitations, but as stated by Reeves & Hedberg (2003) using "a questionnaire as a last resort, as it requires time to prepare, it may not be welcome, and the response rate will almost certainly be lower than that of any other method" (p.122). Face-to-face interviews rely on goodwill of the individual staff, that and issues of time may limit the number completed.

As both of the report evaluators work full time and are required to work around their full time commitments, completing this assignment within the given time frame could also be seen as a limitation.

Audiences

The primary audience will be the evaluators and the course facilitator. The secondary audience will the teachers involved in the survey, leadership team and if required the Board of Trustees.

Decisions

The information from this evaluation will be used to inform each school about recommendations for the professional development which needs to be put in place for the individual and role played by teams. Solutions for professional development which is sustainable will be investigated. If the report is adopted by the leadership team at each school it will influence any future decision making in teachers' effective practice of ICT.

Methods

The evaluation will use a needs analysis approach. This will allow the report facilitators to ascertain the needs and requirements of each school as well as exploring other possibilities. Both qualitative and quantitative forms of data (a mixed methods approach) will be gathered as it will allow the facilitators to gain an in-depth understanding of the ways staff prefer to learn. All teaching staff (approximately 40) will be invited to answer the survey questions. The lead teachers, team leaders and Assistant Principal (in-charge of ICT) will be invited to answer an additional questions in a face-to-face interview. The aim of collecting the data is to talk to key people and create a picture that allows the next stage in ICT professional development to be formed. The survey will be used to gather baseline information, and the interviews to build on survey responses where possible. The final documentation will be presented to management.

It is important to use the triangulation data collection method to collect relevant information, and using a mix of methods ensures triangulation. Triangulation refers to when more than two methods is used in a study with a view to double (or triple) checking results[1]. This research will use different data sources (surveys and interviews) involving different informants (lead-teachers and individual teachers) that complement each other.

• Two interviews will be carried out with lead teachers.

• Three interviews with individual teachers will be carried out face-to-face.

• An online survey consisting of questions will be used to gather information from the present teaching staff and lead teachers, approximately 40 surveyed between the two schools.

Questions to be used for the interview and survey are listed in Appendices 1 & 2.

Data collection matrix

Questions	Survey	Interviews lead teachers	Interviews Individual teachers
TD11 Should staff use a team approach to develop and teach the applications [course]?	X	X	X
How do staff prefer to learn about ICT?	X	X	X
What are the most appropriate learning methods to deliver eLearning applications to teams?	X	X	X
How can you assure collaborative learning is taking place in teams?		X
TO10 Is good e-learning practice available to staff in a way they can adopt and adapt in their own work?	X	X	X
What further professional development is available to staff to engage in eLearning?	X	X	X
What motivates staff to engage in eLearning?	X	X	X
What on-going/extra support is available to new users of eLearning?		X

Logistics and Timeline

Dates:	Events:
23 August to 19 September	Writing up initial plan Contact staff to be sampled Compile questions for interviews (lead teachers & staff) Organise dates and times for interviews Gather and review appropriate readings for research
20 to 26 September	Email survey (During school holidays so may be an issue with getting back results)
11 to 31 October	Conduct interview with lead teachers and class teachers Answer questions from staff if needed Analyse evaluation results
1 to 28 November	Finalise results
29 November to 3 December	Present summary of project to the class Hand in the final of evaluation report

Budget

60 hours @ $50 per hour x 2 facilitators = $6000 + some additional costs for survey participants. These costs are unknown but will involve the payment of chocolate on the completion of survey.

References

Gratton, C. & Jones, I. (2004). Research methods for sport studies. New York: Routledge.

Ham, V & Gilmore, A., Kachelhoffer, A., Morrow, D., Moeau, P. & Wenmoth, D.(2002). What makes for effective teacher Professional Development in ICT? Retrieved 14th September, 2010 from: http://www.educationcounts.govt.nz/publications/ict/5807

Milne, J., & Dimock, E. (2006). eLearning Guidelines. Guidelines for the support of eLearning in New Zealand. Retrieved 17 September, 2010 from: http://elg.massey.ac.nz/Guidelines-questions.pdf

Ministry of Education (2006). ICT Strategic Framework for Education. At: http://www.minedu.govt.nz/NZEducation/EducationPolicies/Schools/Initiatives/ICTInSchools/ICTStrategicFrameworkEducation.aspx

Reeves, T., C. & J. G. Hedberg (2003). Interactive Learning Systems Evaluation. Englewood Cliffs: Educational Technology Publications.

Appendix 1 - Survey Questions for Teaching staff

The survey will be emailed to staff using an online survey.

Statement on opening page: - Dear staff, This survey is aimed at taking less than 10 minutes to complete. Its purpose is to look at what ICT professional development needs to be put into place as we come to the end of the ICT cluster contract. I appreciate the time you have taken to share your ideas and thoughts. Kind regards...(signed by individual evaluators).

Data Collection:	Questions:
How do staff prefer to learn about ICT?	How do you prefer to access professional development (PD) on eLearning? Multiple answer questions - choose as many as appropriate on my own - self paced formal training sessions buddy groups informal groups as the need arises colleague/s showing me Other Comment - please explain further (Comment box) Now looking at your process of learning. Read the following statements make a choice on the scale that best suits you. Learning about ICT is challenging. strongly agree, agree, disagree, strongly disagree I prefer to learn about ICT in a group or team. strongly agree, agree, disagree, strongly disagree ICT is easier to learn when I have the support of others. strongly agree, agree, disagree, strongly disagree I prefer to teach myself how to use ICT. strongly agree, agree, disagree, strongly disagree
What are the most appropriate learning methods to deliver eLearning applications to teams?	In your opinion, which of the following methods are appropriate for teams to learn to use ICT applications. Choose as many as necessary. Please explain your answer. mentoring team discussions team tasks to complete hands on training with an expert presentations by experts show and tell by colleagues get togethers with peers sharing ideas observing colleagues peer training sessions time out to practise with the team time out to practise alone any other suggestions? Looking back at previous professional development you have had (not just eLearning) what learning methods works best for you. Comment - please explain (Comment box)
What further professional development is available to staff to engage in eLearning?	Now that we are at the end of the ICT contract. How do you think you would benefit from further professional development in elearning? Comment - please explain further (Each will have a comment box) What do you believe is offered in the way of PD for eLearning at the school? How are you supported by the school to develop your capability for eLearning? What do you know about the PD which is available to teachers in the outside school? What type of PD support would help you to develop your capability for eLearning?
What motivates staff to engage in eLearning?	What motivates you to engage in eLearning? I am motivated to engage in eLearning so I can ... Choose as many as appropriate use current technologies for teaching my students keep up with national trends provide interactive resources for students develop my skills in using digital materials because the school expects me to eLearning is an area that interests me gaining new ICT skills is required for my role Other Comment - please explain further (Comment box)

Appendix 2 - Interview Questions for Lead teachers/Team Leaders

This is to be a face-to-face interview. Not all lead teachers are team leaders so some adjustments may be required.

Data Collection:	Questions:
How do staff prefer to learn?	How do you think your team prefers to learn? Do they like discussing ideas and new approaches with each other, showing each other what they are doing etc. As a lead teacher you were involved in supporting your colleagues. This quote was taken from Ministry research 'Teachers valued most those professional development strategies which maximised the time available to them to come to grips with ICT skills and uses, and which combined substantial `time out' on the one hand with ongoing access to collegial support on the other. `Trickle down' models appear to have been more successful in relation to the development of skills in schools’ (Ham et al., 2002) Do you agree or disagree with this statement?
What are the most appropriate learning methods to deliver eLearning applications to teams?	What do you believe is working or likely to work when delivering eLearning to your team? Do you think one method of PD would work for all applications? Take for example using a professional blog. How do you think your team needs to be supported to learn about blogs? Prompt: should they set up a professional blog and learn by becoming familiar with doing it themselves? Looking back at previous professional development (not just eLearning) what learning environment works best for you? Your team? Prompt - group support, hands on training
How can you assure collaborative learning is taking place in teams?	What would you do for your team to ensure collaborative learning is taking place? Prompts - Share ideas, feedback, observe other teams
What further professional development is available to staff to engage in eLearning?	Do you think your team would benefit from further Professional development in eLearning? Prompt - As a whole staff, in team meetings or buddy groups
What motivates staff to engage in eLearning?	What engages you personally in eLearning? Prompts - Self taught, given tasks to complete etc What do you believe motivates your staff in eLearning? Prompts - as a team and/or as a whole
What on-going/extra support is available to new users of eLearning?	How do you think professional development in your team can be sustained so their learning is ongoing? Our school? How do you currently support teachers to learn new ICT applications? If a new staff member is employed and in your team what would you do to support eLearning Professional development? Prompts - Is what is in place working?, adequate?, can it be sustained?