Evaluation of eLearning for Effective Practice Guidebook/Class Projects/Mareena - project page

Mareena - Evaluation Plan

[Mareena this is an excellent plan. You have given the reader good detail about the reasons for the project, and obviously thought carefully about the design of the evaluation. The combination of a needs analysis and formative evaluation of the initial use of the Wimba classroom should ver very useful. There is a good description about the methods you intend to use, and there are certainly a range of strategies to collect data. This will add a broad dimension to your findings. The purpose, evaluation questions, decisions and data collection tools all fit together very well. You should obtain some very useful information. I am impressed by the budget and the list of references. You have obviously done a lot of reading and ecploration into the subject. I have added italics to name of the journal, and the book - just continue on with the rest of the list to ensure you are using APA referencing throughout.

I am really looking forward to reading your results and the report. --Bronwynh 23:40, 15 November 2010 (UTC)]

Introduction

This plan describes the evaluation of Wimba pilot project to ESOL students at the Manukau Institute of Technology. A formative evaluation will be carried out during the pilot to find out how effective the Wimba project is in supporting ESOL remote students. The evaluation will include a need analysis questionnaire, the researcher’s own documentation, staff feedback, student survey and a short poll, and an expert review. The plan will introduce the background, purpose, audience for this study. The data collection method and the questions to be addressed will be outlined and the timeline and budget will also be considered.

 Background:

The evaluator is planning to do a Wimba project to support ESOL students in general and the Newmarket branch specifically at the Manukau Institute of Technology (MIT). MIT main campus is in Otara where the researcher’s role at the Learning Support Centre is to give language support to all students, mainly ESOL. Students who need extra academic/language support can access learning support centre service outside their class time. Students can come to the Learning Support Centre individually or with a group depending on their need. Lecturers can also send referral to us when they identify students needing additional support. Learning Support Centre staff also support academic staff by providing contextualised workshops on specific subjects at the lectures’ request.
There is an ESOL branch at Newmarket in the city, but unfortunately the students do not access the Learning Support Centre services because it is too far for them to come. Time and resources are the main factors that limit the learning advisors from going to Newmarket. Therefore, the researcher thought of facilitating students via a Wimba classroom.
Wimba is a live virtual classroom with audio/ video application enhanced with content sharing facility. “Wimba Classroom has been carefully architected to give learners access to teachers, courses, and content beyond the four walls of a physical classroom or campus” (Wimba, Inc., 2009).

Using computer as a supplemental teaching tool is not a new concept in the field of education. Computers can not only provide a rich learning environment but also “can incorporate various learning strategies as well as accommodate a variety of learning styles” (Ybarra, & Green, 2003).
A range of studies have proven that effective use of technology can facilitate and reinforce language learning such as reading, writing, grammar, vocabulary development, speaking and pronunciation skills (Kang and Dennis, 1995; Lewis, 1997; Liaw,1997). According to Case and Truscott (1999), cited by (Ybarra, & Green, 2003) students have been able to improve their sight word vocabulary, fluency, and comprehension using computer as a tool. Cromley (2000) argues that “Technology can tap different skills than do textbooks and group work”.
Although the advent of Wimba is relatively new, the impact of using Wimba in language teaching has already been proved (Ward, 2005; (Lafford & Lafford 2005). A study done in University of California, Los Angeles (Kudyma, Aoki, & Liu) illustrates a wide variety of creative uses of Wimba and explores its benefit to language instruction. The study demonstrated that “the use of Wimba adds a new dimension to existing language instruction and can bring about a great impact on language learning”.
Wimba tools allow teachers to create different types of questions such as multiple-choice, short answer questions and true/false etc and arrange them into assessment. Teachers can then view the answers, grade them and give individualised feedback. Teachers are also able to use multiple mode of teaching using pictures, animation, videos and audios to make the lesson fun and exciting.
Based on this wealth of potential benefits that computers and technology can offer to the ESL students the proposed Wimba classroom will be utilized to facilitate language learning to the distance student population.
A suitable time will be negotiated with the students and staff to run Wimba sessions. Students need to make an appointment in advance with the administrator or a staff member to avoid too many students logging in simultaneously for different areas of help. Approximately one hour will be given to one student or a group at a time.

Purposes

The purpose of this project is to conduct a need analysis about the type of language learning support services which can be provided online, and formatively evaluate the use of the Wimba classroom.

Audiences:

MIT has an ethnically, culturally, and linguistically diverse group of ESOL students. The primary and secondary audiences of this evaluation are ESOL students and teaching staff. Learning Technology Centre Manager will also be involved for an expert review.

Decisions:

 The findings of this study will be used to influence the decision on how to run the Wimba class to support distance students. There will be discussion to decide how the project can be improved, and whether the project be continued or extended to other remote campus locations, and what would be required to do so.

Questions:

The questions this study is attempting to answer are as follow:
1. What type of online learning support services do ESOL students need to assist them with language learning at a distance?
Sub-questions
• What sort of language support do students want to access online?
• How many students used the remote service during the pilot?
• What are the reasons for a high or low turnout of students?

2. How appropriate is synchronous online support using the Wimba classroom for ESOL students?
Sub-questions
• How suitable is the Wimba classroom for online interaction and support?
• What is the response of students to the Wimba class?
• How effective do students find the Wimba classroom for language learning?
• What barriers did the students encounter during the process?
• Were the problems easily rectified? Why/why not?

3. What type of skills (technological, learning, communication) do ESOL students need for online language learning?
Sub-questions
• How do students use the tools in the Wimba classroom for language learning?
• What skills (technological, learning, communication) do students [believe they] require to use the Wimba classroom?

Methodology:

This study will be a formatively evaluated during one month trial. Multiple type evaluation method will be used to evaluate this project using Eclectic-Mixed Methods-Pragmatic Paradigm. Green et al. 1989 (cited by Neto, 1997) highlight that different use of instruments provides consistency of findings and adds richness and detail to the study. “The use of mixed-method is likely to increase the quality of final results and to provide a more comprehensive understanding of analyzed phenomena” (Neto, 1997). This will also enable triangulation of data ensuring validity and reliability of findings ( Hedberg & Reeves, 2003).

Student survey & a short [student] poll, staff feedback, expert review and the evaluator’s documentation will be used for data collection. Student survey and the short poll would give quantitative data while the lecturers’ feedback, expert review and the evaluator’s own documentation will provide qualitative data.

Types of data collection devices
Student survey & a short [student] poll, staff feedback, expert review and the evaluator’s documentation will be used for data collection. Student survey and the short poll would give quantitative data while the lecturers’ feedback, expert review and the evaluator’s own documentation will provide qualitative data.
Student survey and student poll
ESOL staff teaching at both campuses Otara and Newmarket will be approached for an initial survey session. Minimum five ESOL class at intermediate level or above will be used for this initial survey. There will be approximately 15 students in one class. Maximum number of students in one class will not exceed 22. However, it is predicted that the there will be some absentees in the class. The survey will be used to find student demographics, establish their language and to find out if they have the technology skills to access online language support services. The questionnaire can be completed online Student survey.

The link would be posted to the students through the Blackboard-eMIT interface.
In addition to that students will be asked to complete a short poll after each session.

Short poll

The questions are as follows:

Do you agree with the following statements?
Please rate them from 1 (not at all) to 4 (yes, completely).

	No, not at all	No, not really	Yes, a bit	Yes, completely
The lesson was interesting   The lesson helped me understand the topic I am comfortable using the Wimba classroom I would like to see more ESL Lessons through Wimba

Lecturer feedback
Lecturers’ cooperation and support is key to this project so they will be approached to encourage the students to make use of this service. Initially, they will be offered whole class sessions through Wimba just to give a feel of how it works. Lecturers may stay in the class to see students’ response to Wimba, although it is not necessary.

After the initial sessions students will be encouraged to make individual /group appointments according to their needs. As the Wimba session will be held outside of the class time it is not necessary for the lecturers to stay and observe the students. However, some lecturers may choose to stay out of their own interest. Feedback will be gathered from those who are willing to have their say. The following questions will be sent as an email attachment to prompt the lectures.
• What is your opinion on the introduction of Wimba class to ESOL students?
• How easy is it for the students to navigate and find the Wimba class on emit?
• Do you think the students need to up skill in ICT to use the Wimba classroom?
• What are the drawbacks or difficulties of using the online support programme with ESOL students?
• What advantages are there?
• Do you think that the technologies employed were helpful for the students to participate and learn?

Expert Review
Ongoing consultation will be held with the Learning Technology Centre (LTC) manager throughout the trail period. At the end of the pilot programme an informal meeting will be arranged to get the LCT manager’s view on the project. The following questions will be considered for discussion and notes will be taken.

• Do you think the use of Wimba classroom is the best way to support our remote students?
• What ICT support is in place specifically for e-learning and is this support regularly reviewed?
• Does the institution provide a reliable, secure and up-to date information technology infrastructure that allows for the continuation of this programme?
• What on-going support is available for people new to eLearning?
  

Documentation of evaluator observations
While implementing the plan the evaluator will keep a reflective journal to record her thoughts on what went well and what didn’t and why.
At the end of one month period a report will be written outlining the pilot project and critiquing the process. Outcomes will be discusses in terms of the evaluation questions.

Sample

The ESOL student population studying at both campuses Otara and Newmarket will be the target for the project. Initially, intermediate to advanced level students will be involved for easy communication. If time permits we may involve upper elementary to intermediate level students as well.

Limitations:

As the evaluator is planning to involve only upper intermediate to advanced level students in the project the sample size will be small. This gives a limited view for the results. Later, when low level ESOL students access the service, new issues may arise. Certain evaluation methods like Interviews and videos of focus group were avoided due to the time restriction of the requirements for the course assessment. The evaluator also wants to acknowledge errors or misjudgements in her thinking due to her own assumptions.

Logistics and Timeline

The need analysis survey will be conducted during class visits. The lecturers’ permission will be sought prior to the visit. The survey will be collated and majority need would be prioritized. Resources then will be prepared and loaded to the Wimba class based on their needs.
First Wimba workshop will be conducted to selected classes during class hours involving all students to expose them to the Wimba learning environment. A technology staff will be made available to troubleshoot problems and the lecturer may observe the session.
Later on students will make individual or group bookings outside their class time to get language support via Wimba classroom. After each session student feedback will be gathered via s short poll.
Time Line:
Oct. 1- Oct. 22: Prepare and present the evaluation plan
Oct. 25-Oct. 29 : Conduct the survey & analyse the data
Nov.1 –Nov.30: Trial the project & collect feedback from staff & students
Dec.1- Dec.20: Analyse the data and write the evaluation report

Budget

 Budget is estimated based on the cost of the staff time spent in preparation, Professional Development time, travel and parking cost.

Preparation of Evaluation Plan15 hours @ $30	$450
Administering surveys and short polls 7 hours @ $30	$210
Analysing Interpreting data 3 hours@ $30	$90
PD time 4 hrs at $30/hr	$120
Travel time to Newmarket campus 2 hours @$30	$60
Petrol cost 16 km x 4 @ $0.80 per km	$51.20
Parking cost $5 per hour for 6hours	$30
Trial presentation of the project 20 hours @ 30	$600
Collecting feedback & analysing data 2 hours @ 35	$70
Evaluation report 10 hours @ 35	$350
Total	$2771.20

References

Cromley, Jennifer G. (2000). Learning with computers: the theory behind the practice. Focus on Basics, 4(C). Retrieved October 22, 2010 from http://www.ncsall.net/index.php?id=303

Kang, S.H. & Dennis, J.R. (1995). The effects of computer-enhanced vocabulary lessons on achievement of ESL grade school children. Computers in the Schools, 11 (3), 25-35.

Kudyma, Ganna, Aoki, Hiromi, & Liu, Ted. (n.d.). The use of oral computer-mediated communication. Retrieved October 31, 2010 from http://php.csumb.edu/wlc/ojs/index.php/ds/article/viewFile

Lafford, Peter A., & Lafford, Barbara A. (2005). CMC technologies for teaching foreign languages: What’s on the horizon? CALICO Journal, 22(3), 679-709.

Lewis, P. (1997). Using productivity software for beginning language learning Part 1. The word processor. Learning and Leading with Technology, 24 (8), 14-17.

Liaw, M.L. (1997). An analysis of ESL children's verbal interaction during computer book reading. Computers in the Schools, 13 (3/4), 55-73.

Neto, John SydenstrickerInitials. (1997). Research design and mixed-method approach . Retrieved October10,2010 from http://www.socialresearchmethods.net/tutorial/Sydenstricker/bolsa.html

Reeves, T. C., & Hedberg, J. C. (2003). Interactive Learning Systems Evaluation. Englewood Cliffs, NJ: Educational Technology Publications.

Ward, Daniel. (2005). El Maestro’s Voice. Language Magazine, 4(12), 22-23.

Wimba, Inc.(2009). Wimba classroom for higher education. RetrievedOctober18,2010 from http://www.wimba.com/solutions/higher-education/wimba_classroom_for_higher_education/

Ybarra, Renee, & Green, Tim. (2003). Using technology to help esl/efl students develop language skills. The Internet TESL Journal, IX(3), Retrieved October 22, 2010from http://iteslj.org/Articles/Ybarra-Technology.html

Appendix

Survey

I have prepared a need analysis survey using 'surveymonkey' and posting the link here for feedback please. [[<a href="http://www.surveymonkey.com/s/DRK5T39">Click here to take survey</a>|Needs analysis survey]]