Evaluation of eLearning for Effective Practice Guidebook/Class Projects/Meena, Misha and Mareena - project page

Meena, Misha and Mareena - Evaluation Plan

Link to Mareena's - project page

Introduction

Provide an overview of the intentions and design of evaluation project, and introduce major sections of the plan as well as the primary people involved in writing the plan.

MishaI will be conducting a needs assessment evaluation of the digital literacy skill base of the students studying the L4 Certificate in Social Services.

Meena Introduction: Unit 5940: Produce a presentation using a desktop presentation computer application People credited with this unit standard are able to: • describe the use of a personal computer application for desktop presentation; and • design, create, and deliver a presentation using a personal computer application. Current Course delivery Currently we use course manual, and the delivery method is blended learning. Learners go through the book, and get assistance from classroom facilitators in case they have problem. No online help is provided and they use Blackboard to send their work and check their progress in the grade centre. a) Self-Directed Learning (asynchronous) b) Classroom contact (synchronous)

Background

Present any information which is needed to provide the reader with an understanding of the background of the eLearning that is being evaluated and the rationale.

Misha Currently we are in the process of the redevelopment of our L4 Certificate in Social Services. The current Certificate is offered both part time over a period not exceeding 3 years and Fulltime over one Semester. The Certificate is a stand alone qualification for anyone 18 years + who wishes to work in a Social Services Agency as a Social Services Youthworker or Helper. It equips students to take up a positon as a house parent, ...... The Certificate is also seen as a stepping stone to the Bachelor of Applied Social Work or the Diploma in Counselling. The redevelopment process has sought to streamline and focus the Certificate from ? of F2F papers to 4 blended learning theme papers. The current Certificate has suffered a number of additions over the past 14 years resulting in overlapping of content.

Meena Background: Current structure consists of learning form the Text book and exercise files are available online through blackboard emit. It is self paced, self directed learning and is part of Graphics cluster which is part of larger National Certificate in Computing Level 3 course. No part of this unit 5940 is available online except for assessments and access to Digital drop box to post their assignments online to assessor and check their grade through grade center.

Purposes

Describe the purposes of the evaluation, that is, what you are evaluating and the intended outcomes.

Misha We need to know how competent/notcompetent our students are in digital literacy, what access they have to computers, how confident they are about using computers, whether thay can follow a sequence in accessing information online, and related areas preferably before we design the online content of the courses in the  revised Certificate. One of the 4 papers that is being designed around producing an ePortfolio demands a certain degree of fluency in digital literacy. I am aware that the staff don't really understand all that this could entail.

Meena Purpose is to improve course effectiveness by evaluating students and facilitators feedback whether Unit 5940 will benefit going online in an asynchronous environment. It will indirectly determine whether or not current Learning structure of studying through books provides students enough understanding to complete their module successfully (TD2). Along with the need to deliver course online, it would also be necessary to determine what kind of e-learning strategy could be used and would be feasible for our audiences(TD3). This could be achieved through formative analysis by observing students go though sample of course material .

Limitations

Outline any limitations to the interpretation and generalizability of the evaluation. Also describe potential threats to the reliability and validity of the evaluation design and instrumentation. Meena Many factors affected the length of time required to complete the Evaluation Plan:

-Great deal of time was spent collecting information.

- Internal factors like Time and the methodology used were time consuming. -External factors that caused hurdle in meeting deadline was due to the accessibility and diversity of the students and the facilitators. -Analysing findings from interviews was a bit difficult, than with questionnaires, because of the range of answers received from different participants (Facilitators) as well as summarising new points they raised.

Audiences

Meena Audience for the needs assessment plan for this evaluation would be the Education Provider,Students,Course Facilitator,Programme Leader

Misha At this stage the primary audience for the Needs analysis is our Head of School, our Academic Manager and our teaching staff who are currently doing the rewrite of the Certificate.

Decisions

This section is probably the most difficult, but it should be included if the evaluation is to have meaningful impact on decision-making. Both positive and negative outcomes should be anticipated.

Questions

List your 'big picture' evaluation questions and sub-questions here. A key element of a sound evaluation plan is careful specification of the questions to be addressed by the evaluation design and data collection methods. The clearer and more detailed these questions are, the more likely that you will be able to provide reliable and valid answers to them. Note: The eLearning Guidelines can assist here. Meena The needs analysis is based on two eLearning Guidelines (TD2 and TD3) and aims to investigate following two questions:

1. Does the current Learning structure of studying through books provide students with enough understanding to complete their module successfully (TD2)? • Which aspects in Unit 5940 are successful and should be retained? • What are the factors which contribute to successful completion of the unit? • How well do students understand the content of the unit? • Are students able to apply what they have learned? 2. What kind of e-learning strategy could be used and would be feasible for the current [or potential?] audiences (TD3). • What design is preferred for the online content in unit 5940? • What is required for students to study effectively online?

Methods

Meena The evaluator will be conducting needs analysis on delivering course from text book to online web . Needs assessment will help figure: 1] Learner skills, abilities and needs 2] The readiness of Course Facilitators and their preferences in relation to teaching online. [1]. The evaluator will use Thomas C Reeves’s Eclectic-Mixed Methods-Pragmatic Paradigm approach from his Book Interactive learning systems evaluation(Year 2003), Because of its capability of handling complexity of modern technology and its focus on practical problems rather than on theories. [2]

Data Collection Analysis

Data collection comes from two of the stakeholders: 1] Firstly, the current students who have already completed the course or who are in the verge of course completion shared their experience and provided feedback to the survey questionnaire. Evaluator developed questionnaire survey for students, so as to ascertain what aspects of Unit Standard are currently effective and what design could be favoured to meet students learning needs, should the course go online.

2] Secondly, the Facilitators who facilitate the course in a self paced environment using course text book. Classroom facilitators will play major role in analysing the effectiveness of re-development of course delivery from text book to online. Evaluator created interview questions for the educators’ facilitators, who are directly involved with students learning .This will help determine their take on current structure and their opinion of change in course delivery from text book to online.

Mixed technique used: Quantitative data Italic textStudents will be provided with survey questionnaire split into following 3 sections: • Learning Style • Sustainability • Digital Technology This survey will have a preponderance of closed-ended questions. Total 11 students participated in the questionnaire. Quantitative data gathered will be presented in percentage frequency.

Qualitative dataItalic text In this evaluation plan, the second Data Collection are set of interviews questions. Six facilitators participated and shared their thoughts on current structure and about bringing a change in course delivery. This non-numerical data collection method was used, as it would allow for more diversity in responses and be open ended for facilitators to answer the interview questions. It would help in evaluating the current structure of the program as well as evaluate its capacity to adapt to new developments [3]

Data collection Matrix Data will be used to determine whether the programme has attained, or is on track to attain, its objectives using following matrix:

Sample

Meena

Instrumentation

Outline all the evaluation instruments and tools to be used in the evaluation. Actual instruments, e.g., questionnaire, interview questions etc., should be included in appendices. Meena Appendix 1 Questionnaire survery for students

Opening paragraph before taking students survey.
Please read take time to answer multiple choice questions. This Questionnaire is split into 3 sections and is intended to be used as part of a assignment for Needs Analysis to evaluate need to deliver PowerPoint Course (Unit 5940) online.

I Meena Adhikari would like to assure you, that it will be confidential and will be used solely for the purpose of my assignment for applied e-learning evaluation module. Thank you for your time.

Section

Survey Questions

Learning style Section

1) How do you like to learn?

Text based, I prefer to read from a text book than online

I understand better with pictures, and diagrams

I like learning using animations and sound with little text

I can learn using online resources.

I grasp better if I am engaged with lots of activities and examples online.

I prefer to get an immediate feedback, e.g. in an online quiz

2) Are you a self-motivated, independent learner?

I really enjoy the challenge of studying alone

I need the stimulation of a group.

I need to be engaged and need interaction with online study materials

I like to set learning goals for myself

3) Do you find the course book material clear, simple and easy to understand?

Too much text and too much reading

Yes, it is very helpful and step by step instructions are easy to follow

Not enough interactivity

not enough explanation about the topics

4) How capable are you of understanding concepts when reading your course textbook?

I am a good reader.

I have to hear information in order to retain it.

I usually don't remember what I read.

I don't know the level of my reading skills

Sustainability section

1) Do you have access to a computer and internet connection?

Just a computer

Yes, I have access to a computer and a broadband connection

Yes, I have access to a computer and a dial up internet connection

No, I don’t have access to a computer or an internet connection

2) How regularly will you be able to log on to the internet to study online?

Only once a week.

At least twice a week.

Probably once every two weeks.

I don't know for sure.

As often as I need to

Digital technology Section

1) How do you feel about studying online over the internet?

I am nervous about it. I am not sure it is for me.

I do not look forward to it, but I need the course.

I look forward to the experience.

• I am confident learning online

2) Do you know how to use a web browser (e.g., Firefox or Internet Explorer) to get around the Internet?

Yes. I browse the Internet frequently.

Somewhat. I have not had much exposure to it.

I have only seen my friends use it.

No, but I am willing to learn.

3) If you were to study online on web, how will you handle the situation if your computer (or software) breaks down at some point during the course?

I expect my instructor to be understanding and give me extensions.

I will get it fixed immediately and will use another system in the meantime.

I cannot afford for things to go wrong.

Nothing will go wrong. I have good equipment.

4) How good are you at following directions on instructions/assignments?

I like it when instructors go over homework directions orally.

I have difficulty understanding and frequently need clarification.

I can read and follow directions on my own.

I cannot follow directions very well.

5) What kind of support do you need to complete an online course successfully?

Guidance from the facilitators

Regular feedback

Help with using the Internet

Instructions for using the online resources

Other

6) How do you prefer to get help if you need more clarification?

Email

Coming to the classroom

discussion board

other students

facilitators

'Appendix 2 Interview questions for FACILITATORS Opening paragraph before asking the questions to the facilitators Thanks for taking time and providing feedback. Following interview questions are intended to be used for the sole purpose of my assignment, which is needs analysis to evaluate the need to deliver PowerPoint Course (Unit 5940) online. This assignment is part of Evaluation Paper which is part of Applied E-learning course.

Component	Interview questions
Does the current Learning structure of studying through books provide students with enough understanding to complete their module successfully?	1. Would students complete the module more successfully if they had access to online resources?
Which aspects in Unit 5940 are successful and should be retained?	1. What are the current factors which contribute to successful learning in the unit?	2. What kind of challenges do the facilitators anticipate if students study online?
What kind of e-learning strategy could be used and would be feasible for the current or potential audiences?	1. What design is preferred for the online content in unit 5940?	2. What is required for students to study effectively online?	3. What is the best way for students to get help - classroom, discussion board, email?

Meena Questionnaire survey from Students Interview Questions from Course Facilitators

Logistics and Timeline

MEENA

Timeline Schedule for evaluation plan

Dates	Tasks
Oct 04 2010- Oct 23 2010	Decide on evaluation topic and co-ordinate with partner
Oct25 2010 to Nov 15 2010	Identify Data collection methods
Nov 19 2010 to Dec 5 2010	Compile Interview questions for facilitators and questionnaire survey for students
Dec 7 2010 – Dec 18 2010	Review plan and document
Jan 3 2010-Jan 15 2011	Analyse findings and write results
Jan 15 2011 to Jan 24 2011	Documentation, results, recommendations
Jan 25 to Jan 31st	Write up Evaluation report
By 1st Feb 2011	Submit Evaluation report

Budget
Proposed Budget
Evaluator 30 hours @ rate of $30 per hour= $900 So Total Budged for the Evaluation plan= $900

Outline the steps of the evaluation in order of implementation, analysis, and reporting of the evaluation, and include a timeline.

Weeks	Action
11-15 Oct	• Writing up initial plan • Contact staff to be sampled • Compile questionnaires for students, staff and expert • Review appropriate readings for research
18-22 Oct	• Get feedback from Bronwyn and make necessary revisions • Organise dates and times for interviews
26-29 Oct	• Conduct interview with expert • Deliver questionnaire to students and staff)
1-5 Nov	Analyse Results
8-12 Nov	Write up results and draft report
22-26 Nov	Final write up and submit report

Budget

Provide a rough costing of the evaluation process. For example, the evaluator's time and costs, payment of other participants etc.,

Based on a charge out fee of $40 an hour to prepare, deliver and present the results of the evaluation the budget would be as follows:

Preparation and Presentation of Evaluation Plan 15 hours @ $40 = $600
Interviews 2 hours @ $40 = $80
Review of Documentation 1 hour @ $40 = $40
Analysis of evaluation material 5 hours @ $40 = $200
Preparation of Final Evaluation Report 6 hours @ $40 = $240
Printing = $5
Morning Tea for participants $30

Total Budget for preparation ,delivery and presentation of Evaluation Plan $1,195