Evaluation of eLearning for Best Practice

kitten cuteness

Image courtesy of bronwynannh

If you have a new teaching and learning innovation you want to evaluate this could be the course for you. The next session starts on 26 July 2010. The current course is now called: Evaluation of eLearning for Effective Practice and has moved. The course is facilitated by Otago Polytechnic and administered by Manukau Institute of Technology.

Every fortnight the class will participate in real-time web conferencing sessions about specific topics as listed in the schedule. There will also be invited speakers from the 2009 class who will present their evaluation projects. Here is the meeting link to be used for the duration of the course.

What will you be doing?

By planning and conducting a real project you will explore the what, why, where and how of the many aspects of evaluation. You will find out that it is not as simple as just putting out a survey, as an evaluation must be planned carefully if you are to obtain valid and reliable information. Evaluation findings can be very useful if you are trying to make quality improvements in your practice and/or in your organisation. There is nothing like hard data and a well written report for influencing management decisions.

By necessity, so you can experience evaluation in action and in its many forms, this is a very practical course and is based on up-to-date theoretical models. You will find out about both quantitative and qualitative methods for evaluating so you can choose the one which is right for your context; complex situations often require multiple approaches. The evaluation process and all aspects of the project make up the assessment for the course so you are assessed as you go and while you learn the ropes. What could be fairer than that? There is more detail about this further on.

Newsfeed
Evaluation of eLearning for Best Practice course blog Failed to load RSS feed from http://bestpracticeevaluation.blogspot.com/feeds/posts/default|Date|short|max=1: There was a problem during the HTTP request: 400 Bad Request

Formal enrolment

Please contact the administrator at Manukau Institute of Technology (MIT)- details about the course and the programme are on the website.

Informal enrolment

You are welcome to study the course materials and enrol when you are ready to have your assessments marked. Note: it is best if the evaluation plan due week 9 is formally assessed so you can carry on in confidence with the rest of your evaluation project.

Getting started

• Please register with Wikieducator and then click the "discussion tab" at the top of this page and introduce yourself to the others in the course.
• Feel free to add items to the weekly schedule which you believe will assist the class or should be covered and/or links to material in the Resources area - remember to include your wiki signature, eg --bron 03:12, 24 February 2008 (UTC) (click the signature button in the editing menu when logged in).

• Use the quick links in the Contents box or scroll down to the weekly schedule and content.
• Watch the newsfeed from the Course blog.
• Keep an eye on the Calendar on the Course blog.
• And now for the good news - you will need to spend approximately six to seven hours per week studying.

Assessment and Marking Schedule and Resources

• Assessment and Marking Schedule
• Resources

This course is part of the following programme

• Graduate Certificate in Applied elearning.

Learning objectives

By the end of this course you will be able to:

• Identify quality issues for eLearning and evaluation processes;
• Describe the main inquiry paradigms which currently influence evaluation methods and their association with a range of evaluation models;
• Plan and carry out an evaluation of an eLearning course or product.

Why do evaluation?

Evaluation during development of resources is not done particularly well in the tertiary sector, partly because organisations have got by for so long without it. The evaluation of how effective a course innovation is for learning is generally done at the end of a course, by which time, students may have been leaving in droves due to a poor quality experience. To only carry out evaluations only at the end of a course, to satisfy auditing and accreditation requirements for quality, is a dangerous game. Evaluation needs to be done even before an innovation is introduced to find out if the resource is needed by the learners (needs assessment, needs analysis or front end analysis). Evaluation during the development and implementation (pilot) phases (formative evaluation) is also very important. The diagram of the ADDIE model (analysis, design, development, implementation) below illustrates where evaluation fits into the process of educational (instructional) design.

Communication in the course

The primary means of communication for this course will be via an email discussion forum to which you will be invited. It would also be very helpful for the class if you all gave each other feedback on the evaluation processes you put on the blogs (weblogs) each of you will be keeping for one of your assignments; the course blog will be used for weekly announcements and summaries.

• You are encouraged to share as much information as possible about your evaluation processes via your weblogs - see Assessment and Marking Schedule.

Adding material to the wikipage

• If you add resources or information to the wiki pages for the course, be sure and add your signature so we can all keep track of what is new e.g. --bron 22:58, 23 February 2008 (UTC).

Weekly schedule

This is a guide for your weekly activities which all count towards your evaluation project. Reminder: you will need to spend approximately six to seven hours per week on this course. Although you are given a clear structure to follow, you can also explore areas specific to your own discipline. If you need assistance finding material relevant to your area of practice, make sure you ask the class and the lecturer, and if you come across something you think may help someone else, please pass it on.

Week one: Introductions - 26 July to 1 August

FLNW 2006 Dunedin 62 by teachandlearn

Use this week to get yourself set up, familiarize yourself with the course objectives and Assessment and Marking Schedule, course schedule, introduce yourself to the rest of the class and the lecturer.

Things to do:

• Introduce yourself on the email forum.
• Set up an account on WikiEducator.
• Join a computer conference to meet each other, hear about the course from the lecturer and ask questions. There are two opportunities - choose one or you are very welcome to attend both sessions. No password is needed for the Elluminate online meetings - just put in your name when the login comes up.

• You will need a headset with microphone (unless it is built in to your computer) if you wish to speak though there is a chat facility.

You will also need to:

• Establish a blog for recording your progress - details in Assessment and Marking Schedule. Look for things to do to set up your blog.
• Post something about yourself on your blog (or if you need more time to get a blog started - just introduce yourself on the email forum - you can copy over to your blog later):
  • who you are,
  • your work,
  • your interest in evaluation and what you know already,
  • what you hope to get out of this course,
  • ideas you have for an evaluation project if any.

• Let the class know the link to your blog via the email forum.

Week two: Quality and evaluation

quality icecream mural by KB35

This week you will establish a definition of evaluation for eLearning which fits your context, and begin to discuss the importance of quality.

Things to do:

• Watch the 20 minute slide presentation - Why is evaluation so important?
• Instructions for viewing the presentation can be found on this handout.
• Read the section about the importance of evaluation in the eLearning Guidebook - this includes some information about quality.
• Post answers to the following questions to your blog and send a short email out to the class about your main points.
  • Why is evaluation important to you and how do you define it?
  • What sort of evaluations mentioned on the presentation are familiar to you already and why?
  • Why is quality important in eLearning?

Week three: eLearning guidelines for quality

This week we are revving up the action and will look at some eLearning Guidelines which have been developed for New Zealand. These were used by 21 tertiary organisations as part of a Ministry of Education funded and sector wide project on implementing e-learning guidelines. This national project provided a mechanism for developing, introducing, trialling, evaluating, and sharing examples of e-Learning Guidelines.

Things to do:

• Explore eLearning Guidelines for New Zealand: http://elg.massey.ac.nz/
• Select two elearning guidelines relevant to your area of practice - these will be used in your evaluation project.
• Check out this scenario of how the guidelines might be used. There are also other scenarios for you to explore - now or later.
• Identify two issues related to quality in eLearning - these will provide part of the rationale for your evaluation project.
• Read the article of an evaluation study conducted at the University of Michigan (U-M) School of Dentistry by Brittain, Glowacki, Van Ittersum and Johnson, 2006.
• Post the two eLearning guidelines you have chosen (you may prefer to write your own) on your weblog with an explanation of your reasons for choosing them.
  • Include a description of how the guidelines might address the issues you have identified.

Week four: Evaluation paradigms and models

Solar powered car borman818

Over this week you will explore the main inquiry paradigms which currently influence how evaluation is conducted. You will also find out about the different evaluation models and how they fit under these paradigms. The word paradigm conjures up visions of stiff and starchy academia doesn't it? Well think of it as a style or belief. You will all have heard the words paradigm-shift when people talk about how eLearning is changing how we teach; meaning the change from teacher-centred to student-centred learning. Well clothing and learning are not the only areas subject to change - evaluation and research are also at the mercy of what is fashionable.

If you are wondering about the difference between an evaluation paradigm and model, think about this analogy about hybrid cars. Hybrid car paradigm - a belief in cars which address some of the concerns about peak oil. Hybrid car model - a specific car which uses biofuel or petrol and electricity or solar power.

Things to do:

• Attend or listen to the recording of the optional fortnightly class meeting held on Thursday 19 March 2009 - 7 to 8pm (NZST, +12hrs GMT)- recording of web conference. This was a discussion about quality and eLearning Guidelines.
• Read the article on paradigms by Professor Tom Reeves.
• Look at some information about different evaluation models.
• Read the article in which two different evaluation models are compared - Experimental and multiple methods evaluation models - by Bronwyn Hegarty, 2003.

Note: More detail on this topic is available in chapter two of the book by Reeves, Thomas, C. and John G. Hedberg (2003), Interactive Learning Systems Evaluation, Englewood Cliffs: Educational Technology Publications. Available from the Manukau Institute of Technology library.

Weeks five and six: Evaluation methods

Diagram from a presentation by Professor Tom Reeves

For this topic you will need to spend two weeks - it is very important you explore the options carefully, and choose a method appropriate for your evaluation project. There are lots of different functions/types of evaluation i.e. methods you could use in your project. The method you choose will depend on the purpose of the evaluation. You have already heard the words formative and summative, needs analysis etc. Well what does it all mean? Over the next two weeks, you will be investigating terms such as usability, effectiveness evaluation, impact and maintenance evaluation and discovering whether they fit under the formative or the summative umbrella.

Things to do:

• Read the short article evaluation methods in the eLearning Guidebook.
• View the presentation by Professor Tom Reeves - it depicts "six Facets of Instructional Product Evaluation. Review. Effectiveness. Maintenance. Formative. Needs Assessment. Impact. Development Activities", (Google Books, 2008) and is based on the book by Reeves, Thomas, C. and John G. Hedberg (2003), Interactive Learning Systems Evaluation, Englewood Cliffs: Educational Technology Publications - available in the Manukau Institute of Technology library.
• Find an article or two in the literature which fits with the type of evaluation you are interested in for your project.
• Check the Resources page for some examples.

• Post responses to the following on your weblog:
  • Outline some information about the type of evaluation you believe will best suit your project with your reasoning.
  • Summarise the article(s) you have found (300 words) - full reference, evaluation design (paradigm, model, questions, methods), findings and how it aligns with the evaluation project you would like to conduct.

Weeks seven and eight: Negotiate and write an evaluation plan

Conversation by Clairity

Over the next two weeks - remember these weeks are after the two week break over Easter (6 to 17 April) - you will need to plan your evaluation and prepare a formal plan. By now you should have a pretty good idea of what you intend to evaluate. Remember to look at Assessment and Marking Schedule for the pointers on this part of the assessment.

Things to do:

• View the recording of the web conference with Helga Wientjes - speaking about a formative evaluation project conducted to measure effectiveness of a web conferencing training module. You can read about her evaluation plan and report on the blog she kept for the 2008 course.
• Decide on two eLearning guidelines against which you will evaluate.
• Establish the paradigm and model for your evaluation.
• Look at this example of an evaluation plan - Usability evaluation
• Look at the evaluation tools on the companion web site for the book by Reeves, Thomas, C. and John G. Hedberg (2003), Interactive Learning Systems Evaluation, Englewood Cliffs: Educational Technology Publications.
  • You can download a Microsoft Word Rich Text Format (rtf) version of an Evaluation Plan Template.
  • Remember this can be modified to suit your situation.
• Explore the Evaluation Cookbook to decide on some sampling methods for your project.
• Prepare some draft ideas and post them on your blog.
• Contact your lecturer to negotiate your plan.
• Write up your plan more fully using the suggested section listed in the Assessment and Marking Schedule

Note: a lot of the information in your plan can also be used in your final report about the project.

Week nine: Prepare and present an evaluation plan

flickr for social libraries presentation by libraryman

This week you will spend time preparing a digital presentation of your plan. The purpose of this exercise is to help you gather feedback on your ideas.

Things to do:

• Thursday 7 May 7-8pm NZST - attend a guest speaker presentation with Gordon Robinson via the Elluminate Meeting link. Gordon will speak about the evaluation plan he used for the 2008 course when he carried out a maintenance evaluation. The title of his talk is: Evaluation plan for the Hazardous Substances Awareness course.
• Choose a medium for your presentation.
• Check the suggestions for the presentation in Assessment and Marking Schedule.
• Prepare your plan and present it according to the Assessment and Marking Schedule.
• Provide feedback to at least two other plans.
• Incorporate feedback and submit the written evaluation plan (Part two A: Negotiate and write an evaluation plan) in the Digital Dropbox in the course Learning Management System or on your weblog or via email to the lecturer.
• If you prefer you can put your plan in Google docs and post a link to it on your blog - remember to publish it and give the lecturer access.

Weeks ten and eleven: Conducting your evaluation

During these weeks you will be carrying out the evaluation with your target group.

Things to do:

• Recruit participants - this may have been done previously.
• Collect data using the sampling tools you have stipulated in your plan.

Weeks twelve and thirteen: Analysis of data and results

Fitness academic graph by Paul Danger-Williams

During the next two weeks you will need to look at the data you collected and collate it. Keep the analysis simple and use methods with which you are familiar. For example, calculating percentages (frequencies) can transform raw data into manageable chunks.

Things to do:

• View the presentation called Preparing evaluation results - questionnaire responses.

• To view the presentation in full screen click the icon at the bottom right of the slide.
• You can also look at the spreadsheet Descriptive statistics % Frequencies which accompanies the presentation to see how some samples of data have been analysed.

• Look at Helga Wientjes' draft evaluation report. Unfortunately some of the Figures are not appearing - unfortunately Helga is away so permission cannot be obtained to access the other Figures but there are some there to look at.
• Investigate some other examples of results and reports in the Resources area to see what others have done.
• Decide how you will present your data - tables, graphs (what type of graph).

Post your initial ideas to your blog and ask for help if you need enlightenment.

• Write descriptively about your results - again check out some examples.
• Submit a draft of your results to the lecturer via Digital Dropbox, your blog or via email.
• Post a summary of your results to your blog.
• Provide feedback to at least two other people on their blogs.

Weeks fourteen, fifteen and sixteen: Write your evaluation report

This should be the easy bit. This needs to be a professional looking report with referencing to support your writing. You can use a lot of the material you wrote in the evaluation plan, for example, introduction, background and rationale. Follow the Assessment and Marking Schedule for Part Four: Evaluation report, to make sure you have included the headings and material required.

Things to do:

• Take a look at some example reports (scroll down to documents) at: Online information literacy project. You will note some reports are brief e.g Prototype Evaluation Report, and others such as the final report for the project are more detailed. You will need to strike a balance.
• Mock up an outline for your report using the evaluation report checklist or create your own.

• Make sure you check the requirements in the Assessment criteria and marking schedule.

• Submit your final report via Google documents or via email to the lecturer.

• Please note: If you use Google docs please make sure you give the lecturer access.

• It would be great to see your reports, or a link to them, posted on your blogs - remember to remove organisational names as appropriate.

HELP

• Helpdesk for technical computing assistance for Elluminate (computer conferencing) is available from Otago Polytechnic by phone and email.
• Phone: 0800 765 948
• Email: itssATtekotagoDOTacDOTnz

Ensure you tell them your name, my name, name of the course, and a brief description of the problem.