Planning Evaluation

Evaluation of eLearning for Effective Practice Guidebook

Evaluation is the cornerstone of quality.

Home  |  Exemplars  |  Why evaluate?  |  Evaluation Methods  |  Planning |  Results |  Reporting |  References

Planning Evaluation

To develop the evaluation plan over the next few weeks, a series of activities need to be carried out. Note: It is best to leave the final Introduction until you have done the rest of the plan.

Background and rationale, aim, goals and decisions

The reasons for the evaluation inform the design of the evaluation. The plan must include a clear description about the factors leading up to the evaluation as this sets the scene, and also sufficient information about the area under investigation so that the project team is clear about what is involved. The aim or purpose of the evaluation indicates the intention of the evaluation, and includes the type of evaluation to be carried out. The goals for the evaluation (also called sub-purposes) indicate the focus of the evaluation. The decisions are not always foreseeable but they can be anticipated. Generally, the areas which are most likely to be influenced by the evaluation are mentioned in the decisions. The following example demonstrates how all these aspects might fit together.

Example

• Aim: To conduct a formative evaluation of a multimedia language learning module for delivery to distance students.
• Goals:

• 1. To measure whether the multimedia used in the module is likely to support language learning;
• 2. To collect feedback about the design of the module;
• 3. To establish whether the content is relevant to language learners.

• Decisions:

• Finalise the design of the module, prior to piloting it with further groups students.
• Determine whether further funding is needed to develop similar modules.

Activity

Activity Four - describe the background and rationale, goals, and decisions - on your project wiki using the evaluation plan template (on eMIT in Course Activities). Email the lecturer with the name of your group, and the members so that a Group space can be allocated with the project wiki. Decide on the type of evaluation - formative or summative - based on the Reeves and Hedberg model, or another one you wish to use (please discuss this with the lecturer). Explore the reasons for your evaluation project, goals and the decisions to be made - begin the plan. Add the evaluation plan template to your wiki space in your group. Remember to post your ideas about your evaluation project on your blog. Activity Five- aim and evaluation questions. Prepare the aim and two or three evaluation questions, and sub-questions for your plan to fit the decisions and goals. Add to your project wiki. Use the project forum to ask for feedback from the class.

Activity

Activity Six - evaluation methodology (design). Explore and prepare the evaluation methodology - sampling methods (e.g. questionnaire, interviews, focus, and documentation) and likely methods for the analysis of data to ensure triangulation and bracketing, and examples of the sampling instruments (e.g., questionnaire and interview questions). Information about evaluation methodology can be found further on, and you may also wish to consult a book on Research methods for some additional ideas. Collaborate in your groups to design the evaluation methods. Use the resources to further develop your evaluation plan. Add the methodology section to your project wiki.

Evaluation Methodology

For this project, it is ideal to situate the evaluation within the Eclectic-Mixed Methods-Pragmatic paradigm and use a multiple methods model for the evaluation design. Correspondingly, a diverse mix of practical data collection methods can then be used to investigate the complexity inherent in multimedia learning systems. Also, data collection methods must be selected which ensure validity and reliability of the evaluation.

Importantly, the multiple methods model of evaluation, where a number of measures are used to gather data, is more likely to lead to triangulation and bracketing, and a valid and reliable study.

• Triangulation is where multiple measures are used to estimate the actual value of a variable.
• Bracketing enables "a range of estimates of the value of a variable" (Reeves & Hedberg, 2003, p. 46).

An example to demonstrate how triangulation and bracketing can be applied

An evaluation is carried out to investigate the evaluation question: how do blogs enhance students' learning? Therefore, one variable under investigation is learning.

• For triangulation, three methods are used to collect data:

1. questionnaire - to find out how students used the blogs to record their learning;

2. interviews - to explore the perceptions of students about using blogs for learning; and

3. content analysis - to look for evidence of reflective learning.

• Bracketing occurs when learning is investigated in several different ways, and a range of evidence is obtained. To do this, the results from each of the three methods are examined differently using various analytical techniques. For the questionnaire, descriptive statistical analysis is used to determine the frequencies of participants' responses to each of the questions, and thematic analysis is used to examine their responses to open-ended questions.

Creating a survey or questionnaire

If you are creating a survey or questionnaire - you can use two different tools. Survey Monkey and Google Forms. Survey Monkey allows you to create 10 questions. Google Forms lets you create any number of questions you like. Information about how to use Google Forms can be found at: [1].

Analysis of the data

It is advisable to decide how you will analyse your data when you are writing up the evaluation methodology for your plan. For this project, the analysis needs to be kept relatively simple. However, once you become more experienced in conducting evaluations, you may wish to explore more advanced methods of analysing your data.

Quantitative data

• Descriptive statistics such as frequencies - numbers and/or percentage, averages, median, mode and standard deviation. See: Descriptive statistics.

Qualitative data

There are a number of analytical and interpretation methods you can use and this depends on the type of data you are collecting. In the project for this course, it is best to keep it simple. Therefore, you may like to use content analysis and thematic analysis to investigate themes and patterns in the data.

• Theme - this is the specific factor or feature. Themes can be categorised.
• Pattern - this is the general trend shown by the factor or feature.

Terms

• Theme - people say they like using the Blackboard learning management system.
• Pattern - satisfaction with Blackboard - it can be low or high.
• Content analysis - investigating what is written and how it is written.
• Thematic analysis - looking for specific factors or features (themes) which show up more that once.

In thematic analysis, the content of documents, text, speech or other media are examined to determine the frequency at which particular themes emerge. Factors which are most often mentioned become the themes, and how themes interconnect may be of interest. Sometimes what is not mentioned or is avoided can become an important theme. (Ratcliff- no date available.) For example, students may have a low rate of attendance at classes held on web conference, and a high rate of attendance at face-to-face classes, and no-one mentions they dislike online tutorials, but it appears they avoid going to them.

• Therefore, the theme is attendance, and the pattern is the rate of attendance.

Sometimes content analysis is based on a particular theory, e.g., Diana Laurillard: the Conversational Framework, and determines how the data is analysed. For example:

• The size of the data unit - word, sentence, phrase, paragraph. Once a decision is made, the unit must be used consistently.
• Units of meaning are sorted into categories. Sometimes categories are pre-assigned (if using theory), and/or they can be developed once the data is scanned for themes.

Activities Seven, Eight and Nine

Activity

Activity Seven- Continue developing the plan and request feedback. Prepare other sections of the plan to complete a draft plan – limitations, logistics and time line, budget, references and appendices. Finish writing the Introduction. Add to your project wiki and request feedback on the project forum.

Activity

Activity Eight- Finalise and submit the final plan. Incorporate feedback into the plan Organise collection of data – send out invitations to participate.

Activity

Activity Nine- Reflect on the group evaluation processes you have undertaken to develop your plan. Check the criteria for assessment four in the course handbook. Update your blog according to the criteria. In particular, on your blog - report and reflect on the personal contributions you have made to the group process during the development of the evaluation plan - which sections were you responsible for, what did you contribute etc. Indicate what you learned about working in a group.

Resources

• eLearning Guidelines for New Zealand - useful to help formulate your 'big picture' evaluation questions.
• Evaluation Cookbook to help you decide on some recipes for your project.
• Evaluation Planning section of the companion web site for the book by Reeves, Thomas, C. and John G. Hedberg (2003), Interactive Learning Systems Evaluation, Englewood Cliffs: Educational Technology Publications.
• Descriptive statistics.