- 1 Planning Evaluation
- 1.1 Background and rationale, aim, goals and decisions
- 1.2 Evaluation Methodology
- 1.3 Creating a survey or questionnaire
- 1.4 Analysis of the data
- 1.5 Activities Seven, Eight and Nine
- 2 Resources
To develop the evaluation plan over the next few weeks, a series of activities need to be carried out. Note: It is best to leave the final Introduction until you have done the rest of the plan.
Background and rationale, aim, goals and decisions
The reasons for the evaluation inform the design of the evaluation. The plan must include a clear description about the factors leading up to the evaluation as this sets the scene, and also sufficient information about the area under investigation so that the project team is clear about what is involved. The aim or purpose of the evaluation indicates the intention of the evaluation, and includes the type of evaluation to be carried out. The goals for the evaluation (also called sub-purposes) indicate the focus of the evaluation. The decisions are not always foreseeable but they can be anticipated. Generally, the areas which are most likely to be influenced by the evaluation are mentioned in the decisions. The following example demonstrates how all these aspects might fit together.
- Aim: To conduct a formative evaluation of a multimedia language learning module for delivery to distance students.
- 1. To measure whether the multimedia used in the module is likely to support language learning;
- 2. To collect feedback about the design of the module;
- 3. To establish whether the content is relevant to language learners.
- Finalise the design of the module, prior to piloting it with further groups students.
- Determine whether further funding is needed to develop similar modules.
For this project, it is ideal to situate the evaluation within the Eclectic-Mixed Methods-Pragmatic paradigm and use a multiple methods model for the evaluation design. Correspondingly, a diverse mix of practical data collection methods can then be used to investigate the complexity inherent in multimedia learning systems. Also, data collection methods must be selected which ensure validity and reliability of the evaluation.
Importantly, the multiple methods model of evaluation, where a number of measures are used to gather data, is more likely to lead to triangulation and bracketing, and a valid and reliable study.
- Triangulation is where multiple measures are used to estimate the actual value of a variable.
- Bracketing enables "a range of estimates of the value of a variable" (Reeves & Hedberg, 2003, p. 46).
An example to demonstrate how triangulation and bracketing can be applied
An evaluation is carried out to investigate the evaluation question: how do blogs enhance students' learning? Therefore, one variable under investigation is learning.
- For triangulation, three methods are used to collect data:
- 1. questionnaire - to find out how students used the blogs to record their learning;
- 2. interviews - to explore the perceptions of students about using blogs for learning; and
- 3. content analysis - to look for evidence of reflective learning.
- Bracketing occurs when learning is investigated in several different ways, and a range of evidence is obtained. To do this, the results from each of the three methods are examined differently using various analytical techniques. For the questionnaire, descriptive statistical analysis is used to determine the frequencies of participants' responses to each of the questions, and thematic analysis is used to examine their responses to open-ended questions.
Creating a survey or questionnaire
If you are creating a survey or questionnaire - you can use two different tools. Survey Monkey and Google Forms. Survey Monkey allows you to create 10 questions. Google Forms lets you create any number of questions you like. Information about how to use Google Forms can be found at: .
Analysis of the data
It is advisable to decide how you will analyse your data when you are writing up the evaluation methodology for your plan. For this project, the analysis needs to be kept relatively simple. However, once you become more experienced in conducting evaluations, you may wish to explore more advanced methods of analysing your data.
- Descriptive statistics such as frequencies - numbers and/or percentage, averages, median, mode and standard deviation. See: Descriptive statistics.
There are a number of analytical and interpretation methods you can use and this depends on the type of data you are collecting. In the project for this course, it is best to keep it simple. Therefore, you may like to use content analysis and thematic analysis to investigate themes and patterns in the data.
- Theme - this is the specific factor or feature. Themes can be categorised.
- Pattern - this is the general trend shown by the factor or feature.
- Theme - people say they like using the Blackboard learning management system.
- Pattern - satisfaction with Blackboard - it can be low or high.
- Content analysis - investigating what is written and how it is written.
- Thematic analysis - looking for specific factors or features (themes) which show up more that once.
In thematic analysis, the content of documents, text, speech or other media are examined to determine the frequency at which particular themes emerge. Factors which are most often mentioned become the themes, and how themes interconnect may be of interest. Sometimes what is not mentioned or is avoided can become an important theme. (Ratcliff- no date available.) For example, students may have a low rate of attendance at classes held on web conference, and a high rate of attendance at face-to-face classes, and no-one mentions they dislike online tutorials, but it appears they avoid going to them.
- Therefore, the theme is attendance, and the pattern is the rate of attendance.
Sometimes content analysis is based on a particular theory, e.g., Diana Laurillard: the Conversational Framework, and determines how the data is analysed. For example:
- The size of the data unit - word, sentence, phrase, paragraph. Once a decision is made, the unit must be used consistently.
- Units of meaning are sorted into categories. Sometimes categories are pre-assigned (if using theory), and/or they can be developed once the data is scanned for themes.
Activities Seven, Eight and Nine
- eLearning Guidelines for New Zealand - useful to help formulate your 'big picture' evaluation questions.
- Evaluation Cookbook to help you decide on some recipes for your project.
- Evaluation Planning section of the companion web site for the book by Reeves, Thomas, C. and John G. Hedberg (2003), Interactive Learning Systems Evaluation, Englewood Cliffs: Educational Technology Publications.
- Descriptive statistics.