Processes of Evaluation
The processes of evaluation can run smoothly and with precision when you plan carefully and follow a formal structure. Otherwise you can find yourself sitting in a car enthusiast's garage surrounded by engine parts. The engine being the evaluation project. The processes you will explore are: planning the evaluation, collecting and analysing data, writing up your findings, and preparing a formal evaluation report.
Contents
Evaluation planning
To develop the evaluation plan over the next few weeks, a series of activities will be carried out, starting with this one.
Activity Four - describe the background and rationale, goals, and decisions - on your project wiki using the evaluation plan template (on eMIT in Course Activities).
|
Resources
- Revisit the eLearning Guidelines for New Zealand: http://elg.massey.ac.nz/ to help formulate your 'big picture' questions.
- Use the Evaluation Cookbook to decide on some recipes for your project. *Look at the Evaluation Planning section of the companion web site for the book by Reeves, Thomas, C. and John G. Hedberg (2003), Interactive Learning Systems Evaluation, Englewood Cliffs: Educational Technology Publications.
|
Work in progress, expect frequent changes. Help and feedback is welcome. See discussion page. |
|
Tackling the plan
Data collection matrix
| Questions | Data collection method 1 | Data collection method 2 | Data collection method 3 |
Collecting and analysing the data
For this project, the analysis will be kept relatively simple. However, once you become more experienced in conducting evaluations, you may wish to explore more advanced methods of analysing your data. At that stage it is advisable to decide how you will analyse your data when you are writing up the evaluation methodology for your plan.
Analysis
Quantitative data
- Descriptive statistics such as frequencies - numbers and/or percentage, averages, median, mode and standard deviation. See: Descriptive statistics.
Qualitative data
There are a number of methods you can use and this depends on the type of data you are collecting. In the project for this course, it is best to keep it simple. Therefore, you may like to use thematic or content analysis to investigate themes and patterns.
- Theme - this is the specific factor or feature.
- Pattern - this is the general trend shown by the factor or feature.
Examples
Theme - people say they like using the LMS.
Pattern - % satisfaction with LMS - can be low or high.
- Thematic analysis - looking for specific factors or features which show up more that once.
- Content analysis - investigating what is written and how it is written.
Documents, text, or speech are examined to see if particular themes emerge. You may look for items which are most often mentioned, and also how themes interconnect. Sometimes what is not mentioned or is avoided can become an important theme. (Ratcliff- no date available.) For example, students may have a low rate of attendance at classes held on web conference, and a high rate of attendance at face-to-face classes, and no-one mentions they dislike online tutorials, but it appears they avoid going to them. Therefore, the theme is attendance, and the pattern is the rate of attendance.
Sometimes content analysis is based on a particular theory, e.g., Diana Laurillard: The Conversational Framework. Also rules determine how the data is analysed. For example:
- The size of the data unit - word, sentence, phrase, paragraph. Once a decision is made, the unit must be used consistently.
- Units of meaning are sorted into categories. Sometimes categories are pre-assigned (if using theory), and/or they can be developed once the data is scanned for themes.
Writing up the findings
This is the fun part. Once you get responses to the surveys and interviews, think carefully about how you will present the data. There are lots of ideas in the exemplars. Tables are great for most data, and you may decide that a graph shows your findings better. It is best to have either a table of graph in the Results section, but any tables you create can go in the Appendix if they are not used in the Results section. I find bar graphs are generally the most helpful, just make sure you keep them simple. It is better to have several small and easy to read graphs than one big, messy one with everything on it.
Calculations
If you have a small number of responses to a survey, it is best to present the data as number frequencies rather than % frequencies. This is called descriptive analysis and is generally done for the responses to Likert type scales or other closed questions in a survey. My rule of thumb is for up to 9 responses, use numbers, and for 10 or more responses calculate percentages. If possible present the data as a mix of tables and graphs and show one set as tables and another as graphs for variety. So it is up to you to decide which data is best as a graph. Include all the tabulated data in the Appendix if it is presented as a graph in the Results.
APA rules for Tables and Figures
When you prepare a table, the caption goes at the top. For a graph or Figure as it is written in a report, the caption goes at the bottom. There are some examples to show you how to present tables and graphs. They don't have to be exactly like this so don't be afraid to be creative. Click on the images to enlarge them so you can see them more clearly.
Summarising survey comments
Data from open questions or comments in a survey, needs to be collated as a summary of responses. This means you summarise the common responses descriptively as themes, not word for word. Responses which are different are kept separate and summarised. For example, three participants may have responded to a question about the design of the online interface used in a course, and made the following comments.
- Participant 1: I really like using the online links as they make it easy to find resources.
- Participant 2: I do not get lost finding the materials I need as the icons are easy to follow.
- Participant 3: Learning using the online materials is very enjoyable because of the colourful and intuitive pathways in the course.
The main themes from these three comments can be summarised as:
- students found navigation in the online course easy; and
- students were satisfied using the online system.
Two of the comments also indicate that students liked learning online - see if you can spot which ones.
One participant may have said the following:
- Participant 4: finding the online materials was difficult because I am not confident using computers. I find it tricky finding my way around, the computer keeps crashing, and I forget how to find my way back to the modules.
This comment could be summarised as: One participant had difficulty navigating the online materials as he was not a confident computer user.
Preparing an evaluation report
There is anevaluation report template
available. Copy and paste the template into your document, and use the headings as a guide for writing the report. As long as all the information is included, the organisation of the report can vary depending on your context and creativity, and organisational requirements. The main thing is to make it look professional. Include a title page, table of contents, List of Tables and List of Figures – the headings in the template have styles included to get you started with the table of contents. Please remove any identifying information before submitting the report.
Submit the report by providing a link on your blog to the document. If submitting a draft, and wish to obtain feedback from the lecturer before submitting to the class, provide access only to the lecturer at first - if using Google docs you will need to give the lecturer editing rights to the document. Send an email to notify the lecturer that you are ready for feedback on your draft report.
Tips
- Use a web-based platform for your document and ensure identifying information is removed.
- If you need to restrict access to the class, choose an appropriate platform - see below for suggestions.
- Web-based platforms you can use: Google documents (publish as web page), Scribd, or Dropbox (use public feature - or you can restrict access by only including members of the class).
Blog post
One of the requirements in the marking criteria for the evaluation report assessment is that you prepare a blog post. This is a summary of the process of conducting the evaluation, and an overview of the findings from analysis of the data obtained - in other words an executive summary. This link to the summary should be done prior to submitting the final report, or if you can just link to the draft report with a brief explanation on your blog. -->
