WikiEdProfessional eLearning Guidebook/Evaluating the impacts of eLearning/Evaluation methods

There are a number of ways you can gather information during an evaluation project. There are four main methods listed here:

1. Needs analysis (also known as Front-end analysis or Needs assessment) - readiness or need for an intervention.
2. Formative evaluation - feedback during development or piloting.
3. Summative evaluation - measures effectiveness and impact.
4. Monitoring or integrative evaluation - how well an initiative has been integrated.

You should aim to gather data from all stakeholders (i.e., students and staff) regularly using a set of evaluation instruments within a consistent evaluation framework which should include front-end analysis, formative, summative and integrative evaluation.You should also aim to collect a variety of data using a range of data gathering instruments. However, you would want to keep the data gathering process as simple and as less intrusive as possible. There are excellent examples of some of these types of evaluation (front-end or needs analysis, formative) which were done as part of a project (OIL) to develop online information literacy modules for the New Zealand tertiary sector. Several reports are available in the research and evaluation section.

Needs analysis

This comprises a set of ways by which you would plan to ascertain the readiness of students and staff and their preferences in relation to teaching and learning online. Carrying out such surveys periodically and especially prior to the full roll-out of e-learning will enable your organization to get a better handle on how to align its services to meet the needs of prospective users. The information gathered will help to inform the organization on the nature of its user needs, their perceptions and expectations, and any gaps in the provision of existing support.

Formative evaluation

This method involves gathering feedback from users and other relevant groups during the development and implementation process. Its purpose would be to identify problems so that improvements and adjustments can be made during the implementation stages of e-learning in your organization. You may wish to plan to carry out formative evaluations routinely and regularly. It would be best that these evaluations use a consistent set of tools comprising surveys, and focus group interviews with users.

Summative evaluation

This approach will enable you to ascertain the full impacts and outcomes of e-learning on teaching and learning at your organization. You would usually carry this out upon the completion of an e-learning program, even though there is not likely to be a crisp dividing line between formative and summative evaluation phases. As part of this process, your aim is to periodically assess the sum impacts of e-learning on teaching and learning activities in your organization. Data gathered should reveal how e-learning is responding to challenges facing teaching and learning in your organization, and the extent to which you are achieving benchmarks and milestones which you have set.

Monitoring or integrative evaluation

There comes a time when you will want to ascertain the extent to which the use of e-learning or online learning is integrated into regular teaching and learning activities at your organization. Data gathered as part of this process will reveal the extent to which, and how teaching and learning activities in the organization have been impacted with e-learning. This kind of evaluation includes three different types of evaluation:

• Effectiveness - whether the eLearning is working as expected;
• Impact - how the product has influenced the application of learning, and;
• Maintenance - how feasible, up-to-date or cost-effective the elearning when reviewed (Reeves & Hedberg, 2003).

References