OERu project evaluation

From WikiEducator
Jump to: navigation, search


The following page details the proposed role of the OERu evaluation that will be conducted alongside the OERu project planning phase. Comments, suggestions and disagreements are welcome and will be incorporated as far as possible into the evaluation plan. An evaluation brainstorm is underway in the discussion tab. Please feel free to contribute.


Role of the evaluation

A transparent and flexible evaluation framework is an integral component in the development of the OER university project plan as this will ensure that the development of the project is supported by sufficient evidence and input from stakeholders so as to ensure that the resulting plan is effective, actionable and sustainable. The proposed evaluation will be based on a collaborative framework and will be conducted alongside the development of the project plan.

The role of the evaluation will be twofold:

  • Formative, by providing evidence to support decision making and quality assurance.
  • Summative, by providing a retrospective comparison of the success and impact of the project against objectives and goals.

The OERu 2011.11 meeting of founding anchor partners proposed that the project evaluation be guided by the Context, Input, Process, and Product Evaluation (CIPP) model (see Stufflebeam 2003).

Aims

The aim of the project evaluation is to provide an ongoing holistic and collaborative assessment of the context, input, process and impact of the OERu project plan in order to provide evidence for effective decision making and project planning.

The purpose of these evaluation pages are to review and integrate relevant information collected during the recent project planning meeting in Dunedin, information provided by anchor members in the project planning section of the Wiki and the research currently being undertaken by various representatives of the partner universities. The findings of this review will be presented in these pages with the aim of facilitating decision making and providing input into the structure of the OERu project framework.

Evaluation scope

The evaluation will consist of four phases that will be conducted alongside the development of the project plan:

  • An evaluation of the project context, focusing on the goals, priorities, objectives and barriers or challenges anticipated that will impact on the project planning decisions.
  • An evaluation of the available resources or project input, focussing on the steps and resources required to achieve the project goals and objectives.
  • An evaluation of the project process to determine how well the project is being implemented.
  • An evaluation of the project product or output to determine the extent to which the project achieved its aims and made an impact.

Although the evaluation will integrate findings from OERu research studies that are currently under way, the core component of the evaluation will be focused on obtaining the information required to help us do the project.

The data will primarily be collected from elected representatives of each of the OERu anchor partners.

Phase 1 Context evaluation

The first phase of the evaluation will focus on a context and input evaluation. A context evaluation explores the needs, problems, assets and opportunities available to the project while defining goals and actions (Tan, Lee & Hall, 2010). During a context evaluation decision makers are provided the opportunity to consider the needs and issues which arise and prioritise the importance of each component.


Context Evaluation Research Questions:

Overall planning questions:

  1. What are the activities needed in our logic model at this phase of the OERu development?
  2. What policies and processes are required?
  3. What are the goals and key performance indicators for each of the sub-activities to achieve the intended outputs of the relevant activity?
  4. What are the the critical success factors that will contribute to the development of a sustainable OERu project plan?
  5. What are the major issues, barriers and challenges are that are anticipated by the anchor partners and prioritise the importance of these challenges?
  6. Are there any existing programs that can serve as a model for the completed program?

Specific to organisations:

  1. What are the attitudes of people within the partner universities towards the use of OERs?
  2. What capacity building is required within each partner organisation to develop the courses?
  3. What are the reasons that partner organisations have for being involved in the OERu project?
  4. To what extent does the broader university support the use of OERs? University attitudes towards OERs and the OERu and current usage?
  5. To what extent are the organisations using OERs at present and what courses are available?
  6. What gaps are there in terms of policy and protocol that hinder involvement with the OERu?
  7. What are the business models that partner institutions are utilising to support involvement in the OERu?

Specific to course:

  1. What are the anticipated target audiences?
  2. What needs are anticipated from these target audiences?


Outcome of content evaluation:

Matrix by each institution:

  1. Issues / barriers to be resolved by each institution
  2. Policies and procedures identified
  3. Resources available
  4. In depth institutional context

Phase 2: Input evaluation

An input evaluation examines options for achieving goals and provides an assessment of all the resources available to address the identified needs and aims of the project.


Input Evaluation Objectives:

Courses:

  1. Determine which courses each partner will be willing to contribute, the course specifications and the state of readiness of each (underway in the project planning tab).
  2. Determine what criteria has been used to select these specific courses by each instution.
  3. Determine what stage of development each of the OER courses are in that will be contributed to the pilot.(Timeframes)
  4. Provide course specifications
  5. What percentage of a bachelors degree would this course constitute (credit points)?
  6. What processes are in place to ensure the courses will be institutionalised?
  7. What processes will be put in place to mainstream OERu learning in institution?
  8. Is the course ready to be loaded on Moodle in wiki educator? (ICT infrastructure readiness)
  9. Are there any issues or barriers that will stop people from putting the course up?
  10. To what extent will students have access to content?
  11. To what extent will students have access to online support?

Resources:

  1. Determine what OER assets are already available within each partner organisation.
  2. Determine what capacity building will be required internally in order to meet the KPI's.
  3. Determine what the resources are that each partner has lined up to support the course including manpower and budget resources.
  4. What are the budgets? Is any funding required? How will institutions ensure that OERu offerings are financially viable?

Students & volunteers:

  1. Determine ideas for attracting students and advertising of the completed courses.
  2. What types of support will students be give, if any?
  3. How many students does each organisation anticipate being bale to support?
  4. Obtain suggestions on where to get volunteers from?
  5. How many volunteers will there be / are already available?
  6. To what extent do we need discipline expertise from volunteer mentors?
  7. Plan for managing volunteers or process of course?
  8. How will the course we run?
  9. Who will manage to interaction?
  10. Training for volunteers mentors and assessors

Accreditation:

  1. How will the courses be assessed? Who will be marking? - Need academic experience
  2. What is the cost of assessment anticipated by each institution?
  3. What support is required for assessment?
  4. How are partner organisations planning to approach requests for recognition of accreditation given by other institutions in the OERten network?
  5. In what programs could these courses be guaranteed credit?
  6. How and what, will prior learning from other institutions will be considered?

Participation in pilot

Process

A number of these questions have already been asked or are included in components of OERu studies that are already underway. We would like to use this forum as a vehicle for integrating those findings and making them available to the OERu anchor partners to assist in project planning.

Over the next few weeks we will be approaching all stakeholders who are currently undertaking research and requesting their support. We will also be developing a series of questions to answer the research questions stated above. We will be launching these either in the project planning pages under each of the relevant sections of the logic model discussions or in separate pages within the wiki. Please feel free to contribute to the research questions or objectives on this page.

Watch this space, more information to follow

Brainstorm list of ideas

Please contribute any ideas here:

Ideas for evaluation research questions

  • Add suggestions here
  • What is the response of learners or potential learners to the course design? This includes:
OER model of learning
course instructions
navigation through the materials
activities
interactions with other learners
assessments
workload
learning experience.

References:


Retrieved from "https://wikieducator.org/index.php?title=OERu/Planning/OERu_project_evaluation&oldid=906269"