OERu process evaluation

OERu

Towards more affordable education for all students worldwide

Home  |  Quicklinks  |  About  |  Logic model  |  People  |  Planning  |  Communications  |  Meetings  |  Resources  |  Media  |  FAQs

The process evaluation aims to investigate the quality of the implementation of the OERu project. It's purpose is to promote process efficiency for the delivery of OERu products. The development and delivery of the OERu 1^st year of study during 2017 & 2018 will be a major focus of the process evaluation.

Action items

1. Initiate planning for the process evaluation - Done
2. Select and commission evaluator - Done
3. Prepare draft process evaluation plan for OERu 2017-10 partner's meeting - Done

Supporting documents

Context evaluation

Researcher: Dr Angela Murphy, Australian Digital Futures Institute, University of Southern Queensland

1. Summary wiki report
2. Summary of results (pdf 1.2MB)
3. Summary of results on Slideshare

Input evaluation

Researcher: Dr Xiang Ren, Australian Digital Futures Institute, University of Southern Queensland

1. Initial report of the OERu input evaluation survey (Prepared for the OERu 2015.10 partners meeting.)
2. Input evaluation final report, December 2015
3. Input evaluation survey results

Potential survey items and benchmark data

1. New OERu participant survey (We have the advantage of some comparative data from prototype courses - each course invites learners to complete the new OERu participant survey.)
2. Example course survey OCL4Ed 2012.06

Project plan

The process evaluation is the third stage of Stufflebeam’s CIPP evaluation model which is intended to monitor the project implementation process to determine whether the OERu project is being implemented appropriately and effectively.

Aims

Specific aims of this stage of the evaluation are as follows:

1. Documentation of the project implementation process (as reflected in the strategic plan and meeting decisions.)
2. Obtain feedback from the three groups of stakeholders regarding progress towards achieving the OERu Strategic Plan 2015-2017.
3. Identify which components / activities undertaken during the project implementation process has been particularly successful.
4. Identify challenges, barriers, and risks to the successful implementation of the OERu project.
5. Identify whether there are any support or resources gaps for all three stakeholder groups which would impact the extent to which participants are able to underte their roles within the project implementation process.
6. Identify whether there are any gaps in the implementation of certain activities where revisions or adjustments to the OERu project plan are required.
7. Obtain impartial evidence of the extent to which the project implementation is progressing in comparison to objectives laid out in the strategic plan and meeting decisions (records / analytics / progress reports etc).
8. Generate authentic data to inform the OERu open business model.

Stakeholder groups

1. OERu learners (n=2000-6000)
2. Course developers (including learning design consultants, subject matter experts, n=+-32)
3. OERu partners

Method

1. Anonymous online survey with OERu learners to identify student experiences and outcomes from participating in OERu courses for the 1st year of study (largley quantiative with some open questions for insight)
2. Reflective anonymous online survey with course developers and OERu partners to identify progress towards KPI's and satisfaction with activities including open questions and opportunties for reflection on challenges and barriers experienced and means to improve processes (written self-reflection?)
3. Open discussion with course developers and / or other stakeholders to discuss issues and challenges and gather insights.
4. Brief qualitative interviews with the following stakeholders with the aim of identifying key successes, issues, barriers and challenges:
   • 2 learning design consultants who assembled OERu courses for the 1st year of study
   • 2 subject matter experts
   • 2 partner organisations who assembled courses for the OERu 1st year of study
   • 2 senior leaders from the partners who have exit qualifications on the table.

Students who have registered for the OERu courses using email addresses will be contacted to partcipate using email, which will enable targeted reminders. For students who did not register there will be links embedded into the learner survey instruments within the course materials.

Schedule

1. Draft plan published - 30 September 2016
2. Brainstorm survey items and discussion questions for semi-structured interviews at partners meeting - 13 October 2017
3. Finalise process evaluation plan - December 2017
4. Prepare surveys - January/ February 2018
5. Data collection stages - March - June 2018
6. Analyse data - July - September 2018
7. Final report - 31 December 2018

Active projects

1. MVP learning analytics project