OERu/Planning/OERu input evaluation/Initial report of OERu input evaluation

Download OERu Input evaluation initial report (pdf version).

Introduction

This survey, which took place in August 2015, constitutes the Input phase of the Context, Input, Process and Product Evaluation (CIPP) model used to inform the implementation of the OERu. The purpose of the Input Evaluation is to evaluate the design options for achieving the OERu goals and to inform the design decision making through reliable and systematic evidence. A total of 52 valid completed responses were received from a representative demographic. The responses came from 32 organizations in the OERu network across the globe. 34 responses (65%) were from the OERu teaching partners, and 10 (19%) from non-teaching partners. We received 19 (37%) organisational responses submitted by designated respondents. Moreover, 65% of the respondents are members of active OERu working groups; 75% rated their knowledge of the OERu concept as above average/excellent; and 55% are at senior and middle management levels in their institutions. All these factors tend to assure the quality and value of survey data.

The initial report is based on the quantitative questions of the survey only. A comprehensive evaluation report will be provided in December 2015. This report summarises the top-ranked items, the big differences/divergence among respondents, and the key design issues raised. In addition to this summary, an online version of the survey results is accessible at:

https://usqadfi.asia.qualtrics.com/CP/Report.php?RP=RP_2gZSaQVfomq2C1L

OERU programme and curriculum

The majority of the respondents believes the OERu should consider more possible programmes (qualifications), with a priority at the undergraduate level and a focus on flexibility for learners. For example, 85% of respondents agree or strongly agree that the OERu should add more programmes of study (qualifications) in addition to the Bachelor of General Studies (BGS). At the same time, with regard to the speed of expansion, 57% prefer that the OERu should complete the development of courses for BGS first. Among the OERu courses at all levels, undergraduate is most valued while the biggest variance exists in attitudes towards the development of postgraduate courses. There is a strong consensus (98%) that the OERu courses should be designed to provide learners with more flexibility through different streams and pathways. Likewise all respondents agree with providing pre-degree exit points in the OERu Bachelor of General Studies.

The survey reflects an essentially open attitude towards the minimum requirements for entry into the first year of the Bachelor’s degree level of OERu study. Over half (54%) agree that there should be no entry requirements, though a similar proportion of respondents (56%) regard language proficiency as a minimum requirement for international (second language) students. Respondents suggest that the OERu should consider alternative options such as multiple language courses and learners’ self-assessment in light of the goal to remove barriers rather than add them. The open mindedness here implies that the barriers to institutional entry requirements might not be as challenging as thought at the outset of the OERu initiative.

OERu course nomination

The survey shows both consensus and divergence on the important factors that impact course nomination. The quality of the courses, the utility for the OERu partners, and support from relevant academics are the top ranked factors. Interestingly, the lowest ranked items came with the biggest statistical variation, with a wide variation in respondents’ opinions towards revenue generating potential and the availability of additional funding . This perhaps results from the different clusters involved in the OERu, with some expecting increased revenue while others value social good.

The majority of partner institutions plan to reuse/relicense existing OERs for the OERu course design and development while only 6% will create new openly licensed content. Some concerns were expressed about the availability and suitability of existing OERs for developing specific OERu courses;other concerns included copyright issues in openly licensing existing courses. With regard to the most affordable and scalable types of learner support, the methods with less staff workload (labour) were highly rated, including for example, “Independent study materials and online help resources” and “dedicated community social media site(s) for learner peer support”, which also reflects the current direction of learning support design at the OERu. It is worth mentioning that none of the rest of the options were rated low, e.g. learning analytics. Design decisions therefore need to consider the balance between diversity, effectiveness, and sustainability of learning support methods.

Respondents generally agree to the membership commitments in terms of a minimum of two course nominations and many institutions hope to make additional contributions. For example, 42% of the respondents believe their institutions plan to assemble more than two courses and 11% plan to contribute one or more full programmes/credentials for the OERu. Interestingly from the 19 designated institutional respondents, the figures are even higher, 65% and 18% respectively. Further, 44% recommend that the completion of the first course nominated should be within one year of joining the network, and 37% within two years. These intentions seem to be somewhat optimistic considering the actual practical rate of course preparation to date. The majority (63%) of the respondents agree with the standard 0.2 FTE contribution requirement for Gold level partners, but 58% hope the OERu will recognise alternate contribution other than the assembly of two courses. These findings suggest that the OERu community supports the current framework of course nomination requirements, but a more pragmatic focus on the actual rate of course development and subsequent course delivery appears to be warranted..

Credentialing, assessment, and certification

The tension between open dynamics and institutional constraints in assessing and credentialing OERu learners is crucial. The respondents have identified a number of challenges and policy changes required in the open-ended questions of this survey, which will be reported later in the full report. The OERu has developed guidelines for cross-institutional credit transfer which could well be a constructive first step to overcome the obstacles identified. It is worth noting, however, that to date only 25% of the respondents have discussed these guidelines with their organization.

A number of related issues were also raised. Open badges are the most favoured (56%) alternative certification to formal academic credits, followed by micro-credentials (42%) and certified prior learning experience (38%). These preferences will likely influence the design of more flexible OERu certification models. Open badges will make the OERu certification system more compatible and interchangeable with other initiatives in the emerging open learning ecosystem. There was a huge variation in responses about the price that institutions would charge for providing assessment only services for OERu learners. On the other hand, 69% of respondents would support a process whereby OERu would develop ‘recommended retail price’ guidelines to assist in determining particular price levels at the institutional level.

Motivation, challenges, and sustainability

We repeated one question of the previous context evaluation survey regarding the motivations for participating in the OERu network. The three highest rated items are exactly the same as last time, and with least statistical variation (i.e. disagreements); namely, participation in an international network of accredited institutions, philanthropic contribution, and participation in a recognised OER initiative. In contrast, the lowest ranked motivations all come with big statistical variations. The respondents have the most diverse opinions on “Potential to reduce cost and save time associated with the development of learning materials”. Respondents were further asked to rate the most important factors to sustain their institutions’ OERu activities. The commitment to community service is at the top, followed by converting OERu learners to future full-fee students; but significant divergence exists in the questions relating to revenue generation. As such, it seems reasonable to suggest that the OERu should continue to position itself as an internationally recognised OER community with open education as an underpinning value, while at the same time incorporating partner institutions’ diverse motivations for both revenue generation and social good.

The biggest barrier for institutions to participate in the OERu is “competing demands on time and resources to maintain OERu project momentum”. Harnessing open dynamics to improve the efficiency of OERu models/systems could serve to ameliorate this concern. Respondents’ opinions on other challenges are mixed (illustrated by wide statistical variation in survey results), in relation to such concerns as: “Lack of exemplars of OERu courses…”, “Lack of continuity in senior management roles at the institutional level”, “Lack of accountability for non-delivery of agreed contributions”, and “Lack of guidance and support for new partners who join the network”. It is clear that the range of varied responses to these pressing issues should be a key focus for further discussion by the OERu community.

Technology support

The most popular Learning Management Systems (LMSs) in the OERu network are Moodle (62%), Blackboard (33%), and WordPress (31%), followed by a “long tail” of other LMSs . The diversity of LMSs across the network highlights the importance of the current OERu approach of promoting the authoring of course materials in a LMS agnostic fashion and thereby enabling solutions for the integration of outputs across multiple delivery platforms.

The respondents’ selection of the most important OERu technology considerations reflects strong interest in reusing OERu courses in local LMS operations; open source development and associated collaboration^. These considerations are consistent with previous findings that , for instance, most partners plan to develop open courses through reusing OERs, and that partners value local benefits in sustaining their OERu participation.

Open designs and open planning

There is a strong consensus that respondents place significant value on the “distinctively open” planning practices of the OERu. It is the only question in the survey where all options were rated with a statistical mean value of 4 and above. The strong consensus not only reflects agreement on the open paradigm for planning and managing the OERu network in practice, but also reflects the open culture and shared values among members of the OERu community. The survey results also suggest that in addition to collective decisions/actions, there is still an occasional need for top-down approaches in an open community. For example, 94% of respondents express a preference for a top down approach or a combination of bottom-up and top down approaches in decisions related to curriculum development at the programme level; likewise, 69% of respondents expected the OERu network to provide a framework for setting the recommended retail price for assessment services.