OERu/Planning/OERu input evaluation/Input evaluation questions

What are the percentages of different types of OERs used in your course design?

Brainstorming

1. The OERF recruitment approach is a design consideration. In the early days we trialled blind letters of invitation which did not produce good results. The current approach today is to establish a lead, typically through a partner referral or networking channel. The OERF follows with a formal letter of invitation to a senior manager and offers to schedule a skype conference with the leadership team to answer any questions. The Director of the OERF (and sometimes a member of the Board joins the skype conference call.) Once institutions request a skype call to discuss further, we have a +90% success rate. Would it be worth considering a short interview with one or two of the new partners as part of the input evaluation? I also recommend including the same question from the Context survey on why partners have joined the network (or would consider joining) to assess if there are any changes in the rankings of the reasons for joining the network. (See Slide 23 for copy of the survey item.). So we could ask a number of recruitment related questions, eg. How did you fist find out about the OERu collaboration (conference presentation, personal email, membership of an online discussion list, the open web etc.) And then importance of factors contributing to partner decision to join, eg information on OERu website, information provided in letter of invitation, Web conference call with director of the OER Foundation, Existing OERu partner referral etc. --Mackiwg (talk) 03:38, 1 December 2014 (UTC)
2. A key design issue relates to the challenges partner institutions face when identifying potential course nominations. For example, finding faculty who are supportive of the OER model, or internal institutional barriers in identifying and nominating candidates. I think it would be extremely valuable to establish the local institutional barriers to identifying OERu course nominations and solutions / approaches which partners have found useful. The design question is how should the OERu nominate courses for future development taking local constraints into account --Mackiwg (talk) 05:28, 22 October 2014 (UTC)
3. General design question: Are we targeting an OERu partner institutional response, or should we allow multiple responses from different people at the same partner. My preference is to get richer data by allowing multiple responses from the same institution. --Mackiwg (talk) 03:35, 1 December 2014 (UTC)

Evaluation Background

This evaluation will examine the options and resources for achieving the OERu goals from a design perspective and provides reliable and systematic evidence to support relevant decision making and strategic planning. It will be a formative evaluation with short final report.

It has three main aims:

1. To review the design of curriculum, programme,and courses that embeds OERs into open courses and provides formal assessment and accreditation to online learners. Different design practices will be assessed and best practices documented in order to inform future design;
2. To assess the availability, quality, and effective uses of resources including OERs, budget, volunteers, and technology supports and identify the unmet needs of participating institutions;
3. To provide a snapshot of the major barriers and challenges OERu partners are encountering in practice.

To accomplish this, the evaluation is targeted at two groups of participants:

• An online survey will be conducted with all 35 OERu partners and participating institutions; 6 categories of survey questions have been defined according to the aims of OERu working group and 2014 operational priorities
• Interviews will be conducted with selected members of OERu working groups.

A constructivist approach will be adopted in developing survey and interview questions as well as the analysis of data, in which the participants are not passive sources of information, but will help “...affirm foundational values; define evaluation questions; clarify evaluative criteria; contribute needed information; and assess valuation reports...” (Stufflebeam 2003b p.11).

A wiki page will be created as a medium for the collaboration.

Survey Questions (draft for Wiki collaboration)

Institution and Participant Information

Comments on this question

1. may need a couple of questions here, any suggestions?
2. Information which would be useful:
   • Level of respondent (Executive leader, Senior management, Middle management, Educator, Administrator etc.
   • Do we restrict the survey to OERu partner institutions? -- or do we include prospective / future partners?

Curriculum and programme development (Bachelor of General Studies)

Which of the following credentials is your institution able to issue to OERu learners?

1. Certificate
2. Diploma
3. Advanced diploma
4. Associate Degree / Foundation degree
5. Bachelor’s Degree
6. Postgraduate Certificate
7. Postgraduate Diploma
8. Other (please specify)

Comments on this question

1. All OERu partner institutions must be accredited with their national or regional accreditation agency and will be authorised to award credentials. Why do we need this information? is this to determine the ratio, for example of Associate Degrees to Bachelor degrees within the network? --Mackiwg (talk) 00:18, 1 December 2014 (UTC)
2. Wondering if its worth exploring the ability of partners to award alternate exit credentials, eg Certificates of Participation (for learners not interested in formal credit), micro-credentials (eg certificates of learning for successful completion of components of a course which could be recorded for future articulation to transcript credit when the set of micro-credentials relating to full course credit have been completed.) --Mackiwg (talk) 00:18, 1 December 2014 (UTC)

What is the minimum number of courses and credit points needed by your institution for Bachelor of General Studies?

Comments on this question

1. We already have reasonably good data on the local "residency requirement", aka the minumum number of local credits needed for credentialing from the recent BGS and articulation survey. I suggest we drop this question. --Mackiwg (talk) 03:28, 21 October 2014 (UTC) Question Deleted. Xiangren

How many compulsory courses do you think are needed for Bachelor of General Studies and what are they (levels, subjects)?

Comments on this question

1. I think this is an important design question, but don't think that the survey is the best way to collect the information because the answer is dependant on the context of the proposed OERu BGS degree structure. It would be better imo to table an "idealised" BGS structure and then to ask OERu partners who have a BGS on their books if the proposed structure will fit their local qualification. --Mackiwg (talk) 10:12, 6 November 2014 (UTC)

Should an OERu programme of study cater for different streams of pathways in order to provide learners with flexibility?

• Strongly Agree
• Agree
• Disagree
• Strongly Disagree

Comments on this question

1. The OERu partners operate within existing degree structures, so those with an existing BGS are bound by their local degree requirements. In the recent BGS partner survey we know that 52% of partners with a BGS provide streams. I think a better approach is to ask the design question - Should an OERu programme of study cater for different streams or pathways of study ? Perhaps with a SA, A, D, SA scale. --Mackiwg (talk) 03:38, 21 October 2014 (UTC) Question changed. Xiangren

Should the OERu place a restriction on the time allowed for degree completion?

Comments on this question

1. This is a significant design question for the OERu. There may be local / national legislative requirements for the time required for degree completion in the case of students funded from government grant. However, in the case of the OERu, partners will not be claiming government grant so perhaps we need to rephrase the question to ask whether there are restrictions on the time required for completion assuming the partner doesn't claim government grant / EFTs support. --Mackiwg (talk) 10:15, 6 November 2014 (UTC)
2. Taking my previous comments into account, I think we should guage three dimensions 1) Opinion on whether the OERu should place restrictions on time allowed for completion 2) If there are any statutory requirements for time of degree completion if institution is claiming government grant for students 3) If the organisation can waive statutory degree completion requirements if they are not claiming government grant for OERu learners. --Mackiwg (talk) 00:23, 1 December 2014 (UTC)

If answering yes, what is the maximum period of time within which a learner is permitted to complete the degree (please also identify the types of degree, i.e. Bachelor of General Studies, or others)?

Comments on this question

1. Similar challenge, partners will be bound to existing requirements of their degrees. A better approach would be to gauge opinion, for example Should the OERu place a restriction on the time allowed for degree completion and a follow up question indicating the maximum period for degree completion. (We also need to think about how the survey instrument will deal with difference between the BGS for partners that have them and other degrees. --Mackiwg (talk) 21:43, 21 October 2014 (UTC) <strike through>
2. For now, the OERu decision is to work with a Bachelor of General Studies as the inaugural credential, so it may be premature (and more complex) to survey alternate credentials. Within the OERu model institutions work within existing policies, so are bound by local requirements. My sense is that we would not glean valuable input design information by focusing on credentials other than the BGS at this juncture of the OERu journey. --Mackiwg (talk) 03:44, 1 December 2014 (UTC)

In your opinion, which of the following should be minimum requirements for entry into first year of OERu study be?

• None - if the learner passes the summative assessment, they should get the credit
• School leaving credential
• language proficiency in the language of instruction
• Equivalence assessment of other types of qualifications and experiences
• Exemption by age (older learners don't require prerequisite credentials.)
• Others (please specify)

Comments on this question

1. I would consider rephrasing this item a little. Remember that entry requirements may also vary depending on level of study (eg 2nd or 3rd year). There may also be statutory restrictions linked to government funding which would not permit organisations to change entry requirements. I would simplify the item by focusing on the 1st year. Eg In your opinion, which of the following should be minimum requirements for entry into first year of OERu study be? a) None - if the learner passes the summative assessment, they should get the credit b) School leaving credential c) language proficiency in the language of instruction c) Exemption by age (older learners don't require prerequisite credentials.) --Mackiwg (talk) 21:50, 21 October 2014 (UTC) Changed. Xiangren
2. I think we need a 2nd related question here to find out if there are any institutional and/or statutory restrictions to waiving entry requirements for 1st year of study. --Mackiwg (talk) 00:29, 1 December 2014 (UTC)
3. It may also be worth asking if institutions can waive minimum school leaving credentials on the basis of age - some institutions can waive these requirements for older students. --Mackiwg (talk) 00:29, 1 December 2014 (UTC)
4. The other design issue is that some institutions will permit entry for foundation study courses which don't have minimum entry requirements, but if students pass the foundation courses, they will gain entry to degree study. Also if the institution has foundation study courses, we will need to find out if these are credit bearing for any 1st year level of study. --Mackiwg (talk) 00:29, 1 December 2014 (UTC)

What is the percentage of OERu learners in your courses transforming to formal assessment services for degrees, or expected?

Comments on this question

1. We are in the early phases of implementation and we don't have enough reliable data to assess conversion rates. However, we do have a number of surveys completed by participants in a number of the OCL4Ed courses where we asked participants if they would consider formal assessment services if these were available. Perhaps we should analyse and unpack the results from these surveys? --Mackiwg (talk) 10:19, 6 November 2014 (UTC)
2. Perhaps we need to think prospective business models for OERu partners (eg expectations for conversion to full-time study / matriculation at the conferring institution.) I think we need a open business model section in the survey to explore business model design options. --Mackiwg (talk) 00:47, 1 December 2014 (UTC)

Course Design

How many courses are you developing for OERu, what are they, and at what stage of development?

Course Name

Course types*

Development stage**

Credits Points***

Other information

* Course types include

** Development stage information includes: (1) Complete and ready for piloting the OERu model; (2) Undergoing development but nearing completion for piloting the OERu model; (3) Early stages of development; (4) Conceptualised but not yet undergoing development; (5) Not yet conceptualised but participation discussed or agreed to

*** credit points for Bachelor of General Studies or other credits transferable.

Comments on this question

1. We have a separate process for nominating courses which are recorded in the wiki, so you have access to relevant data about existing nominations. I would avoid duplication information we already have. --Mackiwg (talk) 00:58, 1 December 2014 (UTC)
2. We could tweak the question to find out if partners are intending to 1) Stay with the 2 -course contribution 2) Planning to develop more than 2 courses for OERu 3) Planning to contribute one or more full programmes for OERu. --Mackiwg (talk) 00:58, 1 December 2014 (UTC)

What are the percentages of different types of OERs used in your course design?

• original open resources your institution developed
• original open resources collaboratively developed by you and partner institutions
• existing OERs embedded without revision
• repurposed/remixed open licensed existing OERs
• others (please specify)

Comments on this question

1. I assume that respondents will indicate an approximate percentage for each category? --Mackiwg (talk) 21:06, 6 November 2014 (UTC)
2. What are we aiming to determine with this question? OERu courses must be based entirely on OER or open access resources. I suspect that the breakdown among the different categories will be determined by the inventory of existing open materials in the subject area or discipline. --Mackiwg (talk) 03:24, 1 December 2014 (UTC)
3. Perhaps the design question we are asking is to establish the dominant model for OERu course development, eg 1) Assembling courses from existing OER and open access materials versus 2) Releasing existing courses at the institution under an open license? --Mackiwg (talk) 03:24, 1 December 2014 (UTC)

What learner support and administration do you provide in your courses?

  

• social networking functions in LMS
• social media (e.g. Facebook, Twitter, etc.)
• workplace or community organizations
• learning analytics for automated support
• staff workload for student support
• various volunteers
• peer-to-peer learning support
• retired academics
• others (please specify)

How many leaners will be able to access the support?

Comments on this question

1. Question needs clarification. I would imagine that the support mechanisms would be available for all students who participate in the course so not sure what is meant by percentage --Mackiwg (talk) 21:08, 6 November 2014 (UTC)
2. Given that the dominant OERu model is not to provide lecturer student support, we may need to re-craft the question to determine opinion on the forms of "support" which OERu partners deem valuable for the OERu model? --Mackiwg (talk) 03:48, 1 December 2014 (UTC)

Student Assessment and Accreditation

 

What assessment methods have you used for formal credits (degree pathway learners) in your course design?

o   PLAR

o   Challenge exams

o   Workplace-based assessment

o   Same assessment as for fee-paying students

o   Proctored examinations

o   Course-based portfolio

o   Automated online assessment based on learning analytics

o   Use of volunteer assessors

o   Practicums (e.g. Art portfolio, lab work, trades practicums for plumbers, carpenters etc.)

o   others (please specify)

What assessment methods have you used for lifelong and workplace learners in your course design?

o   PLAR

o   Challenge exams

o   Workplace-based assessment

o   Same assessment as for fee-paying students

o   Proctored examinations

o   Course-based portfolio

o   Automated online assessment based on learning analytics

o   Use of volunteer assessors

o   Practicums (e.g. Art portfolio, lab work, trades practicums for plumbers, carpenters etc.)

o   others (please specify)

What is the estimated cost for formal assessment and credentialing (or the suggested price for learners)?

Comments on this question

1. I would avoid confusing cost with price. I suggest that we design an item which focuses on anticipated total fees for assessment services. --Mackiwg (talk) 03:20, 1 December 2014 (UTC)
2. We need a preceding item on determining the average "course size" in notional learning hours, because "larger" courses may carry a higher assessment fee. --Mackiwg (talk) 03:20, 1 December 2014 (UTC)
3. We should also consider a separate item relating to fees for institutions who will provide micro-credentials. --Mackiwg (talk) 03:20, 1 December 2014 (UTC)

What designs/processes/strategies could be used to ensure OERu credits being transferred and accepted by different institutions for degrees?

Comments on this question

1. This information will be covered in the draft guidelines for credit transfer, so we don't need to survey this item.

Quality Control

What are the criterion you use to select existing OERs for your course?

o   existing refereed scholarship

o   institutional approval

o   educator’ recommendation

o   suitable copyright licenses (e.g. full text accessibility, reusability)

o   conduct peer review of OERs for course design

o   relevance and suitability for courses (no need for revision)

o   others (please specify)

Comments on this question

1. In practice, we assume quality by virtue of working with accredited institutions who would not want to compromise their accreditation status or brand. Also, I would imagine that the context determines which methods are most important. Not sure how much value we gain by asking this question? --Mackiwg (talk) 03:51, 1 December 2014 (UTC)

What aspects are essential in the agreement of quality standards (OERu quality guideline) in OERu courses?

Comments on this question

1. This is specified in the guidelines for credit transfer. I don't think we need to survey this. Perhaps we should shift the focus to assess whether the guidelines are sufficient?

Strategy, Management, and Communication

How much investment has your institution made so far and are you satisfied with the ROI in OERu?

• manpower (staff workload)
• budget
• tech infrastructure (e.g. resource archiving, LMS uses, …)
• tech support (staff workload, monetary investment, etc.)
• capacity building
• others (please specify)

What resources or support are most needed from OERu and/or your institution?

• financial support
• recruitment of volunteers
• buy-out time
• committed staff members
• examiners or markers
• training and support
• a supportive community
• others (please specify)

How do you rate the models of operation and management at OERu (please rate by 1 to 5, 5 is the highest rate)?

• transparent information communication
• collective decision making
• volunteer-based creative labor
• trial/pilot based on strategic priorities
• others (please specify)

Working Group Aims and 2014 Operational Priorities Relating to Survey Questions

Curriculum and programme development (Bachelor of General Studies)

[i] Based on the working group of Curriculum and Programme Development

WG aims:

• Develop a clear degree structure with levels, streams and exit points whilst still providing for student flexibility and choice
• Clear pointers to universities that can grant the whole Bachelor of General Studies or equivalent degree.  

2014 Operational Priorities:

• Establish alternate pathways (streams) of study within the OERu programme of study

Course Design

[ii] Based on WG2 (what are the best practices in open course design, how to assemble OER into courses, most effective models/methods, learner supports, comparing different plans/methods, )

WG aims:

• incorporate guidelines and suggestions for course design and development incorporating exemplars of "best practice".

2014 Operational Priorities:

• Develop 20 full courses (or equivalent micro courses) during 2014
• Publish for OERu course design including exemplars of best practice
• Design and implement a streamlined process for nominating OERu courses

Student Assessment and Accreditation

[iii] Based on WG 4

WG aims:

• Agree solutions for communicating degree pathways, cost of assessment and degree completion for prospective OERu learners.
• To develop an OERu course on how to prepare a portfolio for PLAR including possibilities for credit recognition at partner institutions using the OERu model.
• To develop an implementation guidelines document to inform credit transfer and course articulation decisions for the OERu network drawing on the remix scenarios published in the Report on the Assessment and Accreditation of Learners using OER.
• To implement an OERu Transnational Qualifications Framework modelled on the Transnational Qualifications Framework developed by the Commonwealth of Learning for mapping OERu courses according to level and type of qualification.
• To establish foundations in the design of the OERu credit transfer and course articulation guidelines with future scalability in mind to enable the network to implement systems like the British Columbia Council on Admissions and Transfer (BCCAT).

2014 Operational Priorities:

• Publish operational guidelines for credit transfer and course articulation

Quality Control

[iv] based on WG 3

WG aims:

• Design and implement a streamlined process for nominating OERu courses which contributes to building the OERu programme of study incorporating a peer review element for feedback.
• Develop and agree minimum quality standards for OERu courses incorporating the concept of "OERu designated" courses which are judged to meet these minimum standards.
• To specify the minimum requirements related to credit transfer and course articulation for inclusion into the quality standards for an "OERu designated" course (for example, acceptable OERu standards for validating student identity and assessment design and practice).

2014 Operational Priorities:

•   Publish quality guidelines for OERu designated courses

Strategy, Management, and Communication

[v] Based on WG2 and the last two WGs

WG aims:

• provide existing and new partners joining the OERu with the information they need to maximise the benefits of membership and participate actively in the operations of the OERu network.
• To coordinate the open development of the OERu Strategic Plan 2014 - 2017
• To produce an integrated set of documents including the strategic plan and links to associated management structures, and the operational priorities for 2014 for approval by the OERu Council of CEOs.
• To ensure alignment of the OER Foundation strategy with the OERu strategic plan
• To create, deliver and communicate the value of OERu study for prospective learners
• To support the effective recruitment of new OERu partners
• To improve OERu member relationship management guiding interactions of the OERu with its member institutions

2014 Operational Priorities:

• Recruit 10 new OERu partners
• Progress the design and development of Academic Volunteers International

Technology Development

[vi] based on Technology WG

WG aims:

• accommodate the needs and desires of the course development workgroup
• maintain scalable, secure, cost-effective infrastructure for collaborative course authoring and delivery
• interoperate with partner institution legacy systems, ideally federating similar systems
• share open data formats, sources, and services to the greatest extent possible
• deliver content in a variety of formats, including mobile (maintain separation between content and rendering)
• implement solutions for customised local branding and theming of shared content resources
• adapt to the increased importance of audio and video