Analysis of SP4Ed prototype courses
OERu
Towards more affordable education for all students worldwide
Contents
- 1 Introduction
- 2 Provisional analysis of data from the OERu SP4Ed prototypes
- 3 Links
|
Introduction
- SP4Ed 13.05 (22 May to 5 June 2013) was presented as a small pilot to test the design of the course, a new wiki-based registration feature for emailing course instructions and a blog aggregator for the WENotes course feed.
- SP4Ed 13.07 (29 July - 9 August 2013) was offered in parallel mode for students registered for the Change with Digital Technologies in Education course (EDEM630) in the Postgraduate Diploma in Education (e-Learning and Digital Technologies) at the University of Canterbury in parallel with free OERu learners. This was the first OERu prototype mOOC to be offered in parallel mode. SP4Ed 13.07 did not provide assessment services towards formal credit for the free OERu learners, however, participation in this mOOC was a requirement for the EDEM630 students.
A key feature of these pilot SP4Ed mOOCs was the international spread of participants providing a unique opportunity for students registered at the University of Canterbury to interact with an international community of learners during SP4Ed 13.07. This level of internationalisation is not easily replicated with traditional online courses at residential universities. The SP4Ed13.05 offering generated baseline data which could be used to compare engagement metrics when integrating the SP4Ed 13.07 mOOC within a formal academic course. Both SP4Ed instances generated positive evaluation results.
Provisional analysis of data from the OERu SP4Ed prototypes
Registration statistics
| mOOC | Registrations | Countries |
|---|---|---|
| SP4Ed 13.05 | 50 (including 2 facilitators) |
14+ (some learners declined to provide country information) Argentina Australia Canada Fiji Germany India Ireland New Zealand Romania Saudi Arabia Swaziland Trinidad and Tobago United Kingdom USA |
| SP4Ed 13.07 | 121 (including 2 facilitators) |
32+ (some learners declined to provide country information) Argentina Australia Bangladesh Brunei Darussalam Canada China Denmark Fiji Ghana Guyana India Ireland Italy Jamaica Jordan Kenya Korea, Republic of Mexico New Zealand Nigeria Philippines Portugal Russian Federation Rwanda South Africa Spain Sri Lanka Sweden Trinidad and Tobago USA Ukraine United Kingdom |
Who are the OERu mOOC participants?
The data gathered to date suggests that the average OERu mOOC leaner is English speaking (70%), engaged in full-time employment (86%) and has a bachelors degree or higher qualification (90%) and rates themselves as an intermediate or advanced level online learner (83%). Consequently the current OERu mOOC participants may not be representative of OERu learners in the future.
Notwithstanding the relatively high skills level in online learning indicated by respondents, the majority (58%) of learners had not participated previously in a Massive Open Online Course (MOOC). In addition, the majority of participants (73%) did not maintain a regular blog or use RSS readers or feed aggregators (66%) suggesting that many of the OERu mOOC participants were not regular users of the key technologies used for interactions in the SP4Ed mOOCs. In this regard, we believe the SP4Ed mOOCs have contributed to improving the range of digital skills for many of the active participants. Of particular interest is the large majority (76%) indicating that they had a Twitter account before registering for the mOOC course which suggests that the inclusion of microblog activities in our mOOCs offers a channel of communication used by a large number of educators.
How does participation and engagement compare?
Any attempt to analyse participation and engagement in a cMOOC-like course is fraught with difficulties on multiple dimensions, and we can justifiably question the utility of these quantitative metrics in understanding the learning value of open courses.
The open nature of these courses is such that learners can sip and dip into individual concepts of personal interest which is not possible in closed course environments. "Non-participation" in course activities does not detract in anyway from the quality of the learning experience for those actively engaged in the mOOC. On the contrary, a single microblog post from a "visitor" to an open course adds value for those more actively engaged. Quantitative data does not capture the range of reasons why individuals register or interact with these open courses. In the case of the OERu we are only able to gather data about intent from those individuals who actually complete the new participant survey (about 47% of registered OERu participants.) As OERu courses do not require registration for participation, it is difficult to establish with any accuracy how many "non-registered" learners interact with the materials or distributed learning networks associated with these courses.
Nonetheless the SP4Ed 13.05 and SP4Ed 13.07 courses provides a unique opportunity to compare data recognising that this is an oversimplified view of the dynamic of open courses. Both SP4Ed courses used identical materials, the same delivery model facilitated by the same facilitators. The only difference being that participation for SP4Ed 13.07 was a requirement in a formal university course. This comparison provides an initial insight into developing hypothesis for future study and gaining a better understanding of the questions which can inform the integration of micro open course models for the OERu partners.
Respondents who completed the new OERu participant survey indicated a high level of intended participation before the course. 48% indicated that they would be "moderate contributors" (i.e. adding contributions from time to time) and 47% predicated that they would be "memorably active" (i.e. completing >80% of the readings and activities).
Based on the submission rates of the new OERu participant survey combined with the assumption that completion of this initial survey provides a activity benchmark during the initial phases of the SP4Ed mOOCs, we estimate that approximately 54% of those who originally registered for the course intended to participate more than being passive observers. We attempted a rudimentary estimate of SP4Ed 13.07 "course starters" by counting the number of participants who made at least one contribution (for example, a microblog, blog or Google+ post) during the first session. We made an adjustment to accommodate all the international time zones, however this analytic would not capture learners who started the course outside of the allocated time schedule for Session 1 given the asynchronous nature of the course materials. We excluded posts from individuals who had not registered for the course. These would typically be Tweets or Re-Tweets about the start of the course from non-registered participants on the assumption that they were not intending to engage. Using this metric, we conservatively identified 39 "starters" or 32% of the original course registrations. This estimate is conservative when measured against the average page views of the course materials for all registered participants. Judging by the page views, it is not unreasonable to assume that a number of unregistered participants were reading the course materials. What is interesting is that the majority (61.5%) of "course starters" using this metric were free OERu learners.
Summary of participation by type of technology
The table below provides a summary of the number of the posts by source technology including the number of unique individuals who posted using the respective technology. Note that we counted 32 unique bloggers during SP4Ed 13.07 compared with the estimated 39 "course starters" using our rudimentary metric. Of particular interest is that these 32 bloggers generated an average of 5.78 blog posts per poster which is high given that there were 6 recommended E-Activities requiring substantive blog posts in the SP4Ed course.
| Source | SP4Ed 13.05 Posts |
SP4Ed 13.05 Participants |
SP4Ed 13.07 Posts |
SP4Ed 13.07 Participants |
SP4Ed 13.05 Avg. posts per poster |
SP4Ed 13.07 Avg. posts per poster |
Comparison |
|---|---|---|---|---|---|---|---|
| Ask Q&A | 11 | 6 | 4 | 4 | 1.83 | 1 | -45.4% |
| Blogs | 52 | 17 | 185 | 32 | 3.05 | 5.78 | +89.5% |
| g+ | 12 | 7 | 67 | 13 | 1.71 | 5.15 | +200.1% |
| identica | 1 | 1 | - | - | - | - | - |
| Questionstorm | 117 | 12 | 397 | 24 | 9.75 | 16.54 | +69.6% |
| 181 | 27 | 568 | 52 | 6.7 | 10.92 | +62.9% | |
| WEnotes | 205 | 17 | 659 | 31 | 12.05 | 21.25 | +76.3% |
| Favourites | - | - | 130 | 12 | - | - | - |
| Moodle* | - | - | 9 | 6 | - | - | - |
| Total interactions | 579 | 2023 | |||||
|
Notes
| |||||||
We calculated an index benchmark based on the number of posts per unique poster as a basis to compare SP4Ed 13.05 with SP4Ed 13.07 which was incorporated into a formal university course. The number of interactions and learning artefacts created and shared with the group was higher for all source technologies apart from the community-based question and answer forum. Embedding SP4Ed 13.07 within a formal university course has resulted in improved outputs and engagement ranging from 62.9% to 200.1% better than the SP4Ed 13.05 pilot which was not offered as part of a formal course. We suggest that the reduction in the number of new posts to the course student support question and answer forum is because the majority of questions were already answered by contributions during SP4Ed 13.05. So we surmise that the "Ask Q&A" resource was used as "read-only" during 13.07.
Summary of SP4Ed web site statistics
As an open course, learners are encouraged to search and find open access materials based on their own interests. The interactions during the course show evidence of learners sharing resources they found to inform the development of their own scenario's. However, we have not attempted to analyse the links shared among SP4Ed learners in this summary. Consequently, the statistics reported here only show a snapshot of the information we collated from server statistics on WikiEducator. Notwithstanding the narrow perspective of this lens restricting analysis to the WikiEducator web site, there is interesting data showing that learners are consulting the open materials developed for the SP4Ed mOOCs. SP4Ed relies considerably on links to open access resources distributed on the Internet which are used as stimuli and examples for the relevant topics in the course.
The table below provides a summary of the page visits for the course materials hosted on WikiEducator comparing SP4Ed 13.05 and SP4Ed 13.07. We calculated the average page visits for the total number of initial registered students as an index to compare the two courses. The actual page visits of active learners per session is probably higher.
| Pages | Number of "teaching" sub-pages |
SP4Ed 13.05 Visits |
SP4Ed 13.07 Visits |
SP4Ed 13.05 Avg. visits per number of registrations |
SP4Ed 13.07 Avg. visits per number of registrations |
Comparison (Times better/worse than SP4Ed 13.05) |
|---|---|---|---|---|---|---|
| SP4Ed node pages | 9 | 587 | 5873 | 11.74 | 48.53 | 4.13 |
| Session 1 Establishing your personal learning environment |
26 | 44 | 1061 | 0.88 | 8.76 | 9.95 |
| Session 2 Introduction to scenario planning |
8 | 40 | 1072 | 0.80 | 8.85 | 11.06 |
| Session 3 A manager's perspective on scenario planning |
5 | 23 | 537 | 0.46 | 4.43 | 9.6 |
| Session 4 Drivers of fundamental change in education |
6 | 103 | 875 | 2.06 | 7.23 | 3.5 |
| Session 5 Create your own scenario |
3 | 6 | 392 | 0.12 | 3.23 | 26.9 |
| Totals | 813 | 9810 |
How does the interaction and activity of SP4Ed compare?
The page views for the majority of teaching Sessions for the SP4Ed 13.07 mOOC achieved an average of just above 1 page view for each tutorial page based on the initial number of course registrations. This suggests a higher number of "silent observers" who are consulting the course pages when compared to the estimates of "active" course participants above. However, in an open course where than registration is not mandatory to access the course materials it is difficult to provide a definitive answer to who is reading the course pages. Session 2 which focuses on a manager's perspective of scenario planning recorded a lower average of 0.8 page views per initial registration. The new OERu participant survey indicates that only 10% of the learners are managers or administrators which may account for the lower page views or interest in this topic.
It is interesting to compare the data generated from the SP4Ed mOOCs with emerging international data on MOOC participation rates. However this an uneasy comparison given the appropriateness of course completions and what constitutes "massive" as meaningful measures for opening access to networked learning opportunities. The MOOC delivery model is associated with "low" completion rates estimated around 10% of original registrations[1][2]. Our own data collected from the OERu new participant survey, albeit a very small population, indicates that 25% of respondents who had previously participated in a MOOC had completed one or more of these open courses. As suggested earlier, completion rates should not be used as a proxy to evaluate the appropriateness or quality of an open course given the diverse reasons for registering in these free courses.
The SP4Ed 13.07 course would not necessarily "qualify" as "massive" but has utilised a number of cMOOC-like features, for example using a course feed aggregator generated from learner interactions distributed across the Internet. Stephen Downes suggests that Dunbar's Number for massive is 150 active participants referring to the "maximum (theoretical) number of people a person can reasonably interact with[3]". While these SP4Ed courses may not meet the "150" threshold, they do illustrate the point that massive is a question of scale. SP4Ed 13.05 generated 579 "posts" over 10 teaching days which would average about 58 posts per learning session and an achievable number of interactions to read assuming a learner were to read all the posts. However, SP4Ed 13.07 generated 2023 "posts" which would average out at 202 interactions to read per learning session if SP4Ed students were to read everything generated during the course and unlikely to be achievable during a single learning session. Based on these metrics, we believe that SP4Ed 13.07 achieved the critical mass of active contributions to provide the course with sufficient activity to fuel meaningful peer-to-peer learning support.
Of particular interest is that Session 5 of SP4Ed 13.07 has resulted in a significant improvement in the page visits to course materials for the concluding session of SP4Ed. Session 5 contains three "teaching" pages and SP4Ed 13.07 has recorded at least one page view of each of the teaching pages for each registered participant during the concluding session. This suggests that the SP4Ed 13.07 trial of embedding the mOOC within a formal university course may have contributed to improved retention and completion rates when compared with SP4Ed 13.05.
SP4Ed course evaluations
The OER Foundation administered an optional online survey for SP4Ed 13.05 and SP4Ed 13.07. Response rates for these evaluation surveys was low recording 14% on initial course registrations for SP4Ed 13.05 and 17.4% for SP4Ed 13.07. However, if we calculate the response rate on the estimated "course starters" as calculated above, the survey response rate for SP4Ed 13.07 increases to 53.8% which is probably a more realistic estimation of the survey response rate from learners engaging in the course.
Notwithstanding the limitations of generalising the data beyond those who responded, overall the SP4Ed pilots have received positive evaluations. The majority of respondents (82%) rated the course as "excellent" (50%) or "good" ("32%") with the remainder evaluating the course as "fair" (18%). A summary of the evaluation items is presented in the table below.
| Indicator | SA % | A % | Agree % | D % | SD % | Disagree % |
|---|---|---|---|---|---|---|
| The SP4Ed course met the specified objectives | 37 | 56 | 93 | 7 | 0 | 7 |
| The SP4Ed materials are relevant | 56 | 44 | 100 | 0 | 0 | 0 |
| The SP4Ed materials are well organised | 41 | 44 | 85 | 15 | 0 | 15 |
| Instructions for learning activities are clear | 33 | 63 | 96 | 4 | 0 | 4 |
| The learning activities are interesting | 43 | 46 | 89 | 11 | 0 | 11 |
| The difficulty level of the SP4Ed course was appropriate for me | 22 | 70 | 92 | 8 | 0 | 8 |
| I would recommend the SP4Ed mOOC to others | 44 | 41 | 85 | 15 | 0 | 15 |
- Notes
- SA = Strongly agree; A = Agree; D = Disagree; SD = Strongly disagree;
- N = 27 respondents
- Notes
The original course structure and layout of the SP4Ed pilots highlighted areas of improvement potential in the information design adopted by the OERu through earlier design consultations. These design challenges are corroborated by the evaluation data given a lower relative average for the course structure when compared to the other evaluation items. The major challenge identified was a design flaw mixing functional information types, for instance, conflating information required when registering for a course with resources needed to support the learning process. For example, the aggregated course feed (used during the course) appeared on the page providing the course summary (usually only read when registering for the course.) This resulted in unnecessary link redundancy in an attempt to improve navigation for the learner and wasting of real estate on the browser page. These problems were amplified when integrating a wiki-based registration form for both SP4Ed 13.05 and 13.07. The OERu is addressing these information design issues through the 2nd design consultation.
Qualitative feedback on integrating the mOOC model in a university course
Selected student feedback on the open questions for SP4Ed 13.07 provides insight into the experience of integrating a mOOC within a formal university course. See for example the following comments from students:
- "... this has been my first tweet, wiki and mOOc. I have loved it as it is the best [professional development] and learning I have done in decades of academe and teaching workshops."
- "More Please. Most stimulating learning in my entire Masters course papers. Couldn't maintain this pace for too long though. The comparison between this course and others at the same level within the university is huge, both in requirements, what it offers, the presentation, the rigor of thinking required. This is on a different planet- I am better off for having participated- but it does make me view some other courses with a different lens now."
- "... this is an awesome course. I recommend it unreservedly to all teachers, scholars and funders - support this course as it supports teaching and learning like no other."
- "Brilliant experience. I have already recommended this to a few colleagues and friends. It was exciting and challenging to be a participant on SP4Ed."
- "Absolutely delighted and impressed by the quality of world-wide people in the course videos who offered valuable insight into each component of the sessions. ...Also getting to know students quickly by what they posted as tweets/WEnotes. Although they were micro-blogs, you got to know the 'personality' of the participants and you could log in anytime day or night and someone else was on too."
- "I enjoyed creating the blogs and reading other participants and would now have no hesitation to use them with classes now (lack of knowledge before this course and feeling a little insecure of how they work etc has made me not have one before now)."
- "Linking with so many new participants via Twitter has been amazing - as well as 'following' all these new people, I have also 'followed' lots of new Tweeters which is brilliant for networking purposes and keeping up to date quickly."
- "I now have a fascination for open online learning and have recommended it to literally hundreds of people over the last couple of weeks. Thank you for an amazing learning experience particularly into the scenario planning."
- "This was my first mOOC and so a steep learning curve but I really enjoyed this aspect. The course structure was brilliant and well presented in clear episodes."
- "Really opened my eyes to something I new nothing about!"
Evidence of learning and learning behaviours
| Item | Often | Sometimes | Rarely | Never |
|---|---|---|---|---|
| How often did you read the #SP4Ed feed of microblog, WENotes and blog posts? | 68 | 29 | 3 | 0 |
| How often did you read the popular recent notes feed in WikiEducator (New feature in SP4Ed 13.09) | 43 | 38 | 10 | 10 |
| How frequently did you visit the Ask OERu community-based question and answer forum during the course? | 11 | 21 | 36 | 32 |
The aggregated course feed is a powerful feature used to bring distributed interactions on the web from multiple sources into a timeline which represents the pulse of mOOC activity. SP4Ed 13.09 generated 2023 posts with 64% of the traffic being microblog posts or averaging 3.3 microblog posts per day for each active student. The OERu has been trailing a community based question and answer forum with the capability to post questions and for the community to vote-up good questions and answers. We seeded the Ask OERu website with a number of relevant questions and answers. In theory the technology would be able to scale well as one of the mechanisms for providing peer-based student support, however the evaluation data indicates that Ask OERu is not widely used to support learning. When asked whether the question and answer community forum supported learning, 67% of the respondents in the evaluation survey indicated that the question was not applicable suggesting that they did not use the technology. Interestingly, 44% of the respondents recommend that the Ask OERu question and answer database should be incorporated into future courses. It would appear that a more static wiki-based question and answer page would serve future OERu learners equally well as the community features of the Ask OERu technology have not been adopted as a mainstream learning technology by OERu students to date.
The SP4Ed mOOC was designed for a total of 50 notional learning hours including the preparation time for the summative assessment assignment for offerings where the mOOC is embedded in a formal university course. The course was designed for 25 hours of teaching-learning interactions plus 25 hours for the final assignment preparation. The SP4Ed 13.09 participants were asked to report the time they spent each day working through the course materials and activities. The weighted average was 2.3 hours for each of the 10 teaching days for the course, thus totalling an estimate average of 23 hours. The workload of the course falls within the specified design parameters.
Respondent views on micro-credentials
We asked respondents who completed the online evaluation for SP4Ed 13.09 for their views on whether SP4Ed should offer the option for a micro-credential in the future. The majority (76%) suggest that a micro-credential should be considered for the future with 24% of the respondents indicating that they were unsure. None of the respondents suggested that micro-credentials should not be considered. The following table summarises the recommended types of certification:
| Type of certification | Percentage |
|---|---|
| A digital badge issued by an OERu member which learners could display on their personal website | 24 |
| A certificate of participation | 14 |
| Micro-credential which includes formal assessment and certification but without recognition to official course credits | 5 |
| Micro-credential which includes formal assessment plus recognition towards formal course credit | 52 |
| No certification required | 0 |
| Other | 5 |
We asked respondents for their opinion if a micro-credential were to be offered, whether they would consider paying an accredited institution for summative assessment services assuming that these services were limited to covering the costs of assessment. 48% indicated that they would be prepared to pay for formal assessment services towards a micro-credential.
Views from the University of Canterbury
[Extracted from an article written for the DEANZ Magazine (2013, in press) with the authors' permission]
As leader of the University of Canterbury e-Learning Lab Niki Davis had negotiated for the university to become a founder member of OERu, which led her to volunteer this course as a contribution to our philanthropic initiative and commitment to community service. It seems right to innovate while, at the same time, facilitating professional learning about change with digital technologies. There is a synergy and an opening up of many opportunities for both the individual participants and the organisations involved. However, it did result in an uncomfortable degree of exposure for Niki and her university as we designed out in the open on WikiEducator. The feeling was akin to the exposure that can be felt when climbing a steep mountainside and the best way to reduce it is to have a buddy and rope up. Wayne was the buddy that gave Niki the confidence to handle the exposure successfully.
The changes involved in working with OER and OERu have been far more extensive than Niki could ever have imagined! For example, advanced course content design had to be much tighter to handle the unpredictable number of participants who could and did enrol from many diverse contexts and countries with widely varying levels of knowledge and skills. Niki had become accustomed to envisaging what her students needed with many opportunities to add more when additional needs becomes obvious or individuals requested more. Another example was the OERu application of micro-blogging, rather than her accustomed facilitation of a coherent threading of discussion forum posts in Moodle. Instead the mOOC had communication in micro-blogging that OERu technology cleverly drew together in a stream linked to the home page of our mOOC as well as each participant’s OERu course dashboard. The stream of messages was exciting and coherence could be drawn from it, particularly with judicious selection to read and respond to some of longer Blog postings as well as the stream of shorter messages. University students who wished to remain in a quieter backwater could participate from within a threaded discussion forum in the University’s Moodle LMS, and their contributions were passed on to the stream.
...
For Niki Davis, as a university teacher educator [and Professor of e-Learning], the most important feedback has been from the students, most of whom clearly enjoyed the challenge. [already quoted above]... In addition, the OERu report was discussed online with the students in a University of Canterbury discussion forum before the end of the course. Over the years Niki has often had students suffer ‘withdrawal symptoms’ when their online experience comes to an end at the end of the course and that remains one indication that a learning community had been formed. It appears that the intensity of our mOOC experience also engendered that feeling for at least one who said online: “I personally miss it [the mOOC] although at the time it felt like we were racing down a snowy slope with no brakes! The work now feels a little flat and grey. It’s a bit like when you finish a really really good book and realize the author hasn't written a sequel yet. I hadn't realized how many countries had participated in the mOOC.” This provides evidence that the mOOC was of the cMOOC type.
Students’ questions included some that are being actively researched to better understand engagement in open courses, including this mOOC variation (see for example Ossiannilsson & Creelman 2012; Witthaus 2012). For example, a student’s question about the intention of non university participants who studied with mOOC led Wayne to clarify that participants in an open course environment engage for a wide range of reasons, which may not be related to traditional study and, in addition, the nature of open courses is such that it is particularly hard to analyse why they participate.
Our SP4Ed mOOC has generated valuable insights into formulating hypothesis for future research, for example:
- The page views of the learning resources correspond with the number of original registrations suggesting that "silent observers" remained for the duration of the course. This is not a typical phenomenon judging by the data generated from many of the commercial xMOOC offerings that have large attrition rates. We suggest that the calibre of contributions from students enrolled and paying to study generates a critical mass of engagement "motivating" many more participants to continue interacting with the course materials.
- The inclusion of a wide international audience added value to the learning experience for the students enrolled with University of Canterbury, who would not have engaged with participants from over 30 different countries in an online course that is closed to learners that are not enrolled.
This mOOC project provides a practical example of how to integrate the local learning management system with a course on the open web. The SP4Ed mOOC has demonstrated a win-win strategy for universities to provide an authentic international community learning experience while widening access to learning opportunities through an agenda of social inclusion. As universities are challenged with rising costs, the mOOC model combined with OER could contribute to more sustainable education futures.
Perhaps most importantly of all for the University of Canterbury and our students was that the mOOC showed that we are open to sharing our resources and learning in a way that benefits the global community, which is the goal of OERu. Thus the engagement of these students in the University of Canterbury postgraduate course in New Zealand is supporting an international philanthropic collaboration that aims to assist in reducing the challenges of widening access to affordable tertiary education for learners who, for whatever reason, are excluded from the privilege of a tertiary education. In other words, we and our students were prepared to take a risk to make a difference, which is in line with the University of Canterbury’s recently revised vision and graduate outcomes as “people prepared to make a difference”.
Acknowledgements
Thanks to all who made this innovative project possible particularly our students, mOOC participants, OERu staff, and the University of Canterbury including staff in Learning Resources particularly Herbert Thomas who leads the ELM team. This article retains the Creative Commons licence adopted by OERu. The University of Canterbury membership of OERu is a project of the e-Learning Lab.
