Evolving b-learning practices - do they increase the quality of online learning?
Name of the institution: Universidade de Aveiro
Type of institution: Higher Education Institution
3th Cycle in Multimedia in Education (in Portuguese)
Practices described for the WikiEducator Book - Dialogue on Quality of User generated Content
Contents
Context
Brief description of the course
This contribution reports on evolving practices of e-assessment in higher education collaborative blended learning contexts based on the paradigm of assessment for learning aiming to improve the quality of elearning in the contexts of postgraduate studies at the Universidade de Aveiro. Taking into account the results of previous studies (see, for instance, Loureiro et al., 2007, Pombo et al., 2010 or Loureiro et al. 2010) the strategies, including the e-assessment tasks, explored in two Distance Education (DE) modules are described as well as their evolution.
The DE module is part of the curricular year of a doctoral program on Multimedia in Education where students are asked to collaboratively write a literature review about a topic of their choice. The followed strategies (described below) clarifies how quality issues were taken into account. Peer assessment (that is according to Topping et al., 2000, a process whereby students assess or are assessed by their peers), has been used since students can benefit from it as reported by several authors. Although peer assessment, including formative feedback, has been extensively used to support students' learning in traditional classrooms and questions related with its the goals and quality are addressed by different researchers (Gielen, 2007), the literature indicates that there is a lack of studies about online peer assessment. Moreover, little is known about its quality (Ertmer; Richardson; Belland; Camin; Connolly & Coulthard, 2007), namely in online collaborative learning contexts using Web 2.0 technologies and user generated content. Waycott; Gray; Thompson; Sheard; Clerehan; Richardson & Hamilton (2010: p.1041) posit that “there has been little guidance in the published literature on what constitutes good assessment practices when students are asked to create and publish content, or participate in networking activities, using social web technologies”. Our experience is a small contribution to reduce this gap.
Objective
What are the expected learning outcomes?
The research competencies that postgraduate students are supposed to develop in the doctoral program, and that underlies the creation of the 3th cycle in Multimedia in Education, are those required for independent research, such as: seeking and systematizing information, data gathering and analysis, communication, collaborative work and assessment skills (self and peer assessment). Bearing this in mind, the expected learning outcomes of the DE module are
i) to retrieve, select and analyze relevant information (papers, books, dissertations, reports…) about a topic related with DE;
ii) to share, discuss, negotiate meanings/points of view expressed in the selected information and point out criteria to evaluate its relevance for the topic;
iii) to contribute towards the creation of a friendly and participatory atmosphere;
iv) to systematize and synthesize information regarding the construction of an academic publication;
v) to use ICT properly and critically in the research process;
vi) to communicate, orally and in writing, and contribute for the development of that skill among peers;
vii) to assess the progress of the work that was produced collaboratively, as well as individual contributions (self and peer assessment) and provide constructive suggestions, based on the literature;
viii) to reflect upon the competencies that were developed by each one and by the colleagues.
Methodology
Evolving activities to increase elearning quality?
Project work, problem solving, collaborative learning and assessment for learning are valued as strategies that promote quality in online learning. The duration of the DE module was, in the first edition (2008/09), a five-week period with extensive online elements and two face-to-face sessions. In the second edition (2010/11), the module was delivered in a shorter period (four weeks).
Following the results of the evaluation of the first edition (Pombo et al., 2010) some changes in the design of the DE module were introduced, namely concerning the e-assessment activities, as well as the communication technologies used. Table 1 shows that the assessment activities included a formative component to which special emphasis was attributed and that, in both editions, the assessment framework (e.g., the weight of the different dimensions, criteria and indicators, students’ involvement…) was negotiated with the class in the first face-to-face session.
In order to contribute towards the reflection about the ongoing work and the learning outcomes, in the 2008/09 edition students were asked to do a weekly individual reflection (self assessment). In addition, the fist version of the group work was assessed by peers (each group assessed another group) and by the teachers (table 1). The teachers’ and peer assessment of the first version of the ongoing work should include a score as well as qualitative feedback to facilitate the improvement of the work and was provided through the module Wiki based webpage and therefore open. Although this option could strengthen lack of confidence feelings as well as fear of exposing errors, it could also help students fulfil the task by learning from the feedback posted by their peers.
Taking into account Topping et al. (2000) typology of peer assessment, in the first edition of the DE module the peer assessment of the ongoing project work can be described as being both summative and formative, made by groups of students of the same year/module, using an online open tool, and thus not confidential, compulsory, supplementary (teachers assessed the ongoing group work, after peer assessment), contributing to final grades (only the summative peer assessment) and without extrinsic reinforcement.
The analysis of the quality of the formative peer assessment provided by the students in the 2008/09 edition showed that the overall quality of students’ peer assessment could be better, since the majority of the groups didn’t provide enough constructive feedback. Moreover, although peer assessment included criticism (both positive and negative) and suggestions for improvement, the groups didn’t question their colleagues (Loureiro et al., 2010).
To improve the students’ attitudes and perceptions about assessment for learning, in the 2010/11 edition it was decided to provide students with extra opportunities to use the assessment criteria of the group work (it was used for formative e-assessment of three versions of the ongoing work instead of one). On the other hand, the assignment of the task to the different groups was made by email. This way the assessed group didn’t know which of the others groups analyzed their own group work. Consequently, peer feedback was confidential and could therefore be more authentic. As to the teachers’ assessment, like in the first edition, it could be accessed by all the persons involved in the module.
Although, in the first f2f sessions, library technician made workshop sessions about literature search tools, in the former edition students felt difficulties with the criteria and indicators related with the “adequacy of the information search strategies” and the “evaluation of the information using appropriate criteria” (Loureiro et al., 2010). Consequently, the teachers concluded that punctual sessions about information retrieving techniques were important but not fruitful, especially if they were not reinforced throughout the module. Thus, in the 2010/11 edition and because that constraint was identified, the teachers decided to follow the students’ work more closely and provide them regular feedback on the development of this task (based on peers’ and teachers appreciations), and decided to make available to the students the criteria and indicators to be used to assess the ongoing work, from the beginning of the module – which did not happen in the 2008/09 edition. Consequently, the transparency of the assessment criteria and indicators was greater.
Table 1- teaching and learning strategies explored in two editions of the Distance Education module
Assessment
How are students assessed?
The assessment framework includes formative and sumative e-assessment performed by both the teachers and the students (self- and peer assessment) as described above.
User-Generated Content experience
Short description of the experience
As described above, during the DE module, students in small groups write a literature review about a topic related with DE. During the first week and f2f session, students are provided with information and guidelines aiming the development of their information literacy competences. This information includes a Dias Figueiredo’ short paper on how to write a research paper, available at Academia.edu, as well as what is a research literature (personal notes, presentations, videos, articles…), that is accessible in a website aiming to promote collaboration between postgraduate students and facilitate the research supervision process (RedeS I&D em Educação). A workshop is also organized with the collaboration of library technicians. Students have to negotiate the topic, search for relevant information, define criteria for inclusion and exclusion (e.g., time scope, keywords, authors…) and collaboratively structure and write the literature review. As mentioned above all the process is supervised at a distance and formative assessment, based on the ongoing work as well as weekly reflexive tasks, is provided both by the teachers (facilitators) and students peers.
Objective
Which is UGC activity/ies objective?
To develop research competences to be able to do independent research, such as information literacy, academic writing, collaboration…
Users
What kind of learners and number will be involved?
PhD students: 24 from the 2008/09 edition and 18 in the last one
Methodology
How it is developed? Includes technological tools
The methodological options are presented in previous sections. Concerning the tools, in the first edition the activities were developed in a wiki based website (EaD wiki) but students used other web 2.0 tools to communicate, share and discuss ideas. In 2010/11, the communication was done mainly using a social network tool (Ning - Laboratório de Conteúdos Digitais) and Google tools. In both editions the postgraduate organised the literature using EndNote Web. This option facilitated the collaborative development of a common bibliographical database. Since the online version of the EndNote software does not allow the parameterisation (to introduce research notes, like the objectives or research questions, methodological options, conclusions, reflexions about the reading…), a specific worksheet was created for that purpose (an adaptation of a similar instrument created in the context of the European Project – Galapro).
The criteria and indicators that were used to assess the ongoing work are available at the EaD Wiki - http://ead0809.wetpaint.com/page/Feedback+artigos.
Describe just main issues
The described practices are considered practices of user generated content (UGC) since, in the context of the DE module, the content created by postgraduate students (literature reviews) was shared freely and during the writing process the content has not been evaluated through a process of formal peer review (like it happen when a paper is submitted to a peer review journal). However formative assessment of the ongoing work was done regularly both by the teachers (that acted as facilitators) and by students peers. As also reported above, students performed self-reflexive task concerning the group work and the developed competencies.
Taking into account the CONCEDE framework for the peer creation - peer-validation work, the writing process involved the activities presented in table 2. Moreover and since the DE module has been evaluated by the students, by university professor not involved in the module (peer-evaluation) and submitted for publication in journals as well as international conferences, the experience has been evaluated at the first and second level of the quality pyramid (Module 3 of the framework).
Link to CONCEDE Quality Framework
References
Ertmer, P.A, Richardson, J.C., Belland, B., Camin, D., Connolly, P. & Coulthard, G. (2007). Using peer feedback to enhance the quality of student online postings: An exploratory study. Journal of Computer-Mediated Communication, 12(2), article 4. Retrieved 30/12/2010, from: http://jcmc.indiana.edu/vol12/issue2/ertmer.html.
Gielen, S. (2007). Peer assessment as a tool for learning. PhD dissertation, University of Leuven. Retrieved 20/11/2010, from: https://lirias.kuleuven.be/handle/1979/1033.
Loureiro, M. João, Pombo, L. & Moreira, A (2007) follow the link above.
Loureiro, M.João, Pombo, L. & Moreira, A.: (2010). The quality of peer assessment in a wiki based online context: a qualitative study in a post-graduation module. Proceedings of the ICEM & IODL Conference, Turkey (2010).
Pombo, L., Loureiro, M.João & Moreira, A.(2010). Assessing collaborative work in a Higher Education blended Learning context: strategies and students' perceptions. Educational Multimedia International, 47(3), 217-229.
Topping, K. J., Smith, E.F., Swanson, I., Elliot, A. (2000). Formative Peer Assessment of Academic Writing Between Postgraduate Students. Assessment & Evaluation in Higher Education, 25( 2), 149-166.
Waycott, J., Gray, K., Thompson, C., Sheard, J., Clerehan, R., Richardson, J., & Hamilton, M. (2010). Transforming assessment in higher education: A participatory approach to the development of a good practice framework for assessing student learning through social web technologies. In C.H. Steel, M.J. Keppell, P. Gerbic & S. Housego (Eds.), Curriculum, technology & transformation for an unknown future. Proceedings ascilite Sydney 2010 (pp.1040-1050).
