Peer Evaluation/Recommended rubric refinements

Key points

These refined rubric proposals are: Based on experiences using the alpha release prototype Additional desktop research on assessment rubrics Attempts to simply the system for users without comprising the potential validity of ratings.

Components of the rubric system

1. Criterion: Dimension to be evaluated, for example knowledge, relevance, critical thinking.
2. Levels: the rating scale used, eg:
   • Not achieved, achieved, merit
   • Below average, average, good, excellent.
   • Beginning, intermediate, exemplary
3. Descriptors: Definitions describing desired performance level.
4. Weightings: The relative proportion of the contribution of individual criteria to overall score.

Types of rubrics

Criterion referenced rubric (optional)

Criterion referenced rubric comprises objectively verifiable criteria of the Yes/No type. It is typically used to assess the completeness of a submission.

1. Optional means that this is not a required component of the evaluation system, but if included functions as described below.
2. It can be used alone or in conjunction with an analytic or rating scale rubric.
3. The "Completeness" criterion aims to assess whether the post meets the minimum criteria for "Achieved" and specifies criteria for a "Merit" rating. If the post does not meet the minimum criteria is it designated as "Not achieved" or other suitable 3 level typology.
4. Results for the "Completeness" dimension are displayed separately from the average score determined by and analytic or rating scale rubric, and can simply be displayed as "Not achieved", "Achieved" or "Merit":
5. The respondent does not rate the post as such, but merely completes a "check list" of yes/no type questions to verify if the minimum specified elements are present in the submission.

Example based on Activity 3.1

Rating	Criteria	System rules
Not achieved	Does the post relate to <insert activity name>? Yes /No	If respondent indicates that post does not relate to the question - the post is marked "Not achieved" and the peer evaluation system does not continue with the other rating items. If one or more of the criteria listed under the "Achieved" category are missing. The post is marked as "Not Achieved".
Achieved	Does the post contain two MCQ questions? Yes/No Does the post indicate which options are the correct answers Yes/No Does the post provide formative feedback for correct and incorrect answers Yes/No	If all criteria are marked "Yes" the system marks the post as "Achieved" If one or more of the criteria are marked "No" the system marks the post as "Not achieved"
Merit	Does the post embed or link to authentic examples on the Internet, eg images, rich media etc? Yes/No	If all the criteria for achieved are marked "Yes" plus the merit item(s) marked "Yes" - the system awards "Merit" If there is an item missing from the achieved list but the learner has included a merit item, the system still marks the post as "Not Achieved" but notes the inclusion of the merit item in the popup feedback of the evaluation. (In other words, learners cannot achieve merit if there are missing items from the achieved list.)

Notes

1. Include comment text field for the item on "does this post related to the activity".
2. Every peer evaluation submission should include the validation question confirming that post relates to the question.

Example based on Activity 4.1

Rating	Criteria	System rules
Not achieved	Does the post relate to <insert activity name>? Yes /No	If respondent indicates that post does not relate to the question - the post is marked "Not achieved" and the peer evaluation system does not continue with the other rating items. If one or more of the criteria listed under the "Achieved" category are missing. The post is marked as "Not Achieved".
Achieved	Does the post include text copied from a Creative Commons licensed resource? Yes/No Does the post include an extract from an all rights reserved source? Yes/No Does the post incorporate an image? Yes/No Have all sources been attributed including reference to the license of the source material? Yes/No Does the post include sources from two different Creative Commons licenses? Yes/No Has the learner applied a Creative Commons license to the derivative work? Yes/No Does the post include a short learning reflection on this activity? Yes/No	If all criteria are marked "Yes" the system marks the post as "Achieved" If one or more of the criteria are marked "No" the system marks the post as "Not achieved"
Merit	Does the post include a resource dedicated to the public domain (properly attributed)? Yes/No Does the post include sources from three different Creative Commons licenses? Yes?No Does the post embed a video resource from an Creative Commons licensed source? Yes/No	If all the criteria for achieved are marked "Yes" plus the merit item(s) marked "Yes" - the system awards "Merit" If there is an item missing from the achieved list but the learner has included a merit item, the system still marks the post as "Not Achieved" but notes the inclusion of the merit item in the popup feedback of the evaluation. (In other words, learners cannot achieve merit if there are missing items from the achieved list.)

Note: Include comment text field for item on does this post related to the activity

Example based on Second Learning Reflection

Rating	Criteria	System rules
Not achieved	Does the post relate to <insert activity name>? Yes /No	If respondent indicates that post does not relate to the question - the post is marked "Not achieved" and the peer evaluation system does not continue with the other rating items. If one or more of the criteria listed under the "Achieved" category are missing. The post is marked as "Not Achieved".
Achieved	Does the post approximate 300 - 400 words? Yes/No Does the post reference what the learner learned? Yes/No	If all criteria are marked "Yes" the system marks the post as "Achieved" If one or more of the criteria are marked "No" the system marks the post as "Not achieved"
Merit	Does the learner expand on what was learned by including how it was learned? Yes/No	If all the criteria for achieved are marked "Yes" plus the merit item(s) marked "Yes" - the system awards "Merit" If there is an item missing from the achieved list but the learner has included a merit item, the system still marks the post as "Not Achieved" but notes the inclusion of the merit item in the popup feedback of the evaluation. (In other words, learners cannot achieve merit if there are missing items from the achieved list.)

Note: Include comment text field for item on does this post related to the activity

Analytic rubrics

Analytic rubrics provide descriptors for each criterion at every level of the rubric. They can be used in conjunction with the criterion referenced rubric but not in conjunction with rating scale rubric below.

• Suggested UI behaviour: User clicks on the "cell" or corresponding star which best describes the performance for the criterion concerned.

Example based on Activity 3.1

Criterion	Level			Weight
	Beginner	Intermediate	Advanced
Knowledge of copyright	A number of errors suggesting that the learner does not understand the basics of how copyright functions. ☆	One or two errors interpreting the application of copyright in the context of the MQC questions. ☆	Exemplary answer. No errors in the application of copyright as demonstrated by identifying the correct answer and quality of the feedback on correct and incorrect answers. Learner has provided sufficient information relating to context to facilitate answering the question and/or avoiding ambiguity. ☆	0.7
Learning value of questions	Questions were confusing and many contraventions of suggested good practice for MCQ items, for example: Ambiguity created by using double negatives Instructions are unclear - respondent does not know what is required It appears that the author is trying to "trick" the respondent Distractors for wrong answers are not plausible ☆	Learner provides adequate context for the MCQ questions and has done a satisfactory job of implementing the requirements for good MCQ items. Only a few errors or suggested improvements to achieve an exemplary question. ☆	Exemplary answer. No errors in the application of copyright as demonstrated by identifying the correct answer and the quality of the feedback on correct and incorrect answers. Learner has provided sufficient information relating to context to facilitate answering the question and/or avoiding ambiguity. ☆	0.3

Notes

1. Scoring: 1 = Beginner, 2 = Intermediate, 3 = Advanced multiplied by weighting
2. Final score: could be displayed to users as 3 stars (eg ☆ ☆ ☆) but partially shaded according to aggregate score from all evaluations. For example an aggregate score of 2.5 would show the 1st star shaded full and 2nd star half shaded.
3. No of levels: System should allow user to specify the number of levels, that is 2 or more, although in practice 4 or more levels will become increasingly difficult to provide descriptors which will succeed in discriminating successfully. I suggest restricting to a maximum of 5
4. Layout Current table layout is not optimised for mobile display. Perhaps each level descriptor could be a row under the heading of the criterion.
5. Comments: Provide one open text field for comments.

Example based on Activity 4.1

Criterion	Level			Weight
	Needs improvement	Sufficient	Excellent
Critical thinking	The learning reflection is missing or statements relating to licenses choices are superficial without analysis. ☆	The learner provides sufficient analysis and evaluation of the license choices made. ☆	Exemplary answer. In addition to the analysis and evaluation of the license choices, the learner considers the implications of these choices and provides valuable advice with justifications on alternatives which educators should consider. ☆	0.5
Reflective learning	No references to new insights gained from the activity. ☆	An average reflection which references new insights gained from this activity. ☆	Exemplary answer. An excellent reflection which highlights exceptional insight into the complexities of remixing resources and how this experience will inform new practice. ☆	0.5

Notes

1. Scoring: 1 = Beginner, 2 = Intermediate, 3 = Advanced multiplied by weighting
2. Final score: could be displayed to users as 3 stars (eg ☆ ☆ ☆) but partially shaded according to aggregate score from all evaluations. For example an aggregate score of 2.5 would show the 1st star shaded full and 2nd star half shaded.
3. No of levels: System should allow user to specify the number of levels, that is 2 or more, although in practice 4 or more levels will become increasingly difficult to provide descriptors which will succeed in discriminating successfully. I suggest restricting to a maximum of 5
4. Layout Current table layout is not optimised for mobile display. Perhaps each level descriptor could be a row under the heading of the criterion.
5. Comments: Provide one open text field for comments.

Rating scale rubrics

A rating scale rubric lists the criteria and corresponding descriptions Respondents rate the quality of the item using a 5 point star system. It is useful to provide examples of what constitutes a weak or strong answer but not a requirement depending on the format of the descriptor.

Example based on 2nd Learning reflection

Criterion	Descriptor	Weighting	Rating
Relevance	Is the reflection relevant and meaningful to the learner and the aims of the course? Weak: Reflection is not related to the aims of the course and/or does not justify why the learning is relevant or meaningful Strong: Learner justifies why the learning is relevant and meaningful in relation to the aims of the course.	0.2	☆ ☆ ☆ ☆ ☆
Connection with prior experience	Does the answer provide example(s) of connection(s) with prior experience (what the learner already knows)? Weak: Learner has omitted to provide a connection with prior experience Strong: Learner provides relevant examples from prior experience which are related to the topic of study.	0.2	☆ ☆ ☆ ☆ ☆
Evidence of new learning	Does the answer provide evidence of new learning? Weak: Learner has omitted to reference new things learned Strong: Learner provides examples of new insights and new things learned related to the topic of study.	0.2	☆ ☆ ☆ ☆ ☆
Critical thinking	Does the learner think critically about the topic at hand? Weak: Learner has omitted to provide evidence of critical thinking or accepts things at face value without questioning or analysis. Strong: Learner assesses and evaluates perspectives and opinions and links these evaluations with own perspectives on the issue.	0.4	☆ ☆ ☆ ☆ ☆

Notes

1. I've deliberated on whether the system should provide options for rating scales, but feel we should restrict the rating scale to a standard 5 star system.

Thoughts about colour coding stars

I've been thinking about the idea of the "traffic light" analogy / assessment approach for the colour coding of the stars used for the rating system as a visual cue. The stars can be used as for displaying results and also used as input for the rating scale rubric approach.

From the learner's perspective (self evaluation) using a 5 star system:

1. Red (Star 1): I need help or need to work more, my post is not up to standard
2. Amber (Stars 2 - 4): I'm getting there - still need to improve on a few aspects
3. Green (Star 5): I've mastered this!

From the evaluator's perspective

1. Red (Star 1): Learners post is below standard - needs more work
2. Amber (Stars 2 - 4): Learner gets the basics but there are areas for improvement
3. Green (Star 5): Model answer which learners should emulate

For three star reporting

1. Red (Star 1)
2. Amber (Star 2)
3. Green (Star 3).