Comments from Brian

Fragment of a discussion from Talk:Peer Evaluation
Jump to: navigation, search

A colleague just sent me a link to this paper which is relevant: "Tuned Models of Peer Assessment in MOOCs" I have not got around to reading it but thought it would be useful so I'm posting it immediately.

Chris Piech Stanford University Jonathan Huang Stanford University Zhenghao Chen Coursera Chuong Do Coursera Andrew Ng Coursera Daphne Koller Coursera


Brianmmulligan (talk)02:08, 17 May 2014

Thanks for posting the link to that paper!

As a technician, it suggests several elements that need consideration:

  • having students review submissions that have been reviewed by "experts" (ground truth) which is a variation on Mika's comment about a library of sample works
  • partitioning reviewers by native language in an attempt to remove that bias
  • recording "time spent grading" a submission is challenging in a distributed environment like the OERu courses that have been offered to date
    • (Their "sweet spot" of 20 minutes spent grading an assignment sounds like a significant time commitment for our mOOC assignments.)
  • if karma is used, it maybe necessary to factor the marks an evaluator has received, not just those he has given (and had commented on)
  • a large discrepancy in scores might signal the need to add additional reviewers of a particular submission
  • how to present scores in a meaningful way especially if there are different weights being applied, or some evaluations are discarded, etc. in an environment where individual evaluations are open
JimTittsler (talk)19:21, 19 May 2014