Foundation Skills/Developing an assessment/Objectives

Objectives

Objectives

Working through the resources, tasks and activities on this page will enable you to: Develop an understanding of a range of question types used in assessment tasks Understand the role and importance of using appropriate action verbs when designing assessment questions Understand the value of using Bloom's Revised Taxonomy when designing assessment questions Reflect on the factors that can influence the design of assessment tasks Explore ways of ensuring that assessment expectations are communicated to students Understand the difference between a marking scheme and a rubric Design a short assessment task Develop a basic rubric

Overview of question types

Web Resources

The questions used in assessment tasks fall into two broad categories: Closed questions (or fixed response questions): Questions in this category require students to provide a single correct answer to a question (e.g. a word, phrase, short decription, number, matching terms, selecting from a list of possible options) Open-ended questions: Here, students are required to write an original answer (e.g. an argumentative or persuasive essay, a plan for a new house, a marketing straegy for a new product). there are no 'correct' answers for this category of question. For suggestions on when to use these two types of questions, see the University of Texas Austin resource Deciding which exam question type to use. Questions can come in a variety of types, depending on what learning the examiner wants students to demonstrate. You will be familiar with the following commonly-used question types: multiple choice, true/false, matching, short answer, essay, computational, and oral. Waterloo University has a great resource Exam Questions: Types, Characteristics, and Suggestions which provides an overview of the main question types. Read through the Waterloo resource and then identify the question types that you most commonly use. Are there any pitfalls associated with the question types that you use?

Bloom's taxonomy

In 1956, Benjamin Bloom developed a taxonomy of thinking skills (cognitive skills). His six categories, in order of increasing cognitive demand, were:

• Knowledge, Comprehension, Application, Analysis, Synthesis, Evaluation

In 2001, Anderson and Krathwohl revised Bloom's Taxonomy - their six categories of cognitive skills, once again in order of increasing cognitive demand, are:

• Remembering, Understanding, Applying, Analyzing, Evaluating, Creating

The first three categories are sometimes called the lower order thinking skills (LOTS), whilst the latter three categories are often referred to as the higher order thinking skills (HOTS).

I've developed the following diagram to illustrate the hierarchy in the revised version of Bloom's Taxonomy, together with a sample of action verbs. Action verbs are the critical doing words used in questions. Quite often, the action verb indicates the complexity (level) of the question, e.g. when you ask a student to "justify a method of investing money", the cognitive demand is obviously greater than simply asking the student to "name one way of investing money".

When planning assessment tasks, first revisit the learning outcomes for your course. The action verb used in an assessment question must match the level intended in the learning outcome, e.g. if a learning outcome requires students to "list the tools used to install a wall-mounted flat-screen TV", then appropriate action verbs can include words such as list and name. You wouldn't ask students to "justify the use of tools ...".

Web Resources

Bloom's Taxonomy can be applied to assessment questions and also to classroom teaching, e.g. careful selection of appropriate questions to challenge students' thinking. A very useful resource, Bloom's Taxonomy “Revised”: Key Words, Model Questions, & Instructional Strategies, gives examples of action verbs, model questions and instructional strategies for each of the six categories or levels of the taxonomy. Examples of action verbs or clue words can be found on the University of New South Wales Learning Centre's website - see Exam Skills: Clue Words. Monash University provides two simple resources to support your understanding of Bloom's Taxonomy. The Identifying exam questions resource explains what examiners expect when using particular clue words, whilst the Identifying exam questions quiz allows you to classify a number of questions according to Bloom's Taxonomy using a drop-down menu. Try to quiz to see how well you understand Bloom's taxonomy.

Designing an assessment

The method of assessment ( e.g. observation, written theory test, practical report), the types of questions (e.g. multiple choice, computational), and the level of questions (e.g. remembering, creating) that you pose in an assessment task for your students will depend on a number of factors:

• the nature of the learning outcomes for your course (e.g. level according to Bloom's Taxonomy, theoretical or practical)
• the level of your course according to the NZQA framework (e.g. Level 4, Level 7)
• the abilities and experiences of your students at the time of assessment (e.g. basic, advanced)
• the resources available (e.g. space, computers, supervisors, consumables)

Ask yourself the following questions when designing assessment tasks:

• To what extent does the assessment task address graduate attributes as outlined in the graduate profile?
• Does the assessment task match the learning outcome(s) of the course?
• Are all learning outcomes covered by my assessment tasks?
• Is the assessment task reliable, valid, fair, clear, transparent, authentic, sufficient, manageable?
• Is the wording of the questions appropriate to the learning outcomes and the level of the students?
• To what extent do my assessment tasks in the course cater for different learning styles?
• Are students familiar with the method of assessment and the types/level of questions?
• Have students been given opportunity to develop their competence and confidence?
• To what extent do the assessment tasks support the scaffolding of knowledge and skills?
• What evidence of learning is appropriate and sufficient?
• Have students been made aware of your expectations? Do they understand these?
• How will meaningful feedback be provided to students?
• What level of moderation will be used?

Can you think of any additional factors to be aware of when designing assessment tasks? make a list if you can think of any.

Web Resources

The NZQA website gives a brief overview of factors to consider when designing assessments, but if you want to gain a more in-depth knowledge of designing assessments, then the Western Australian Government document Designing assessment tools for quality outcomes in VET is a good resource.

If you are new to designing assessments, you might want ot see what your colleagues have done previously (although don't assume that their assessment tasks are perfect!) - ask questions for clarification. The Internet and published books of question banks are both also valuable sources of new assessment questions.

Communicating expectations to students

In Unit 3 of this course, you were introduced to Chickering and Gamson's seven principles of good undergraduate teaching one of which was that good teachers communicate high expectations. Just as you communicate high expectations around course participation, so should you also communicate high expectations in your assessments.

in addition to providing students with written requirements for an assessment task, try to do the following:

• Post an electronic copy of the assessment task online (e.g. in Moodle) together with requirements and expectations
• Have a face-to-face assessment briefing in class to explain the assessment task and answer any questions that students may have
• Have a face-to-face assessment de-briefing in class about a week after the briefing session to clarify requirements and address queries
• Show examples of good, average and poor student work from the past (get permission from students!)
• Get students to judge pieces of work and to justify their decisions
• Provide opportunity for students to develop competence and confidence through practise examples
• Explain what you expect in response to particular action verbs
• Use suitable action verbs in your questions in class or online to develop student thinking
• Set aside time in class for discussions around assessment
• Provide marking schedules and rubrics where possible and exlpain to students how these work
• Educate students to avoid plagiarism
• Design assessment tasks to minimise plagiarism
• Provide useful and timely feedback on student performance andc encourage and advise students on how to do better

Web Resources

Plagiarism is a niggly problem that seems to be on the increase. When designing your assessment tasks, you can reduce the chances of plagiarism by: Ensuring that students know what plagiarism is and why they should avoid it Setting different assessment questions from year to year Placing less emphasis on questions that require recall of information Using essay topics that are less likely to come up in a search on the Internet via search engines Using Turnitin to screen student essays for possible plagiarism A word on Plagiarism: A great resource to educate students about the 'evils' of plagiarism is the Referencite academic referencing resource. It includes information about plagiarism, drop-down lists to teach your students how to do referencing, as well as video clips showing staff and students of different nationalities talking about plagiarism from their cultural perspectives.

Using marking schedules and rubrics

The Collins Online Dictionary defines a marking scheme (or marking schedule) as:

"(education) a plan or guidelines used in the marking of school children's or students' written work by teaching staff"

The marking scheme gives an indication of the mark allocation for various aspects of an assessment task (usually a task that has a low objectivity of scoring, such as an essay). The use of a marking scheme gives students an indication of what will count for marks, which aspects of the assessment task carry more weight and can also enable students to be more focused. For teachers, the use of a marking scheme increases the reliability of the assessment task. Examples of simple marking schemes can be found on page 2 of the resource Preparing examination questions (essays). The marking scheme does not usually indicate how an assessor will make judgements.

Andrade defines a rubric as:

"... a scoring tool that lists the criteria for a piece of work, or “what counts” (for example, purpose, organization, details, voice, and mechanics are often what count in a piece of writing); it also articulates gradations of quality for each criterion, from excellent to poor."

So, a rubric, in addition to providing the criteria (or aspects) that count for marks, also indicates how the teacher (assessor) will arrive at a judgement (and a mark) for each criterion (or aspect). See Figure 1 in Andrade's article Understanding Rubrics for an example of a rubric used to grade reports on student inventions. The Andrade article also provides a rationale and tips for creating rubrics.

Web Resources

If you are interested in using rubrics in your assessments, the following two resources will come in really handy: For plenty of links to examples of rubrics used in a variety of contexts, see the University of Wisconsin's Quick links to rubrics. If you would like to design your own rubrics, there is a free online Web 2.0 tool to help you! Rubistar allows you to create rubrics for a number of contexts, allowing you to choose criteria and judgement statements. In addition, there are also examples of rubrics on this website.

e-Activity

Assessment

e-Activity 7.4 For this assessment you are required to design an assessment task to test any of the higher order thinking skills that your students are expected to develop.(around 150 words): Which assessment method will you use? Which question type will you use? Write down your actual question, using an appropriate action verb. Using Rubistar, develop a basic rubric with only two criteria to assist in marking your assessment task. For those students enrolled in the Foundation Skills course through Otago Polytechnic, please capture your responses in a Word document with your name clearly visible (the rubric can be copied and pasted into the Word document) and submit your completed Word document using the appropriate Moodle assignment activity.

Summary

Summary

At the end of this topic you should have achieved the following outcomes. You will have: Developed an understanding of a range of question types used in assessment tasks. Understood the role and importance of using appropriate action verbs when designing assessment questions. Understood the value of using Bloom's Revised Taxonomy when designing assessment questions. Reflected on the factors that can influence the design of assessment tasks. Explored ways of ensuring that assessment expectations are communicated to students. Understood the difference between a marking scheme and a rubric. Designed a short assessment task. Developed a basic rubric. For those participants that are enrolled in the Foundation Skills course through Otago Polytechnic, you would have submitted evidence for the last two outcomes via the appropriate Moodle assignment activity. in addition, your rubric for the last outcome will have been developed using Rubistar. Another big topic, so well done if you've managed to complete everything!