'Right' calibration

Use calibration to ensure that grading is reliable

Key questions addressed in this section

What is good practice in ensuring students and staff are clear on how to validly and consistently allocate different grades?

Suggestions from Fellowship participants

The approach to assuring the clarity, validity and reliability (consistency) of grading is ideally confirmed and supported by a process of intermarker ‘calibration’ (see references below). The process for ensuring inter-marker calibration discussed at the workshops includes attention to the steps summarised in Box Nine.

Box Nine

A suggested process for using inter-marker 'calibration'

A small group of experienced graders in a unit of study comes together each year:

Each gives a grade to a selected and de-identified piece of student work (using the marking rubric/grading indicators provided for the unit of study concerned);
They then compare the grades each of them allocated to the same piece of work and identify what indicators they used to determine the grade;
From this stocktake they use a process of consensus to come to an agreed position on a common grade for the selected piece of assessment and the indicators they would all use to justify it;
This is repeated a number of times with different samples of student assessment;
From this a ‘lonely planet guide’ for other markers on what they have agreed to and the indicators used is written by the group and distributed to all future markers.

It was noted that producing the self-teaching guide for grading suggested in Box Nine is one way of avoiding the extensive time and opportunity costs of repeatedly bringing all markers (full time and sessional) together for face-face consensus building.

It was also strongly recommended that the results be shared with students at the outset of each unit of study using exemplars of different grade levels so that students know exactly how grades will be allocated; and that the use of assessment-focused learning guides for each unit of study similar to those developed at universities like Western Sydney University be considered.

Resources and further guidelines

Barrie et al (2012): Assessing and assuring Australian graduate Learning outcomes, OLT, Sydney pgs 48ff.
Griffith University: Assessment matters! – click on the section marked ‘consensus moderation’.
Hancock, P & Freeman, M (2014): Achievement matters: external peer review of accounting standards, OLT, Sydney. See especially Sections 3.3: External calibration and 3.4: External Assurance pgs 25ff and the discussion of project success factors (Section 6.1), Challenges (Section 6.2) and Key Lessons (Section 6.3) on pgs 49ff.