Assessment Moderation

—
DSC assessment moderation guidelines

These guidelines explain what assessment moderation is and why is it important. The guidelines provide a holistic moderation process complete with examples, recommendations and a checklist for adaptation to your learning and teaching (L&T) context.

Download PDF Version1MB

What is assessment moderation?

A cyclical process in five stages (see figure 1) that assures systematic fairness in assessment by including internal and external processes where multiple stakeholders identify, agree upon, assure and monitor standards (see figure 2).

Figure 1: Phases of the assessment moderation cycle

Download Assessment Moderation ChecklistA guide to key steps in the process (428 KB)

Phase 1: Designing assessment for equity (fairness and consistency)

Focus: planning for the quality of assessment tasks with reference to course and program learning outcomes and to achieve fairness and consistency.

Points to check:

Do all students have equal opportunity to demonstrate learning outcomes?
Is there any possible bias or discrimination in the assessment criteria, their possible interpretation or application ?

Processes: informal or formal peer scrutiny, or review by a committee or professional accreditation body.

Effective Practice : Calibrating assessment tasks (Examinations in the School of Property, Construction and Project Management)

The School of Property, Construction and Project Management (PCPM) has developed rigorous moderation processes and tools to moderate examinations. In collaboration with the responsible program manager, the course coordinator appoints one or more examination moderators to check a proposed exam before it is released. Using a pro forma (refer PCPM Exam Moderation Checklist), the moderator reviews content coverage, technical accuracy, the academic validity of the examination questions and the marking scheme, clarity of questions and instructions and the quality of the layout. The course coordinator actions agreed amendments to create a new moderated version that is then referred to the program manager for approval. Only after associate dean or program manager approval is the exam finalised.

Phase 2: Calibrating understanding about assessment

Focus: assessors reach agreement and shared understanding about the assessment task(s), requirements, performance criteria and standards to assure the rigor and consistency of assessment tasks and processes.

Point to check:

Has consensus building involved the use of professional or disciplinary judgement, genuine debate and reference to student work?

Processes: sharing samples of student work, independent marking, comparison and justification of judgements through a process of ‘consensus moderation’, or discussion held to reach agreement on a final mark.

Effective Practice : Calibrating assessment practice – feedback and critique in the School of Art.

In the School of Art program teams across the discipline of art have developed a framework for educational practice related to feedback and critique on students’ creative practice. Over a series of workshops lecturers calibrated their ideas about the knowledge and skills required to engage in effective feedback and critique in group tutorial sessions, peer-to-peer learning and student-lecturer discussions about student artworks. Collaboratively, lecturers agreed upon four domains that are required for effective and meaningful feedback and critique. These domains include: 1) Engagement: the personal attributes required to engage in ongoing dialogues about creative practice; 2) Process: the methods and processes used to make creative works; 3) Context: the disciplinary, social, political and historical context of the practice and the discourse surrounding those contexts; 4) Encounter: the experience of the artefact/ installation/ performance/ creative outcome.

These domains have been calibrated amongst program teaching teams for authenticity and then translated into assessment rubrics. Each domain has been defined to support shared understandings and assessment practices that foster learner agency in developing creative work. An online resource is being created to support both students and staff in this practice.

Phase 3: Justifying marking judgements (credibility and reliability)

Focus: internal processes for assuring the quality of assessors’ judgements.

Points to check:

To what extent are assessors’ judgements aligned to assessment criteria?
How credible is the evidence supporting assessors’ judgements?
Have assessors agreed common standards?
How consistent are assessors’ judgements?

Processes: random checks, double marking, consensus moderation and checking of grade distribution.

Note: Review of grade distribution may be useful as a starting point for discussion of standards, but assessment criteria, student work and the alignment between these and assessors’ decisions should also be considered.

Effective Practice : Consensus moderation – School of Media and Communication

In the Bachelor of Communication (Media) consensus moderation occurs at course level. Each assessment piece worth more than 20% is moderated by pairs of markers who share samples at different grade levels and check these against the marking rubric. Major assessment (worth 40% or more of the total mark) is consensus moderated by the full teaching team. All fails and borderline grades are cross marked.

Phase 4: Externally validating processes and judgements

Focus: benchmarking internally agreed standards through external comparisons.

Points to check:

Has benchmarking included review of course outlines, assessment tasks and marking criteria, as well as marking?
Were external reviewers trained?
Were external review processes calibrated?
Was time for discussions provided to enable external reviewers to build shared understandings and standards?

Processes: peer review or professional accreditation. Some disciplines and university groupings have agreed processes for cross institutional peer review to benchmark assessment outcomes, e.g. https://www.iru.edu.au/iru-work/calibration/.

Phase 5: Monitoring and evaluation of assessment efficacy

Focus: evaluation of overall quality and efficacy of assessment and moderation process to assure quality and contribute to effective educational practice development as an ongoing cycle of reflection and adaptation.

Processes: reviewing samples of student work, student evaluation data, examiners’ reports, grade distribution and lecturers’ perceptions about the adequacy of assessment task design, information, criteria and standards, including any marking guide. This may lead to adaptation in future iterations of the course.

Please refer to the moderation checklist for further guidance.

Effective Practice : BP309 Bachelor of Design (Digital Media).

In the Bachelor of Design (Digital Media), all Digital Media courses at RMIT Vietnam are moderated at the program level by course coordinators and lecturers. At the end of semester, final moderation involves review of final grades, with instances of a Fail (NN) grade supported by written reasons for that grade. Grade distribution is also reviewed and discussed with any revisions or adjustments undertaken through a further moderation process by the RMIT Vietnam teaching team before final results being published. The grade moderation offers opportunities for reflection and development, as both program managers gain better insight into successes, challenges, similarities and differences at each campus for the semester. Program managers conduct regular meetings to discuss all updates and issues relating to the program. As part of this peer review process, L&T discussions focus on course content and approaches at each campus.

Why is assessment moderation important?

Assessment moderation:

Is critical for the development of effective educational practice, including ongoing reflection
assures academic quality by verifying the credibility of assessment as evidence of student achievement and the credibility and reliability of assessors’ judgements
may support equity
may help to meet professional and system requirements

Equivalence and comparability

Where courses are delivered at multiple locations, assessment moderation will require consideration and planning for equivalence and comparability.

Equivalence: maintaining quality standards across multiple offerings in terms of course management, learning outcomes, resources, assessment, teaching, work-integrated learning (WIL) and internationalisation.

Comparability: extent to which offerings have been contextualised and customised to meet local needs, including those of the student cohort, to achieve equivalence.

Contextualisation: Customisation of learning activities, resources, assessment, delivery, teaching practice and student learning support to facilitate learning for a specific student cohort.

See RMIT program and course review process for further information

Who should be involved in moderation?

Course coordinators have principal responsibility for assessment design and quality, including moderation processes, but this responsibility needs to involve multiple stakeholders in ongoing discussions and genuine exchange.

Other stakeholders include teaching teams (local and international) and program advisory committees and may also include professional and industry bodies, relevant community partners and contacts.

Genuine collaboration and reciprocity among teaching teams and other stakeholders is especially important when designing curriculum and assessment in transnational teaching contexts. This means:

valuing and drawing upon the specific and contextual professional knowledge and expertise that all members of the teaching team bring
using data to learn about the student cohort
differentiating core curriculum content from customisable content
constructively aligning curriculum
being adaptive during delivery
evaluating performance data.

Discussions need to happen:

when designing assessment (alignment with professional standards, qualification level, learning outcomes and activities), assessment criteria and marking guides
through calibration of assessment requirements, standards and achievement and validation of judgements through ongoing evaluation and adaptation of assessment design and practice.

Figure 2: Key stakeholders in assessment moderation

Recommendations for effective practice

Assessment moderation should involve ongoing dialogue

Talk to your teaching team as early as possible about assessment and keep up the conversation during the teaching period.
If possible, collaborate on developing assessment criteria, standards and evidence of performance, keeping in mind the need to align assessment criteria, assessment tasks, learning outcomes and pedagogy. This will help all assessors to internalise requirements and standards and inform conversations with students about the assessment requirements.
Keep in mind the role of culture in shaping communication, interaction, pedagogy (student and teacher roles, teaching practices) and assessment, including underlying values and assumptions.
Be alert to stereotypes and ethnocentricity and remember the importance of context for L&T, particularly for transnational offerings.

Consider developing marking guidelines and exemplars and aim to moderate assignments such as oral presentations in real time.

Involve students in understanding assessment criteria

Consider asking students to review each other’s work using developed assessment criteria and/or a marking rubric to help them to internalise standards. This can help students to understand those criteria and develop evaluative judgement about their own performance.

Practical tips

Align feedback assessment criteria and performance descriptors so that students can relate comments back to these
Make sure that your Part B course guide accurately describes assessment and related administrative matters such as extensions, special consideration and penalties for late submission
Clearly define the group component of group assignments (see clause 1.20 RMIT Assessment Processes)
Include a clearly defined process for when a group becomes dysfunctional so that academic staff can intervene before assessment submission (see clause 1.18 of RMIT Assessment Processes).

Guiding policy

External Policy

Higher Education Standards Framework Threshold Standards 2015 (Cth) (HESF)

Consistency and validation of assessment (HESF standard 1.4.3)

Requirement for consistency between assessment and demonstration of learning outcomes and validation of student attainment and grades awarded.

Monitoring and review (HESF standard 5.3)

Requirements around monitoring, review and improvement of courses, including assessment moderation processes, such as comprehensive reviews of assessment methods and students’ achievement of learning outcomes (at least every 7 years); interim monitoring of student progress (grade distribution); and regular peer review of student success against comparable programs of study from another provider (student progress, assessment methods, grading with cohort analysis) to ‘calibrate’ assessors’ grading.

RMIT Policy

The RMIT Assessment processes (clauses 3.1-3.130) (clauses 3.1-3.30) set out practices for moderating assessment during the teaching period, such as specification of a marking scheme, allocation of a single marker (and a moderator) and double marking of assessment where there are new markers in a course. To ensure consistency, cross marking and review of assessors’ marking may occur.

Consensus moderation (clause 3.1)

Before results are finalised, Course Assessment Committees (CACs) moderate assessment by checking grade distribution and results across different offerings of the same course.

Reporting to CAC (clause 3.25.6)

Course coordinators report to the CAC on any issues arising with assessment, reasons for adjustments made to marks, and any other matter relevant to approval of final results Adjustments need to be documented, with reasons included. All narrow fails (45-49) must be double marked (clause 3.7).

CAC moderation (clause 3.23)

may lead to adjustment of student results to overcome disparities in the difficulty of assessments and/or severity of marking
adjustments must be made before the marks for the assessment are communicated to students where possible (clause 3.23.1).

Program assessment boards – PABs (clause 3.13)

PABs consider CAC recommendations on student or program cohort performance, how effective program assessment is and whether it needs enhancement.

Equivalence and comparability

RMIT program and course review process includes provisions for assuring equivalence and comparability of courses offered across multiple locations.

Section 24 of the Program and Course Policy states requirements for RMIT programs delivered by partners.

Further information and support

DSC Academic Development Group

DSC Quality Team

DSC Academic Services

Selected references

Beutel, D., Adie, L., & Lloyd, M. (2017). Assessment moderation in an Australian context: Processes, practices, and challenges. Teaching in Higher Education, 22(1), 1-14.

Bloxham, S., Hughes, C., & Adie, L. (2016). What's the point of moderation? A discussion of the purposes achieved through contemporary moderation practices. Assessment & Evaluation in Higher Education, 41(4), 638-653.

Clarke, A., Johal, T., Sharp, K., & Quinn, S. (2016). Achieving equivalence: A transnational curriculum design framework. International Journal for Academic Development, 21(4), 364-376.

Higher Education Standards Framework Threshold Standards 2015 (Cth) (Standard 5.3) – Download from https://www.legislation.gov.au/Details/F2015L01639)

Krause, K., Scott, G., Aubin, K., Alexander, H., Angelo, T., Campbell, S., Carroll, M., Deane, E., Nulty, D., Pattison, P., Probert, B., Sachs, J., Solomonides, I., Vaughan, S. (2013). Assuring final year subject and program achievement standards through inter-university peer review and moderation.

https://www.ed.ac.uk/institute-academic-development/learning-teaching/staff/assessment/moderation-guidance

https://www.iru.edu.au/iru-work/calibration/

https://lo.unisa.edu.au/course/view.php?id=8539

RMIT program and course review processes

TEQSA guidance note on academic quality

TEQSA external referencing guidance note