'Right' program level outcomes first
Assure the quality of program level outcomes & graduate capabilities as a first step in course design and review
‘ Universities determine their standards of education having regard to a range of expectations, including the capabilities they seek in their graduates, and their own professional academic expectations and understandings of good quality. They make reference to external standards as guides to their decision-making. These references may include: the national qualifications framework and descriptors of learning outcomes; statements issued by professional bodies relating to program requirements for graduates preparing to practise in registered professional occupations; statements issued by disciplinary communities; standards set by similar universities elsewhere; findings from surveys of students, graduates and employers; and innovative approaches being undertaken elsewhere’.
(Gallagher, M (2010: 173): The accountability for quality agenda in higher education, Group of Eight, Canberra).
What are the key reference points and sources of information we should consider when seeking to validate our program level outcomes and develop graduates who are work ready plus?
Are we agreed on the relative weight to be given to these reference points and sources of information?
How do we know the program level outcomes we have identified are relevant?
Have we taken into account all five dimensions of the professional and graduate capability framework (see Getting Started Section 3.2) along with their subscales and specific items when articulating our Program Level Outcomes?
In Box Three a strategy for addressing these questions efficiently and productively which was developed and tested during the Fellowship workshops is summarised:
Suggested strategies for developing valid program level outcomes
Whenever your program team is developing or reviewing a degree, diploma or certificate use a peer review process similar to that carried out to assure the quality of research as you seek to determine and review the relevance and desirability of the learning outcomes and graduate capabilities to be developed and assessed in the program.
Fellowship participants suggest that to do this efficiently and effectively each program team undertakes steps like the following:
Box Four provides a list of the potentially relevant reference points/sources of validation information discussed and tested in the fellowship workshops which you may find useful as you seek to confirm that your program level outcomes are relevant and desirable.
Potentially relevant reference points when seeking to validate PLOs developed and tested at the Fellowship workshop
The relevance and ease of use of the process suggested in Box Three and the reference points in Box Four was beta tested during the Fellowship workshops and found to be workable and helpful by teams of participants from the following areas of higher education:
Arts & Humanities, Accounting, Business, Business management, Business Administration, Marketing, Sports Coaching, Engineering, Engineering & IT, Information Systems, IT and Business, Journalism, Media, Law, Medicine, Nursing, Nursing and marketing, Public Heath, to Heath Promotion, Physiotherapy, Theology, Ministry, Teacher Education, Higher Education, Higher Education Transition, Psychology, Clinical Psychology, Social Science, General Science, Biological Science, Chemistry, Agricultural Science, Environmental sciences, Work safety.
The specific suggestions to program teams on getting the most out of the processes outlined in Box Three and Four that were consistently made during the workshops are summarised below:
- Use peer review by team members (Box Three ) against multiple, appropriately weighted, reference points (Box Four ) with a view to producing work ready plus graduates.
- Accommodate the results into the professional and graduate capability framework and subscales outlined in Figure Two.
- Ensure this takes place at the start of the course development or review process (this, said participants, is currently not always consistently or well carried out).
- Consider using external peer review by teams teaching the same program elsewhere (building on the key lessons learnt in the OLT inter-university moderation and calibration projects ) to confirm the veracity with which multiple reference points have been used by your program team and how well they have been incorporated into the graduate and professional capability framework. This was seen as being a key way to ensure sector diversity whilst assuring standards and avoiding the imposition of a ‘one size fits’ all outcomes test. This, said participants, is an ideal way to assure standards whilst avoiding standardization.
It was noted that this approach aligns with the requirements for external referencing not only by TEQSA but also that recommended in the Australian higher education L&T standards and similar requirements internationally.
- Make more systematic use of studies of successful early career graduates and feedback from employers using all of the capability items in Tables 1-3 as an additional reference point to ensure the relevance of program level outcomes (see the successful graduate studies in the References section of the site).
- Review each institution’s course development and review policies and procedures to ensure that they align with and support this focus. It was suggested that this might best be supported by an online course development and review process which is shaped around the ‘six keys’ framework and the checkpoints in all four dimensions of overall quality and standards framework for Learning and Teaching outlined in Figure One (See Using the Guide & Getting Started Section 3.1).
- Distinguish between using external peer review to confirm, on the one hand, the veracity and comprehensiveness with which program level outcomes have been validated and, on the other hand, its use as an assessment grading and moderation tool. Participants at the workshops noted that the former uses peer review to confirm the quality of the inputs to assessment whereas the latter is about confirming the quality and consistency with which student assessment products have been graded. Many current university peer review systems are, say participants, primarily focused on the latter.
- Consider clustering the threshold learning outcomes identified in the ALTC/OLT LTAS project into the domains and subscales in Figure 2 (See Using the Guide & Getting Started Section 3.2). It was suggested that the same process might also be undertaken in relation to the learning outcomes set down by professional accreditation agencies and that there may be room to work with such agencies on ensuring these outcomes are validated using the guidelines in Boxes 1 & 2.
- An important ‘sleeper’ issue is to confirm the extent to which the capabilities sought are culturally based.
When reviewing the relevance of your program level outcomes, ask to what extent have we:
- Taken into account all of the reference points identified in Box Four (or any additional ones)?
- All agreed on the relative weight we are giving to each of these when deciding what is most relevant, desirable and feasible for focus in these Program level outcomes and the capabilities they will be measuring in our graduates?
- Taken into account all five dimensions of the professional and graduate capability framework (Using the Guide & Getting Started Section 3.2) along with their subscales and specific items when articulating our PLOs?
Resources and further guidelines on effective approaches to assuring the quality and relevance of program level outcomes and graduate capabilities
- Romy Lawson's Assuring Learning website at: http://www.assuringlearning.com/
This site, developed as part of Romy Lawson's OLT Fellowship on the area, has a wide range of practical tips and resources on mapping graduate attributes in higher education and leadership strategies for engaging staff in these processes. It includes practical resources covering writing and embedding course (program) level outcomes, constructing whole of course rubrics, designing course level outcome assessments, productive learning activities and leading the way along with quality enhancement resources and a curriculum design workbench (tool).
- Noyd, R et al (n.d.): A Primer on Writing Effective Learning-Centered Course Goals. This US primer builds on Dee Fink’s on framing program level outcomes work of Dee Fink and outlines a practical way in which to ensure course designs are integrated and outcomes’ focused.
- Tina Acuna (2013) OLT-funded project ‘A consensus approach to defining standards for learning outcomes and informing curricula design for agriculture’, UTAS, Dr Tina Acuna
- Simon Barrie 2014: Assessing and assuring Australian Graduate learning outcomes and AAGLO website
- Freeman, M & Ewan, C (2014): Good Practice Report: Assuring learning outcomes and standards, OLT, Sydney
See for example Pgs 13-15 where the requirements and suggestions for external referencing and a course level focus from TEQSA and the HE Standards Panel are discussed along with outcomes of the Discipline Scholars Networks funded by ALTC to undertake the Learning and Teaching Academic Standards Project along with other initiatives focused on assuring the quality of course (program) level outcomes (pgs 16ff).
- Hanover report Engaging Employers in the Assessment Process, 11th October 2014: enacting strategies for graduate employability.(subscriber password required)
- Beverley Oliver’s Assuring Graduate Capabilities site covers how to:
Specify Course Learning Outcomes drawing on institutional goals, professional accreditation requirements and disciplinary threshold standards
Engage students in authentic experiences to progressively develop graduate employability
Assess authentic tasks, encouraging students to create artefacts that demonstrate employability
Evidence, curate and communicate artefacts to potential employers and clients
Credential warranted achievement of capabilities in ways that are trustworthy and meaningful
Enhance courses - maintaining strengths and addressing gaps - focused on evidence of capability achievement.
- The National Qualifications Frameworks used in various jurisdictions around the world
- TEQSA HE Standards Framework
- Australian Qualifications Framework
- NZAQANZ QF
- US NILOA Degree qualification profile (Lumina)
- WASC 2013: WASC handbook for accreditation
- South African Quality Framework
- UK Quality code for Higher Education
- OLT reports/commissioned reports and resources on assuring the quality of program level outcomes. See, for example, the overview of relevant commissioned OLT projects at: including: :
- Mark Freeman: Assuring learning outcomes & standards;
- Shelley Kinash: Supporting Graduate employability through employer and private institution collaboration;
- Margaret Jollands: Graduate employability through partnerships with industry and professional associations;
- Dawn Bennett: Enacting strategies for graduate employability;
- Beverley Oliver: Curate, credential and carry forward digital learning evidence;
- Pierre Benckendorff: enhancing student learning outcomes with simulation-based pedagogies
- Krause, K, Scott, G et al (2012): Assuring learning and teaching standards through inter-institutional peer review and moderation: a user guide and handbook OLT, Sydney
- Mohammad Rasul (2012) OLT-funded project ‘Assessing final year engineering projects (FYEPs): ensuring learning and teaching standards and AQF8 outcomes’, CQUniversity.
- Higher Education Quality Council of Ontario (2015): Learning outcomes assessment: a practitioner’s handbook, HEQC, Toronto, Canada. Sections 1.2 & 1.3.