'Right' program level outcomes first

Assure the quality of program level outcomes & graduate capabilities as a first step in course design and review

‘ Universities determine their standards of education having regard to a range of expectations, including the capabilities they seek in their graduates, and their own professional academic expectations and understandings of good quality. They make reference to external standards as guides to their decision-making. These references may include: the national qualifications framework and descriptors of learning outcomes; statements issued by professional bodies relating to program requirements for graduates preparing to practise in registered professional occupations; statements issued by disciplinary communities; standards set by similar universities elsewhere; findings from surveys of students, graduates and employers; and innovative approaches being undertaken elsewhere’.

(Gallagher, M (2010: 173): The accountability for quality agenda in higher education, Group of Eight, Canberra).

  1. What are the key reference points and sources of information we should consider when seeking to validate our program level outcomes and develop graduates who are work ready plus?

  2. Are we agreed on the relative weight to be given to these reference points and sources of information?

  3. How do we know the program level outcomes we have identified are relevant?

  4. Have we taken into account all five dimensions of the professional and graduate capability framework (see Getting Started Section 3.2) along with their subscales and specific items when articulating our Program Level Outcomes?

In Box Three a strategy for addressing these questions efficiently and productively which was developed and tested during the Fellowship workshops is summarised:

Box Three

Suggested strategies for developing valid program level outcomes

Whenever your program team is developing or reviewing a degree, diploma or certificate use a peer review process similar to that carried out to assure the quality of research as you seek to determine and review the relevance and desirability of the learning outcomes and graduate capabilities to be developed and assessed in the program.

Fellowship participants suggest that to do this efficiently and effectively each program team undertakes steps like the following:

  • Consider all of the reference points listed in Box Four below, adding others which your team has identified as relevant when necessary;
  • Come to a judgement on which reference points/sources in Box Four are to be given most or least weight in the context of your particular university, location and the profession/discipline concerned, noting why you came to this judgement;
  • Use the professional and graduate capability framework’s subscales and items (See Using the Guide & Getting Started Section 3.2) to ensure comprehensive coverage as different reference points and sources of advice are consulted on what graduate capabilities are most relevant, desirable and important for development in the program;
  • Formulate program level outcomes based on this, locating them into the framework and subscales in Figure Two (See Using the Guide & Getting Started Section 3.2) and keeping in mind the need to focus on developing graduates who are work ready plus (See Using the Guide & Getting Started Section 3.3);
  • If desired, ask a team teaching the same program in a partner institution to validate the quality of your evidence-based peer review process in formulating and validating your program level outcomes. In doing this ask them to check the comprehensiveness of coverage, the variety of reference points and sources on input used and the arguments provided for their different weighting;
  • The outcomes of this external peer review can be used as evidence of benchmarking and validation for any external auditing/ accreditation agencies .

 

Box Four provides a list of the potentially relevant reference points/sources of validation information discussed and tested in the fellowship workshops which you may find useful as you seek to confirm that your program level outcomes are relevant and desirable.

 

Box Four

Potentially relevant reference points when seeking to validate PLOs developed and tested at the Fellowship workshop

  • National Qualifications Framework or equivalent;
  • The University’s mission & its graduate attributes;
  • Learning outcome standards determined by ALTC discipline groups, UK subject benchmarks; UK Quality Code; the Australian HE L&T standards;
  • Suggested program level outcomes from the Assessment of HE Learning Outcomes project, US bodies like the Western Association of Schools & Colleges, the US National Institution for Learning Outcomes Assessment, the UN Decade of Education for Sustainable Development, the Business Council of Australia etc;
  • The learning outcomes for courses of the same name in other places;
  • The requirements for post-graduate study in the discipline/profession concerned;
  • External professional accreditation standards and requirements (when applicable);
  • Results from inter-institutional benchmarking, peer review;
  • The work-ready plus capabilities including graduates being sustainability literate, inventive, change implementation savvy and having come to a considered position on the tacit assumptions driving the 21st century agenda;
  • Academic experts’ input, inter-institutional peer review and moderation;
  • Key capabilities and future trends identified by successful early career graduates/alumni/entrepreneurs/and the capabilities sought in job advertisements;
  • Employer feedback; input from External Course Advisory Committees;
  • The results of School/Department Reviews;
  • Government policy and funding incentives;
  • What peak industry, social and scientific bodies are calling for;
  • Regional development priorities and opportunities;
  • What parents, prospective students & others say they want
  • Plus?

 

The relevance and ease of use of the process suggested in Box Three and the reference points in Box Four was beta tested during the Fellowship workshops and found to be workable and helpful by teams of participants from the following areas of higher education:

Arts & Humanities, Accounting, Business, Business management, Business Administration, Marketing, Sports Coaching, Engineering, Engineering & IT, Information Systems, IT and Business, Journalism, Media, Law, Medicine, Nursing, Nursing and marketing, Public Heath, to Heath Promotion, Physiotherapy, Theology, Ministry, Teacher Education, Higher Education, Higher Education Transition, Psychology, Clinical Psychology, Social Science, General Science, Biological Science, Chemistry, Agricultural Science, Environmental sciences, Work safety.

The specific suggestions to program teams on getting the most out of the processes outlined in Box Three and Four that were consistently made during the workshops are summarised below:

  • Use peer review by team members (Box Three ) against multiple, appropriately weighted, reference points (Box Four ) with a view to producing work ready plus graduates.
  • Accommodate the results into the professional and graduate capability framework and subscales outlined in Figure Two.
  • Ensure this takes place at the start of the course development or review process (this, said participants, is currently not always consistently or well carried out).
  • Consider using external peer review by teams teaching the same program elsewhere (building on the key lessons learnt in the OLT inter-university moderation and calibration projects ) to confirm the veracity with which multiple reference points have been used by your program team and how well they have been incorporated into the graduate and professional capability framework. This was seen as being a key way to ensure sector diversity whilst assuring standards and avoiding the imposition of a ‘one size fits’ all outcomes test. This, said participants, is an ideal way to assure standards whilst avoiding standardization.
    It was noted that this approach aligns with the requirements for external referencing not only by TEQSA but also that recommended in the Australian higher education L&T standards and similar requirements internationally.
  • Make more systematic use of studies of successful early career graduates and feedback from employers using all of the capability items in Tables 1-3 as an additional reference point to ensure the relevance of program level outcomes (see the successful graduate studies in the References section of the site).
  • Review each institution’s course development and review policies and procedures to ensure that they align with and support this focus. It was suggested that this might best be supported by an online course development and review process which is shaped around the ‘six keys’ framework and the checkpoints in all four dimensions of overall quality and standards framework for Learning and Teaching outlined in Figure One (See Using the Guide & Getting Started Section 3.1).
  • Distinguish between using external peer review to confirm, on the one hand, the veracity and comprehensiveness with which program level outcomes have been validated and, on the other hand, its use as an assessment grading and moderation tool. Participants at the workshops noted that the former uses peer review to confirm the quality of the inputs to assessment whereas the latter is about confirming the quality and consistency with which student assessment products have been graded. Many current university peer review systems are, say participants, primarily focused on the latter.
  • Consider clustering the threshold learning outcomes identified in the ALTC/OLT LTAS project into the domains and subscales in Figure 2 (See Using the Guide & Getting Started Section 3.2). It was suggested that the same process might also be undertaken in relation to the learning outcomes set down by professional accreditation agencies and that there may be room to work with such agencies on ensuring these outcomes are validated using the guidelines in Boxes 1 & 2.
  • An important ‘sleeper’ issue is to confirm the extent to which the capabilities sought are culturally based.

When reviewing the relevance of your program level outcomes, ask to what extent have we:

  1. Taken into account all of the reference points identified in Box Four (or any additional ones)?
  2. All agreed on the relative weight we are giving to each of these when deciding what is most relevant, desirable and feasible for focus in these Program level outcomes and the capabilities they will be measuring in our graduates?
  3. Taken into account all five dimensions of the professional and graduate capability framework (Using the Guide & Getting Started Section 3.2) along with their subscales and specific items when articulating our PLOs?

Resources and further guidelines on effective approaches to assuring the quality and relevance of program level outcomes and graduate capabilities

  • Romy Lawson's Assuring Learning website at: http://www.assuringlearning.com/
    This site, developed as part of Romy Lawson's OLT Fellowship on the area, has a wide range of practical tips and resources on mapping graduate attributes in higher education and leadership strategies for engaging staff in these processes. It includes practical resources covering writing and embedding course (program) level outcomes, constructing whole of course rubrics, designing course level outcome assessments, productive learning activities and leading the way along with quality enhancement resources and a curriculum design workbench (tool).

back to six keys

Submit an example