‘Assuring the quality of achievement standards and their valid assessment in Australian higher education’

"The starting point for any discussion of the quality of higher education is the quality and relevance of its purposes. Assessing the quality of higher education according to the extent to which it achieves its purposes – i.e. assessing its fitness for purpose without assessing fitness of purpose - equates quality with efficiency and is therefore of limited value."

Stephenson, J (1992: 2)

Higher education students graduate into a volatile and rapidly transforming world in which globalisation, ‘digital disruption’ and the challenges of social, cultural, economic and environmental sustainability have created a powerful change cocktail. Graduates now practice in a context of continuous flux, where technical and human factors constantly interact in complex and unique ways; where unpredictability and change are always in the air and their capability is most tested when the unexpected happens, when an unanticipated opportunity arises, or when things suddenly go awry and they are faced with a ‘wicked problem’ or dilemma – a ‘forked road’ situation in which there is a range of potentially relevant ways to go and they have to decide which is likely to be the most productive and then deliver it.

This guide aims to assist Australian higher education institutions to ensure that the quality of their graduates keeps pace with the rapidly changing needs and operating context of the 21st century. A particular focus of the Fellowship has been on developing the change-leadership capability of the key players in ensuring that this agenda is addressed in practical and productive ways at the local level - Associate Deans and Directors (Learning and Teaching) and Programme Directors and Heads. Over 2014-16 we have worked together to identify how best to develop their proficiency to identify, validate, develop and robustly assess relevant and desirable graduate capabilities, in order not only to assure the fitness for purpose of assessment but also its fitness of purpose; and we have worked together to identify the most effective ways to support this agenda into consistently effective daily practice.

Extensive, excellent work has been undertaken in earlier Office for Learning and Teaching (OLT)/Australian Learning and Teaching Council (ALTC) projects and fellowships on assuring the quality of assessment in Australian higher education, with particular attention being given to assuring the fitness for purpose of assessment, assessment integrity and the use of assessment for learning as well as of learning. The majority of these important initiatives have concentrated predominantly on how assessment might best be carried out and built into the total student learning experience. That is, they are mainly concerned with the processes of assessment, its use as a learning tool and its support.

However, less work has been undertaken to explore whether the outcomes set down for university learning and assessment are, in the first place, relevant and desirable, that is to determining:

• what assessment and learning in different fields of education should be giving focus to in the context of the rapidly changing needs of the 21st century; 
• whose voice and what reference-points and sources of advice should be given most/least attention when seeking to confirm that the capabilities and competencies to be developed by our university and college students are what is needed for productive professional performance and societal participation in the new, highly volatile global context;
• that the graduate capability and outcomes framework used to accommodate the input on what would be most relevant from these many sources is valid, comprehensive and easy for all the stakeholders to understand;
• what assessment tools are best suited to evaluating the graduate capabilities and competencies to be developed in an integrated, valid and scalable fashion;
• the optimum ways in which this agenda can be enacted – consistently, effectively and sustainably.

Ensuring that higher education institutions confirm the fitness of purpose of their desired program level outcomes and the capabilities to be developed and assessed in their graduates is critically important because it is these people who will make up the vast majority of our leaders, entrepreneurs, inventors and professional practitioners in the years to come. It will be they who will be tasked to ensure that the nation has a future which is socially, culturally, economically and environmentally sustainable.

The guide has been developed by and for higher education program development and review teams and those local Learning and Teaching leaders (including Heads of School, Associate Deans, Heads of Program and Directors of L&T) who assist them with improving the quality of achievement standards and assessment in our universities and colleges.

At the end of each of the dozens of workshops and conferences held around the world those attending were asked to identify what, for them, were the most significant and helpful issues and strategies explored during these events. It is to these areas of greatest relevance to the 3700 Fellowship participants that the guide gives focus. The recurring themes in this feedback include:

The importance of:

• Giving much more focus to assuring the quality of the outcomes of higher education not just to the quality of the teaching and learning inputs or to assuring only the reliability of assessment.
• Assuring the fitness of purpose of assessment not just its fitness for purpose and of distinguishing more explicitly between the two – it is no good to assess reliably or ensure comparable standards, said participants, if what we are assessing is irrelevant or undesirable.
• Identifying and classifying program-level outcomes using a comprehensive, proven graduate and professional capability framework. And doing this with a view to developing graduates who are not just work ready for today but also work ready plus for tomorrow (for example graduates who are sustainability literate, change implementation savvy, creative & inventive and who have come to a considered position on the tacit assumptions driving the 21st century agenda).
• Ensuring everyone is clear on the distinction between ‘capability’ and ‘competence’ when we profile the sorts of graduates we seek to produce from our higher education programs.
• Giving more focus to personal and interpersonal capabilities (‘mindfulness’) as well as to more specific development of cognitive capabilities like diagnosis, contingent thinking and invention rather than concentrating only on the development of set skills and knowledge (competence).
• Using evidence-based peer review of multiple, appropriately weighted reference points/sources of evidence as an ideal way to assure the validity, quality and relevance of what is assessed in our degrees and diplomas in order to avoid the imposition of a ‘one size fits all’ exit test.
• Confirming that everyone involved is using common terms like ‘program’, ‘course’, ‘subject’, ‘assessment’ and ‘evaluation’ with shared meaning.
• Giving more emphasis to dilemma-based assessment and learning in higher education and to helping students become more adept at managing the most common ‘wicked problems’ of early career practice (Rittel & Webber, 1973) in their profession or discipline.
• ‘Flipping’ not only the classroom but the curriculum development and review process itself – by starting with establishing and validating the Program Level Outcomes (PLOs) that count for effective early career/disciplinary practice and then ‘backward mapping’ to ensure we have scaffolded, aligned unit outcomes, valid assessment, reliable grading and aligned learning methods, content, support and resources to our desired program level outcomes in that order.
• Using an overall Quality Assurance (QA) framework for Learning and Teaching so that staff are able to see how all the elements fit together, where assessment fits into the bigger quality picture and the important part each of their roles plays as part of a broader, mutually reinforcing L&T quality and standards system.
• Adopting a systems thinking and alignment approach to implementation.
• Sharing situated tips and exemplars of good practice on how best to formulate valid program level outcomes, map them to units of study, assess and grade with integrity, calibrate graders and produce aligned learning strategies and resources that engage students in productive learning.
• Establishing a ‘one stop shop’, searchable clearing house which enables staff to quickly and precisely access proven approaches to these six key elements of a comprehensive assessment system specifically in the field(s) of education in which they teach.
• Ensuring clarity of expectations in assessment (for both staff and students) – and exploring the use of assessment-focused learning guides in each unit of study as one proven way of achieving this.
• Making the ‘why bother’ case for undertaking this work explicit and comprehensive; and then directly aligning key elements of this case for change to the particular motivators of different players.
• Producing this co-created local leaders’ self-teaching guide in order to consolidate, link, leverage and share what we are all doing individually in a systematic way; and doing this via the user-centred design and refinement process adopted in the Fellowship - with links to the multiple good practice websites on the area whenever ever possible.
• Emphasising that, as a local leader of change in this area and as the final arbiter of whether any desired change actually gets implemented the importance of keeping the following insights in mind;
• Good ideas with no ideas on how to implement them are wasted ideas;
• Change doesn’t just happen but must be led, and deftly;
• Effective leaders always listen, link, leverage then lead, in that order; and they model to their staff how best to respond when the unexpected happens, things go awry or an unanticipated opportunity for taking a new direction arises.
• Paying attention to the key lessons on effective change implementation and leadership in higher education and the importance of building a change capable culture by using, inter alia, careful selection of leaders who model how to do this with their staff; of seeing change as a learning and unlearning process for all concerned not as an event and of making sure that those who are to implement developments like those identified throughout the guide see what is being advocated as personally relevant, desirable, clear and feasible.

To engage all staff, not just the enthusiasts, with ways of improving the quality of program level achievement standards and assessment we need a robust case for change and, as noted above, for everyone who is to put the agenda into practice to see that what is proposed is, from their perspective, relevant, desirable, clear, endorsed, supported and feasible. For people to engage with change (i.e. learn how to do something new) the benefits of doing so need to outweigh the costs.

Below is a summary of the key reasons identified during the Fellowship for giving focus to seeking to improve and assure the quality of achievement standards and their assessment:

• Assessment drives learning (and teaching);
• The assessment domain in the CEQuery analysis of some 280,000 best aspect and needs improvement comments written on Australia’s Course Experience Questionnaire (Scott, 2006) has the lowest odds of a ‘best aspect’ comment;
• Valid, well managed and transparent assessment significantly decreases litigation and time-consuming student appeals’ processes regarding their assessment;
• It is no good to assess well if what we are assessing doesn’t count;
• Higher education is a key tool for assisting the nation to develop graduates capable of inventing the enterprises and new sources of income we need to replace the lost income and jobs that have resulted from the decline in our traditional manufacturing industries and the ‘resources boom’;
• Employer and graduate satisfaction with the capabilities builds demand;
• It is assessment that confirms universities are achieving their mission and ‘moral purpose’;
• A large percentage of the world’s political leaders are graduates (e.g. in the US the Congressional Research Service in its 2015 analysis pg 4 reports that 94% of House Members and 100% of Senators hold at least a Bachelor’s degree; in Australia it is 84%) and their higher education experience should assist them to clarify their position on the tacit assumptions driving the 21st agenda so that they make policy decisions that are transparently consistent with a stated value position;
• Internationally
• The shift is towards assuring the quality of the impact the extensive investment in Higher Education Learning and Teaching is having on graduates, not just assuring the quality of inputs or levels of student satisfaction;
• There is growing interest in developing graduates who are not only work ready for today but work ready plus for tomorrow ;
• There is strong support for the use of processes of evidence-based peer review, similar to those used to assure the quality of research, to determine relevant program level outcomes;
• The UN, as part of the outcomes of the Decade of Education for Sustainable Development 2005-14, is calling for universities and colleges to give particular focus to developing graduates who are sustainability literate in both their chosen profession and more generally;
• Many higher education quality codes (e.g. the UK HE Quality Code) now require the use of multiple reference points to confirm the fitness of purpose of program level outcomes;
• There is increased interest in giving focus to inventiveness, ethical entrepreneurialism, and well-developed personal and interpersonal capabilities (‘mindfulness’) as key graduate outcomes.
• The need to assure consistent Learning and Teaching quality and achievement standards whilst avoiding a ‘one size fits all’ compliance system;
• The importance of systems’ thinking, capacity building & alignment to improve quality.

The self-teaching guide has been collectively developed, tested for relevance and enhanced using:

• The feedback and practical suggestions provided from participants in 65 workshops and 21 keynote addresses held with some 3700 Learning and Teaching leaders around the world during 2014-16.
• Interviews with a wide range of key Higher Education leaders in Australia, NZ, the Pacific, Malaysia, Canada, USA, UK, and Europe.

The development approach used to produce the guide has been ‘user-centred’. This has involved an iterative process of identifying, testing and refining what is most relevant and directly helpful to the local leaders participating in the Fellowship workshops and keynotes. In doing this, particular focus has been given to bringing together proven, practical guidance and exemplars of good practice in assuring both the quality of our achievement standards in higher education and their assessment for review and further enhancement by those attending.

It is in this way that the guide has been ‘co-created’ and is firmly situated in the world of the local change leader and is ‘in the voice’ of the user. This approach has been supplemented by making reference to some 200 pages of the detailed feedback provided by participants after each workshop and in validation meetings with key figures in the countries involved.

This approach is an example of how to apply one of the key lessons identified in earlier OLT research on effective change leadership in higher education (Scott, Coates & Anderson, 2008): effective higher education leaders always listen first (with a case for change and a menu of options), link (what practitioners see as being most relevant and feasible for follow-up action), leverage successful practice (engage those further down the change path to assist those just starting out) and only after this do they then lead (assist the process of scale up and continuous, peer-supported learning and improvement).

The guide is structured around how best to ensure that we have both the right ‘what’ and the right ‘how’ of effective change leadership and implementation when seeking to assure the achievement standards and quality of assessment in our universities and colleges. It will, said participants, be the effective combination of both the right ‘what’ and right ‘how’ in this area that will be most telling if we are to ensure that our institutions of higher education and the graduates they produce are to remain relevant to the needs of the coming decades.

The two key lessons derived from 40 years’ research and experience in effective change management in post-secondary and higher education which underpin how the guide was developed and is structured are, therefore:

Good ideas with no ideas on how to implement them are wasted ideas
and
Change doesn’t just happen but must be led, and deftly

With these lessons in mind the guide gives particular attention to helping local Learning and Teaching leaders and their program teams ensure that both what they assess in our higher education programs is going to produce graduates who are ‘work ready plus’ (see Section 3.3) and also how this agenda is put into practice is effective, scalable and sustained.

First, there is a section on how to use the six ‘keys’ strategy for assuring the quality of achievement standards and assessment by ‘flipping’ the curriculum development and review process (see Six "Keys" to flipping the curriculum). This section addresses the ‘what’ of change in this area and the key question “How do we ensure that we achieve high quality achievement standards and powerful assessment?”

Second, there is a section called ‘Making it happen’ on how best to make sure any developments in this area are successfully, consistently and sustainably put into practice (implemented). This section addresses the ‘how’ of change and the key question “What do we do on Monday?” in order to take what we agree is a ‘good idea’ in this area and make sure it is actually implemented (put into practice) – both consistently and effectively.

The ‘Using the guide and getting started’ section brings together the overall tips from Fellowship participants on how to get prepared for using the guide. They suggest you read through and discuss this preparatory section before moving on to access the two major sections that make up the site.

Whenever possible, as we have just seen above, links are given throughout the guide to practical examples and to the findings of parallel projects on the area being discussed.

The guide is complemented by a broader Key Insights discussion paper on the key insights generated during the Fellowship in both its workshops and in the meetings with key higher education players from around the world. That paper sets the broader context for the practical implementation suggestions outlined in the guide.

A video produced for the 2015 HERDSA conference on the Fellowship that discusses the key points made in the guide is available.