Skip navigation

Handbook for enhancement-led institutional review: Scotland

QAA 033 04/2003

Annex 2: Suggested outline for the full report

1 Introduction

  • Standard statement about the purpose and conduct of the ELIR process, including the scope of the review.
  • Background information about the institution.
  • Statement of the institution's strategy for quality enhancement.

2 Internal monitoring and review of quality and standards and public information

Overview of:

  • the institution's internal arrangements for assuring the quality of programmes and maintaining the standards of its academic awards and credit;
  • the use made of external reference points for assuring quality and standards including: the academic infrastructure (including reports of SCQF in particular), external examiners and PSBs;
  • the institution's approach to ensuring that the information it publishes about the quality of its provision is complete, accurate and fair.

Commentary on:

  • the ability of the institution's internal review systems to monitor and maintain quality and standards at the level of the programme or award;
  • the institution's arrangements for ensuring that the information it publishes about the quality of its provision is complete, accurate and fair.

3 The student experience

Overview of:

  • the institution's approach to engaging students in the assurance and enhancement of the quality of teaching and learning;
  • the institution's approach to the promotion of effective student learning, and to providing an appropriate learning experience for all its students;
  • the institution's approach to the promotion of the employability of its students.

Commentary on:

  • the effectiveness of the institution's approach to promoting an effective learning experience for students.

4 Effectiveness of the institution's strategy for quality enhancement

Overview of:

  • the institution's approach to managing improvement in the quality of teaching and learning;
  • the linkage between the institution's arrangements for internal quality assurance and its enhancement activity;
  • the institution's approach to recognising, rewarding and implementing good practice in the context of its strategy for quality enhancement.

Commentary on:

  • the combined effect of the institution's policies and practices for ensuring improvement in the quality of teaching and learning;
  • the effectiveness of the institution's implementation of its strategy for quality enhancement.

5 Summary

  • Overview of the matters raised in the review.
  • Commentaries on:

i the ability of the institution's internal review systems to monitor and maintain quality and standards at the level of the programme or award. This commentary leads to a judgement on the level of confidence which may be placed in the institution's management of quality and standards;

ii the institution's arrangements for ensuring that the information it publishes about the quality of its provision is complete, accurate and fair;

iii the effectiveness of the institution's approach to promoting an effective learning experience for students;

iv the combined effect of the institution's policies and practices for ensuring improvement in the quality of teaching and learning;

v the effectiveness of the institution's implementation of its strategy for quality enhancement.

Annex 3: Criteria for the selection of reviewers and secretaries

General

1 Reviewers, including student reviewers, and review secretaries will be selected by the Agency on the basis of the criteria set out below, and, with the exception of student reviewers, generally from nominations made by institutions. Student reviewers will be nominated by student representative bodies. All reviewers and secretaries will be provided with induction and training to ensure that they are familiar with the aims, objectives and procedures of the ELIR process and their own roles within it.

2 The qualities required in reviewers and review secretaries are outlined below. Student reviewers require to have current or recent direct experience of a Scottish higher education institution. Other reviewers and review secretaries will be drawn from across the UK. Every attempt will be made to ensure that the cohorts of reviewers and review secretaries reflect appropriate sectoral, geographical, gender and ethnic balances.

3 Neither reviewers nor review secretaries will be appointed to teams reviewing their own institutions, or any other institution with which they have a conflict of interest.

4 Training for reviewers and review secretaries is provided by the Agency. The purpose of the training is to ensure that all:

  • understand the aims and objectives of the ELIR process;
  • are fully familiar with the procedures involved;
  • understand their own roles and tasks, the importance of team coherence, the Agency's expectations of them, and the rules of conduct governing the process;
  • have an opportunity to explore and practise the techniques of data assimilation and analysis, the development of programmes for visits, the construction and testing of hypothesis, the formation of commentaries, conclusions and judgements and statements of confidence, and the preparation of reports.

Qualities required in all reviewers

5 All reviewers will be able to demonstrate the ability to:

  • understand a range of perspectives, to relate to a range of individuals including students and senior managers, and to hold discussions at a high level about strategic and operational approaches to the management of quality and standards in general and the enhancement of the student learning experience in particular;
  • assimilate a large amount of disparate information; to analyse and draw reliable conclusions about complex arrangements; and to undertake research and investigation into documentary and oral information in order to arrive at discursive conclusions and to form evidence-based commentaries, conclusions and judgements;
  • communicate clearly, orally and in writing;
  • work productively and co-operatively in small teams delivering to tight deadlines;
  • maintain the confidentiality of sensitive matters.

Additional qualities required in reviewers other than students

6 All reviewers, other than students, will be able, in addition, to demonstrate:

  • current or recent (within three years), wide experience of academic management, quality assurance and enhancement at the institutional level in the UK;
  • personal and professional credibility with staff including senior managers and heads of institutions in the higher education sector;
  • knowledge and understanding of the Agency's standards infrastructure, including the Code of practice, SCQF, and the subject benchmark information;
  • awareness of the distinctive features of the Scottish HE system in general and the enhancement-led approach of SHEFC in particular, including the SHEFC and US vision for the sector in Scotland. It should be noted that the training will build on this.

In addition to the qualities outlined above, the Agency will be interested in reviewers identifying other relevant experience that they may have, for example: experience of good practice in quality assurance and enhancement in an international setting or experience of institutional mechanisms to manage employer and industrial links.

Additional qualities required in student reviewers

7 All student reviewers will be able, in addition, to demonstrate:

  • current or recent (within three years), experience within a higher education institution in Scotland amounting to a minimum of one year's full time study (or its equivalent). The Agency is keen to receive applications from students representing the broad spectrum of higher education students in Scotland, including: part-time students; those who have not entered higher education institutions direct from school; as well as those studying in 'traditional' full-time mode;
  • experience of representing students' interests at institutional (including faculty or school) level;
  • general awareness of the diversity of the Scottish higher education sector, beyond their 'home' institution and awareness of the arrangements for quality assurance and enhancement in Scotland. It should be noted that the training will build on this and, in the first instance, the Agency is primarily looking for applicants who have the ability to build on experience they already have (which may have been gained from a variety of sources) and to assimilate new information that will be provided during induction and training.

In addition to the qualities outlined above, the Agency will be interested in student reviewers identifying their other relevant experience.

Qualities required in review secretaries

8 Review secretaries will be able to demonstrate:

  • current or recent (within three years) experience of senior academic administration at institutional (including faculty or school) level in UK higher education;
  • wide experience of working with senior committees in UK higher education;
  • awareness of the distinctive features of the Scottish HE system in general and the enhancement-led approach of SHEFC in particular, including the SHEFC and US vision for the sector in Scotland. It should be noted that the training will build on this;
  • ability to assimilate a large amount of disparate information, and to analyse and make reliable judgements about complex arrangements;
  • ability to keep a reliable record of discussions; to summarise the key outcomes; and to draft notes in a specified format to set deadlines;
  • ability to work productively and co-operatively in small teams delivering to tight deadlines;
  • ability to maintain the confidentiality of sensitive matters.

Annex 4: The Agency's operational principles and process standards (applies across the UK)

1 The Agency's approach to undertaking institutional reviews draws upon the practices and process standards developed and enhanced by its predecessor bodies. Since those bodies began their work, good practice in reviewing (guided by published standards of reviewing/auditing practice) and requirements relating to accountability and reporting, have developed considerably. The Agency recognises that some of the process standards it has observed in the past have been implicit rather than explicit, and that the ELIR process should be underpinned by a more explicit statement on operational principles and process standards.

2 In developing its operational principles and process standards, the Agency has taken note of the principles underpinning the AA1000 series accountability standard and the Seven Principles of Public Life developed by the Nolan Committee.

Principles

3 The Agency seeks to observe and promote several general principles within both the strategic and operational levels of its work. The principles are:

  • Inclusiveness - taking into account the needs of all stakeholder groups and facilitating their participation in aspects of the Agency's work.
  • Openness - transparency in the work and methods of the Agency, to build trust and confidence among stakeholders, and to provide information about the Agency's work to the wider public.
  • Accountability - demonstrating that the Agency is using its resources to good effect and with probity; conducting its work with integrity and impartiality; and ensuring that stakeholders are able to depend on the information provided.
  • Timeliness - the need for regular, systematic and timely action in all reporting processes to support the decision-making of the Agency and its stakeholders.
  • Comparability - using experience drawn from within the Agency and other organisations as a means with which to inform future work.
  • Relevance - ensuring that the information provided by the Agency is useful to, and understood by, all stakeholders.

4 These principles have been used to develop explicit service standards for ELIR, the details of which will be published on the Agency's web site.

Quality assurance mechanisms

5 The Agency is committed to the regular monitoring and evaluation of its policies, procedures and processes, to ensure their ongoing credibility and to continuously improve its performance in response to the results. In respect of institutional audit, this commitment includes providing the opportunity for participants in the process, including students, to provide structured feedback on their experiences.

Annex 5: SHEFC Guidance to institutions on the characteristics of internal (subject) review

Introduction

The aim of this document is to provide guidance to institutions on the process of internal review, in order to ensure that:

  • internal review is consistent with the principles described in SHEFC's consultation document on quality;
  • there is clarity throughout the sector on institutions' scope for flexibility in the design and operation of internal review; and
  • institutional audit can engage fully and consistently with the evidence generated by internal reviews, with no clash of expectations over the nature of internal review processes.

This document has been considered and agreed by the members of SHEFC's Quality Working Group.

Internal review is a key component of SHEFC's new enhancement-led approach to quality, which has been derived in partnership with the sector, the QAA Scottish Office, and the student body in Scotland. This guidance should be interpreted within the broader context of this new approach and the full range of activities which it encompasses.

Characteristics of internal review

In its consultation HEC 02/2002, issued on 28 February 2002, the SHEFC identified a range of characteristics of effective internal reviews. We believe it may be helpful to expand on these characteristics.

All provision should be reviewed on a cycle of not more than 6 years

It is a matter for each institution to decide how to schedule and aggregate its provision. However, good practice would be to ensure that programmes and subjects are aggregated in ways which provide coherence (eg reviewing all programmes in a subject at the same time; reviewing all programmes within a department at the same time). Excessive aggregation would mean that the process cannot examine the 'fine structure' of provision and may not be able to identify specific issues affecting a small number of programmes; large groupings may also become unwieldy if they involve too many members of staff. We suggest that the typical level of aggregation should be at department level.

Although the primary focus of internal review is likely to be on undergraduate degree programmes, we expect institutions to include within the review programme all taught provision in the subject area(s) being reviewed, including postgraduate awards, CPD, collaborative and overseas provision, supervision of research students, online and distance learning and provision (such as extra-mural courses) which provide only small amounts of credit. It will be a matter for institutions to decide how to aggregate such provision (eg by subject, mode of delivery, or level).

The timescale of six years was intended to provide continuity with the schedule which had been set in 1999. However, institutions may now wish to take advantage of the greater flexibility now available for aggregation and this may lead to a truncation of the cycle length. It would not be good practice to compress reviews into (say) a period of three years, followed by three further years of inactivity. We expect that internal review will inevitably identify a range of developmental issues and there is benefit to the institution from generating such insights on an ongoing basis. We therefore propose that, however the timetable is constructed, there should normally be some form of internal review activity taking place within each academic session.

Reviews should take full account of benchmarks and the code of practice

Benchmark statements provide a useful guide to national expectations about the characteristics of programmes in different subjects. We accept that benchmarks are less helpful in considering curricula and learning outcomes in interdisciplinary programmes and in modular structures which offer wide choice between options. Internal review processes should be designed in such a way as to establish that providers have engaged with relevant benchmarks and are able to demonstrate that programme design and learning outcomes are consistent with relevant benchmarks.

The QAA Code of practice contains a section on programme approval, monitoring and review. Institutional auditors will look for evidence that this Code has been embedded in institutional systems. More generally, the Code contains helpful guidance on a wide range of institutional functions, and internal review processes should be designed in such a way as to effectively monitor the implementation of the Code at the programme or subject level.

Significant amounts of provision in Scottish HEIs is accredited by professional and statutory bodies (PSBs). We would encourage institutions to engage with PSBs to explore appropriate ways in which the burden of audit might be further reduced, for example through the use of common documentation, or through joint processes which meet the needs of both internal review and external accreditation. We would also look to institutions to reflect on the outcomes of relevant PSB accreditations within internal reviews.

At a wider level, we would encourage institutions to take full advantage of activities (such as SHEFC's new quality enhancement engagements) which provide opportunities for reflection on their performance and how it might be usefully compared with that of others.

Reviews should take full account of the Scottish Credit and Qualifications Framework

The Scottish Credit and Qualifications Framework (SCQF) has been identified as a key strategic element in Scotland's education system, and the HE sector has been highly influential in its development. We look to institutions to actively engage with SCQF. SHEFC has stated its intention to move to a credit-based measurement system, with a view to later consideration of credit-based funding.

Internal review should be designed in order to promote scrutiny and discussion of the provider's approach to SCQF, with an expectation that providers will have adopted a proactive approach aimed at exploiting the flexibility which SCQF provides. This should include consideration of:

  • strategy for recognition of prior learning, eg through articulation arrangements with FE providers and/or general statements about criteria for entry with advanced standing;
  • consistency in the allocation of credit and level;
  • approach to credit-rating of non-standard learning components, eg work placement; and
  • flexible pathways to awards.

Reviews should provide an objective review of provision based on an understanding of national and international good practice: each review team should include at least one external member with a relevant background

It will be a matter for institutions to determine the composition of review teams and hence to select appropriate external member(s). Institutions will wish to select members who are able to appreciate the specific context in which programmes are presented. However, good practice would be to ensure that review teams are able to bring a range of experience to the process and hence are able to act as 'critical friends' to the institution. Team size and composition must also take account of the range and volume of provision to be reviewed. We suggest that institutions should consider more extensive use of external members, eg one from another Scottish institution and at least one from outside Scotland.

We do not expect internal review teams to routinely include members from outside the UK (although, where this can be achieved, it may be valuable). However, review processes should be designed to include some element of reflection on international good practice, such as a reflective statement from the provider on how their provision compares with similar practice in some other countries. Institutions may wish to consider how they can support such informal 'benchmarking' at a central level.

Internal processes should take full account of student feedback, and include procedures to obtain student views of the provision being reviewed

We expect all institutions to have ongoing processes of obtaining student feedback; SHEFC is actively considering ways in which it can support institutions to develop good practice in this area. Institutional processes for student feedback will be explicitly considered as part of institutional audit. Internal review should be designed in order to explore the ways in which providers have generated, considered and acted on feedback from their students in the design and operation of their programmes and the organisation of students' learning environment. We also propose, as a separate measure, that each internal review process should gather additional specific information from students as part of the evidence base for reviews. An appropriate methodology would be one which:

  • generated holistic evidence about student views of provision and of their learning experience;
  • differentiated between the views of different categories of students where these are likely to be significant (eg part-time and full-time, junior and senior, entrants from school and entrants from FE, etc);
  • allowed identification of distinctive characteristics of major subsets of provision; and
  • took account of the views of graduates on the relevance of provision for their careers.

SHEFC has indicated that institutional audit teams will include a student or someone nominated by the student body. There is no equivalent requirement for internal review teams to include student members. However, it is a key factor of SHEFC's new approach to quality to seek ways of involving students in quality processes and we would encourage institutions to follow a similar approach. Specifically, we would encourage institutions to consider the potential merits of including student members on internal review teams, and also to put in place mechanisms to engage effectively with their student body during internal reviews.

Internal reviews should consider the effectiveness of annual monitoring arrangements and follow-up action for programmes covered by the review

We expect each institution to operate systems of annual monitoring across the full range of provision; this is likely to include not only student survey data, but also performance data on recruitment, progression and achievement. Good practice in such monitoring would be to include mechanisms which allow some benchmarking of provision against other areas of the institution's activities, as well as equivalent provision elsewhere. Institutions should also make appropriate use of external reference points including external examiners' reports. A key element of quality assurance and enhancement is the extent to which the outcomes of such monitoring are scrutinised and acted on in order to address shortcomings and spread good practice. Internal review processes should be designed in a way which allows reflection on the effectiveness of monitoring and follow-up, eg by following audit trails of previous monitoring outcomes and considering processes in place to reflect and act on feedback and performance data.

At the institutional level, SHEFC expects institutions to reflect on strategic issues arising from regular monitoring, and to make use of this information in its strategy for continuous quality improvement. This will be explicitly considered during institutional audit; however, institutions may wish to consider how they can best design internal review processes in order to facilitate institution-level reflection on the outcomes of monitoring. Good practice would be to have reporting procedures at the programme, subject or department level which passed on relevant issues for consideration at institutional level. Internal review processes should be designed to allow constructive reflection on the effectiveness of these procedures.

Assurance and enhancement

The primary purpose of internal reviews is to provide assurances about the quality and standards of provision. Since there will no longer be a programme of external subject reviews, it is vital that internal reviews provide robust, comprehensive and credible evidence that standards and quality of provision in Scottish HEIs are being maintained. However, SHEFC's new approach gives a central role to quality enhancement, and we would therefore encourage institutions to develop internal review processes which also:

  • promote dialogue on areas in which quality might be improved;
  • identify good practice for dissemination within the institution and beyond (such as quality enhancement engagements); and
  • encourage and support providers' efforts to reflect critically on their practice.

TopTop