Handbook for institutional audit: England

QAA 021 7/2002

PDF version (143KB)

Contents

• Introduction
• The aims and objectives of institutional audit
• The institutional audit process in summary
• How the process works
• Annex A: Outline of the institutional audit process
• Annex B: Guidelines for producing institutional self-evaluation documents
• Annex C: Guidelines for producing self-evaluation documents for discipline audit trails
• Annex D: Written submissions from students
• Annex E: Information
• Annex F: Selection and training of auditors, audit secretaries and specialist advisers
• Annex G: Criteria for confidence judgements, and the relationship between confidence judgements, recommendations and follow-up action
• Annex H: Indicative report structure
• Annex I: Reference documents relevant to the institutional audit process
• Annex J: The Agency's operational principles and process standards

---

Introduction

Background

1 The mission of the Agency is to promote public confidence that the quality of provision and standards of awards in higher education are being safeguarded and enhanced. To this end, the Agency carries out audits of the academic performance of institutions. This Handbook describes the Agency's method and procedures for undertaking audits of higher education institutions (HEIs) in England. The arrangements for auditing institutions in Scotland, Wales and Northern Ireland are described in separate documents.

2 The process of institutional audit described in this Handbook has been developed by the Agency in partnership with the Higher Education Funding Council for England (HEFCE), the Standing Conference of Principals (SCOP) and Universities UK (UUK), and has been endorsed by the Department for Education and Skills (DfES). For institutions in England, it replaces the previous processes of continuation audit, undertaken by the Agency at the request of UUK and SCOP, and universal subject review, undertaken by the Agency on behalf of HEFCE, as part of the latter's statutory responsibility for assessing the quality of education that it funds.

Process

3 Institutional audit is a new process that pays due attention to the quality of programmes and the standards of awards at the point of delivery, as well as to institutions' ultimate responsibility for what is done in their names and through the exercise of their formal powers. It is an evidence-based process carried out through peer review, and balances the need for publicly credible, independent and rigorous scrutiny of institutions with the recognition that the institutions themselves are best placed to provide stakeholders with valid, reliable and up-to-date information about the quality of their programmes and the standards of their awards. At the centre of the process is an emphasis on students - in terms of the quality of the information they receive about their programmes of study, the ways in which their learning is facilitated and supported, and the academic standards they are expected to achieve, and do achieve in practice.

4 As part of the development of the institutional audit process, a HEFCE task group was established to identify the categories of data, information and judgements about quality and standards that should be available in HEIs and that should be published. The resulting document Information on quality and standards in higher education: Final report of the Task Group (HEFCE 02/15) expects that HEIs will be maintaining and publishing a range of up-to-date information on quality and standards, and will be undertaking their own internal reviews, in the context of their strategies for teaching and learning. The institutional audit process builds upon the recommendations set out in HEFCE 02/15 and draws upon the information sets, both published and to be available within HEIs, identified by the task group.

5 The institutional audit process is intended to combine scrutiny of internal quality assurance systems at an institutional level with investigations of how those systems operate at the level of the discipline (used within this Handbook to describe defined areas of academic study) and to assure the quality of the programme (used within this Handbook to mean the full diet of modules, options, and other structured learning opportunities, individual research study, and associated learner support, which together comprise a pathway that leads to an award). For the purposes of discipline categorisation and other calculations, the process has as its main organising principle the Subject Groups defined in the Joint Academic Coding System (JACS). However, to avoid confusion with the Agency's subject review process, the use of the word subject within this Handbook is limited to references to the 42 subject groupings used by the Agency in relation to both subject review and the Subject benchmark statements developed by academic communities under the aegis of the Agency.

The audit cycle and transitional arrangements

6 Institutional audits will be introduced progressively from 2002-03. All HEIs will have participated in an audit by the end of 2005. Thereafter, audits will take place on a six-year cycle. At the three-year mid-point in the cycle, the Agency expects to make a brief visit to each institution to review progress since the previous audit and to discuss the institution's intentions in respect of managing quality and standards over the three years until the next audit. It is assumed that, throughout the cycle, institutions will continue to meet the expectations set out in HEFCE 02/15.

7 During the first two years of transitional period, 2002-03 and 2003-04, it is a HEFCE requirement that institutions awaiting their first audit participate in some form of review or activity with the Agency at the level of the discipline. For most institutions, the activity will take the form of new, development-focused engagements at discipline level, intended to offer institutions an opportunity to test, in co-operation with the Agency, their internal procedures for assuring quality and standards. A smaller number of discipline-level interactions will take the form of subject review, using the method described in the Agency's Handbook for academic review. The arrangements during the transitional period are described in a separate note entitled 'Arrangements during the transitional period 2002-2005 for Higher Education Institutions in England', and will be the subject of separate correspondence between the Agency and institutions.

The Agency's operational principles and service standards

8 The process of institutional audit will require a high degree of openness, transparency and trust in the partnership between the Agency and each HEI. To ensure that the process is robust, impartial and deserving of that trust, the Agency's work will be underpinned by a set of general principles and the adoption of explicit service standards. Further details are provided in Annex J.

---

The aims and objectives of institutional audit

Aims

9 The aims of institutional audit are to meet the public interest in knowing that institutions in England are:

• providing higher education, awards and qualifications of both an acceptable quality and an appropriate academic standard; and (where relevant);
• exercising their legal powers to award degrees in a proper manner.

Objectives

10 The objectives are:

• to contribute, in conjunction with other mechanisms, to the promotion and enhancement of high quality in teaching and learning;
• to ensure that students, employers and others can have ready access to easily understood, reliable and meaningful public information about the extent to which institutions are individually offering programmes of study, awards and qualifications that meet general national expectations in respect of academic standards and quality;
• to ensure that if the quality of higher education programmes or the standards of awards are found to be weak or seriously deficient, the process forms a basis for ensuring rapid action to improve them; and
• to provide a means of securing accountability for the use of public funds received by institutions.

---

The institutional audit process in summary

Scope

11 Institutional audits examine three main areas:

• the effectiveness of an institution's internal quality assurance structures and mechanisms, in the light of the Agency's Code of practice for the assurance of academic quality and standards in higher education (the Code of practice), and the way in which the quality of its programmes and the standards of its awards are regularly reviewed and the resulting recommendations implemented. This provides public information on an institution's soundness as a provider of tertiary qualifications of national and international standing;
• the accuracy, completeness and reliability of the information that an institution publishes about the quality of its programmes and the standards of its awards, drawing upon the information sets described in HEFCE 02/15. This provides information on the trust that can be placed in an institution's own published descriptions of the quality of its provision; it also makes that description more useful to students and other interested parties;
• several examples of the institution's internal quality assurance processes at work at the level of the programme ('discipline audit trails') or across the institution as a whole ('thematic enquiries'), in order to demonstrate the validity and reliability of the information being generated by these internal processes. As a general guide, the discipline audit trails are expected to represent some 10 per cent of the institution's higher education programmes in terms of full-time equivalent student numbers.

12 In examining these areas, audit teams focus in particular on:

• internal quality assurance reviews and their outcomes, especially at the level of the discipline and/or programme;
• the use made of external reference points, including the Code of practice, The framework for higher education qualifications in England, Wales and Northern Ireland (the FHEQ), and Subject benchmark statements;
• publicly available information about the quality of programmes and the standards of awards;
• internal systems for the management of information and their contribution to the effective oversight of quality and standards;
• the development, use and publication of programme specifications;
• the academic standards expected of and achieved by students;
• the experience of students as learners;
• the quality assurance of teaching staff, including appointment criteria and the ways in which teaching effectiveness is appraised, improved and rewarded.

Collaborative provision

13 Where practicable, the audit process includes consideration of provision offered by institutions in collaboration with other providers, both in the UK and overseas, in the light of the relevant section of the Code of practice and (where relevant) HEFCE's Indirectly funded partnerships: codes of practice for franchise and consortia arrangements. However, where an institution's collaborative provision is too large or complex for a reliable scrutiny to be undertaken, it is not included in the audit. In due course, the Agency expects to conduct separate audits of the way in which such provision is managed by the institutions concerned. The Agency also expects to continue with its programme of audits of specific partnerships between UK institutions and providers overseas.

Information

14 To enable them to form their judgements, audit teams have available to them a variety of information sources about an institution, including:

• the information sets described in HEFCE 02/15, including both the information to be available in all institutions and the information for publication (see Annex E). The Agency is aware that institutions will need time to meet all of the requirements of HEFCE 02/15 and will provide advice as appropriate to audit teams visiting institutions early in the audit cycle;
• the institution's self-evaluation documents (SEDs), including SEDs specific to the discipline audit trails, and supporting documentation;
• information submitted by representatives of students of the institution;
• information from the institution and other sources (such as professional, statutory and regulatory bodies) about the discipline areas selected for trailing, including evidence of student achievement;
• reports on the institution by the Agency and other relevant bodies within the six years preceding the audit;
• information (written or oral) acquired during and after the briefing visit, and during the audit visit.

15 To help them, audit teams also have available a digest and analysis of relevant data, produced by the Agency's Information Unit. The Unit supports the key stages of the audit process. Matters relating to the Agency's handling and management of information are described in Annex E.

Judgements and reports

16 Each institutional audit results in a report published by the Agency. The report sets out the audit team's judgements on:

• the confidence that can reasonably be placed in the soundness of the institution's present and likely future management of the quality of its programmes and the academic standards of its awards (a judgement based, in part, on the direct scrutiny of academic standards through primary evidence);
• the reliance that can reasonably be placed on the accuracy, integrity, completeness and frankness of the information that an institution publishes about the quality of its programmes and the standards of its awards. This judgement takes into account the team's findings in respect of the provision it has considered in the discipline audit trails, augmented where necessary by advice from specialists from the discipline concerned.

17 In making these judgements, audit teams give particular attention to the Agency's expectations in two key areas. The first expectation is that institutions are making strong and scrupulous use of independent external examiners in summative assessment procedures. The second is that a similar use is made of independent external persons in the internal periodic review of disciplines or programmes. Teams are unable to make a judgement of broad confidence in an institution if either of these elements is deficient.

18 Institutional audit reports also provide comment on other matters, including the characteristics, strengths and limitations of the institution's internal quality assurance methods, and the quality of programmes and standards of awards achieved in practice, drawing upon the findings of the discipline audit trails. The reports highlight features of good practice, make recommendations for further consideration by the institution, and identify any area where the audit team considers there is good reason for a full review at the discipline level to be carried out, or where it considers that an action plan at either the discipline or institutional level should be implemented by the institution.

Students

19 Students are central both to the principal focuses of institutional audit and to the audit process itself. Audit teams scrutinise a range of matters directly relevant to students, including the quality of the information provided for them, the ways in which their learning is facilitated and supported, and the academic standards they are expected to achieve, and achieve in practice. In each audit, students are invited to participate in the key stages of the process. Their representative body - normally the Students' Union, or equivalent - has the opportunity to participate in the preliminary meeting between the Agency and the institution and may make a written submission to the audit team in advance of the audit visit. Officers of the representative body and other students are invited to participate in specified meetings during the briefing and audit visits to the institution, and have the opportunity to ensure that the team is aware of matters of primary interest or concern to them.

Audit personnel

20 Audit teams comprise a minimum of four and a maximum of seven auditors, and an audit secretary. The size of the team is determined by the Agency on the basis of the size and complexity of an institution's provision. Each team includes at least one 'core' auditor who focuses wholly on institutional level matters and has a particular role in ensuring that the work of the team provides the evidence necessary for the overall judgements to be made; the other auditors participate in both institutional level enquiries and discipline audit trails. All auditors have expertise and recent experience relevant to their roles.

21 In the light of its enquiries during an audit, an audit team may wish to call upon specialist advisers to give a second opinion on specific aspects of an institution's provision at discipline level. Specialist advisers have expertise in the relevant academic area and, for the purposes of the audit, work in groups of two or more. Their remit is to look in depth at particular aspects of a discipline, as identified by the team, and to report back to the team on their findings.

22 Auditors, audit secretaries and specialist advisers are selected by the Agency, generally from nominations made by institutions, on the basis of published selection criteria. All are provided with training to ensure that they are familiar with the aims, objectives and procedures of the audit process, and their own roles and tasks within it. Auditors are recruited on the basis that they agree to undertake at least three audits over a period of two years. They may continue beyond this period by mutual agreement. Further information about the Agency's arrangements for selecting and training audit teams and specialist advisers is provided in Annex F.

23 Each audit is co-ordinated by an assistant director of the Agency. In the period preceding the audit visit, the assistant director provides advice to the institution on its preparations for the audit, and works with the Agency's Information Unit and the audit team on the initial analysis of documentation. He or she accompanies the team during the briefing visit and for part of the audit visit, providing advice as appropriate. It is the responsibility of the assistant director to test that the team's findings are supported by adequate and identifiable evidence, and that the audit report provides information in a succinct and readily accessible form.

Next > How the process works