Skip navigation

Handbook for institutional audit: England

QAA 021 7/2002

Annexes


Annex E: Information

Information requirements

1 The institutional audit process depends to a large extent on the availability of information for the audit team to consider. For the most part this is covered by the recommendations in Information on quality and standards in higher education: Final report of the Task Group (HEFCE 02/15). These comprise two sections, the first (Part A) being information which should be available in each institution for internal management information purposes, and the second (Part B) which is routinely published by the institution. The full list of the HEFCE 02/15 information sets is provided at the end of this annex.

2 The audit team will routinely require access to all the items in Part B of the information sets. It will also need access to some of the material in Part A, according to the precise nature of its enquiries.

3 The audit team will also need access to some information additional to that listed in HEFCE 02/15. This additional information includes:

  • the institution's self-evaluation documents (SEDs), including SEDs specific to the discipline audit trails, and supporting documentation;
  • information submitted by representatives of students of the institution;
  • information from the institution and other sources (such as professional, statutory and regulatory bodies) about the discipline areas selected for trailing, including evidence of student achievement;
  • reports on the institution by relevant external bodies within the six years preceding the audit;
  • information (written or oral) acquired during and after the briefing visit, and during the audit visit.

The information is limited in all cases to no more than is necessary to inform the audit team's proposed enquiries.

Submission of information to the Agency

4 The assistant director will discuss with the institution the timing and format of submission of information for the audit. Where possible, information should be submitted in an electronic format. Further advice on preferred formats for information submission will be provided on the Agency's web site.

The Agency's approach to the management of information

5 The Agency recognises that clarity of process in relation to the management of information, derived from institutions and from other sources, is essential. A formal information policy is under development and will be published on the Agency's web site in due course. The policy covers matters relating to the collection, collation, evaluation, use and dissemination of information. It is intended to promote a common understanding, both within the Agency and amongst external stakeholders and institutions, of the ways in which the Agency uses information, taking into account the requirements of the Data Protection Act (1998) and other prevailing legislation.

6 In relation to institutional audit, formal submissions of information by institutions are received and processed in the first instance by the Agency's Information Unit. The Unit supports the key stages of the audit process and, by using standard approaches and techniques, endeavours to provide a consistent approach to the use and handling of information by the Agency.


The information listed in HEFCE 02/15

Part A: Information which should be available in all HEIs

1 Institutional context:

(a) Mission statement.
(b) Relevant sections of the HEI's corporate plan.
(c) Statement of quality assurance policies and processes.
(d) Learning and teaching strategy and periodic reviews of progress.

2 Student admission, progression and completion:

(a) Student qualifications on entry.
(b) Range of entrants classified by age, gender, ethnicity, socio-economic background, disability and geographical origin, as returned to HESA.
(c) Progression and retention data for each year of each course/programme, differentiating between failure and withdrawal.
(d) Data on student completion.
(e) Data on qualifications awarded.
(f) Data on employment/training outcomes from the First Destination Survey.

3 Internal procedures for assuring academic quality and standards:

(a) Programme approval, monitoring and review:

  • programme specifications;
  • a statement of the respective roles, responsibilities and authority of different bodies within the HEI involved in programme approval and review;
  • key outcomes of programme approval, and annual monitoring and review processes;
  • periodic internal reports of major programme reviews;
  • reports of periodic internal reviews by departments or faculties;
  • accreditation or monitoring reports by professional, statutory or regulatory bodies.

(b) Assessment procedures and outcomes:

  • assessment strategies, processes and procedures;
  • the range and nature of student work;
  • external examiners' reports, analysis of their findings, and the actions taken in response;
  • reports of periodic reviews of the appropriateness of assessment methods used.

(c) Student satisfaction, covering the views of students on:

  • arrangements for academic and tutorial guidance, support and supervision;
  • library services and IT support;
  • suitability of accommodation, equipment and facilities for teaching and learning;
  • perceptions of the quality of teaching and the range of teaching and learning methods;
  • assessment arrangements;
  • quality of pastoral support.

(d) Evidence available to teams undertaking HEIs' own internal reviews of quality and standards:

  • the effectiveness of teaching and learning, in relation to programme aims and curriculum content as they evolve over time;
  • the range of teaching methods used;
  • the availability and use of specialist equipment and other resources and materials to support teaching and learning;
  • staff access to professional development to improve teaching performance, including peer observation and mentoring programmes;
  • the use of external benchmarking and other comparators both at home and overseas;
  • the involvement of external peers in the review method, their observations, and the action taken in response.


Part B: Information for publication

4 Quantitative data:

(a) HESA data on student entry qualifications (including A-levels, access courses, vocational qualifications, and Scottish Highers).
(b) Performance indicators and benchmarks published by the HE funding bodies on progression and successful completion for full-time first degree students (separately for progression after the first year, and for all years of the programme).
(c) HESA data on class of first degree, by subject area.
(d) Performance indicators and benchmarks published by the HE funding bodies on first destinations/employment outcomes for full-time first degree students.

5 Qualitative data:

(a) Summaries of external examiners' reports on each programme.
(b) A voluntary commentary by the HEI at whole institution level on the findings of external examiners' reports.
(c) Feedback from recent graduates, disaggregated by institution, collected through a national survey.
(d) Feedback from current students collected through HEIs' own surveys, undertaken on a more consistent basis than now.
(e) A summary statement of the institution's learning and teaching strategy as presented to the HEFCE under the Teaching Quality Enhancement Fund programme.
(f) Summary statements of the results of, and the actions taken in response to, periodic programme and departmental reviews, to be undertaken at intervals of not more than six years.
(g) Summaries of employer links, included in item 5(e) above and in programme specifications.


Annex F: Selection and training of auditors, audit secretaries and specialist advisers

Introduction

1 Auditors, audit secretaries and specialist advisers are selected by the Agency on the basis of published selection criteria, and generally from nominations made by institutions. Specialist advisers may be also be selected from nominees of professional and statutory bodies or from direct applications from appropriately qualified and experienced people. All are provided with induction and training to ensure that they are familiar with the aims, objectives and procedures of the audit process and with their own roles within it. Auditors and audit secretaries are recruited on the basis that they agree to undertake at least three audits over a period of two years. They may continue beyond this period by mutual agreement. Specialist advisers are drawn from the register of those trained to fulfil the roles of subject specialist reviewers in the Agency's subject review process, with the expectation that they will participate in up to six review activities (audits, subject reviews and/or developmental engagements) over a period of two years. Again this period may be extended by mutual agreement.

2 The qualities required in auditors, audit secretaries and specialist advisers are outlined below. Every attempt is made to ensure that the cohorts of auditors and specialist advisers reflect appropriate sectoral, discipline, geographical, gender and ethnic balances. Both institutional auditors' discipline expertise and an institution's spread of disciplines (based on the JACS Subject Groups) are taken into account in the construction of audit teams, so as to provide a sufficient spread of knowledge for an informed view to be taken of primary evidence relating to quality and standards.

3 When it is necessary for auditors to request a second opinion from specialist advisers, specialist advisers are selected so as to provide a match to the specialisms and level of awards of the discipline trail.

4 Audit secretaries are recruited from amongst administrative staff in institutions. In common with auditors and specialist advisers, audit secretaries are not appointed to teams auditing their own institutions.

5 Training for auditors, audit secretaries and specialist advisers is undertaken by the Agency in collaboration with appropriate training providers. The purpose of the training is to ensure that all:

  • understand the aims and objectives of the audit processes;
  • are acquainted with the procedures involved;
  • understand their own roles and tasks, the importance of team coherence, the Agency's expectations of them, and the rules of conduct governing the process; and
  • have an opportunity to explore and practise the techniques of data assimilation and analysis, the development of programmes for visits, the construction and testing of hypotheses, the forming of judgements and statements of confidence, and the preparation of reports.

Qualities required in all auditors

  • Wide experience of academic management and quality assurance at institutional level in UK higher education.
  • Personal and professional credibility with heads of institutions and senior managers in the higher education sector.
  • Ability to assimilate a large amount of disparate information; to analyse and draw reliable conclusions about complex arrangements; and to undertake research and investigation into documentary and oral evidence in order to form judgements.
  • Clear oral and written communication skills.
  • If representing a specified academic discipline, to have current or recent (within the last two years) experience of delivering teaching and supporting learning at level H or M.
  • If representing a specified academic discipline, to have current or recent (within the last two years) experience of examining at level H or M.

Qualities required in all audit secretaries

  • Current or recent experience (within five years) of administration of academic management and/or quality assurance at institutional level in UK higher education.
  • Wide experience of working with senior committees in UK higher education.
  • Ability to assimilate a large amount of disparate information, and to analyse and make reliable judgements about complex arrangements.
  • Ability to keep a reliable record of discussions; to summarise the key outcomes; and to draft notes to a specified format to set deadlines.

Qualities required in all specialist advisers

  • Personal credibility, in the subject area, with academic peers in UK higher education, or equivalent industrial/professional credibility.
  • Current experience of delivering teaching, supporting learning, and examining at level H or M within the subject area.
  • Experience of working with programme specifications written for programmes in the subject area; a good understanding of programme entry requirements, and an ability to interpret progression statistics; and familiarity with comparable programmes and standards of awards in other institutions.
  • Ability to assimilate a large amount of disparate information, and to analyse and draw reliable conclusions about complex arrangements.
  • Ability to identify, plan and follow lines of investigations to meet a task specified by an audit team, using a variety of sources, including documentary and oral evidence, in order to draw secure conclusions.


Annex G: Criteria for confidence judgements, and the relationship between confidence judgements, recommendations and follow-up action

Set out below are the criteria to be used by the audit team in judging the confidence that can reasonably be placed in the soundness of the institution's present and likely future management of the quality of its programmes and the academic standards of its awards. The relationship between the confidence judgement, the nature of the audit team's recommendations, and the follow-up action after the audit, is also summarised.


Broad confidence

A judgement of broad confidence indicates that the institution is judged both to possess rigorous mechanisms for the management of the quality of its programmes and the standards of its awards and to be using these effectively and consistently. The mechanisms will include a strong and scrupulous use of independent external examiners in summative assessment procedures and independent external persons in the internal periodic review of disciplines or programmes. The institution will also have provided evidence to demonstrate that it is has the capacity to, and is very likely to continue to, secure and maintain quality and standards in the future and that the publicly available information it provides is complete, accurate and reliable.

Evidence: The judgement will be reached on evidence that demonstrates that the institution has sound structures and procedures for the assurance and enhancement of quality and standards; that it is successful in the management of those structures and procedures at institutional level; and that the procedures are applied effectively at discipline level. A judgement of broad confidence implies confidence in the institution's capacity and commitment to identify and address any situation that has the potential to threaten the quality of programmes or standards of awards.

Recommendations: A judgement of broad confidence may be accompanied by a small number of recommendations that are considered advisable and it will not be unusual for there to be a number that are considered desirable, but there will be none that are considered essential. Broad confidence in an institution indicates confidence that the recommendations set out in the report are likely to be considered and dealt with through the institution's normal structures for quality enhancement.

Follow-up

After one year: the institution is asked to submit a brief report to the Agency on how it has responded to the report.

After three years: the Agency expects to make a brief visit to the institution to review progress since the audit and discuss the institution's intentions in respect of managing quality and standards over the three years until the next audit.


Limited confidence

A judgement of limited confidence indicates that there is evidence that the institution's capacity to manage the quality of its programmes and/or standards of its awards soundly and effectively is limited or is likely to become limited in the future. The reason for this judgement may be notable weaknesses either in the management of the institution's structures and procedures or in their implementation at discipline level. Confidence may be limited either because of the extent or the degree of weaknesses identified; significant weakness in any one discipline area will limit overall confidence. The determining factor in reaching a judgement of limited confidence is not simply evidence of problems in some programmes - no institution could be expected to avoid these entirely. It is, instead, the fact that the institution may not be have been fully aware of the problems and/or has failed to take prompt and appropriate action to remedy them. The audit team may also express limited confidence where there is reason for reservations about whether the publicly available information provided by the institution can be considered complete, accurate and reliable, or where the institution makes a less than full use of independent external examiners in summative assessment procedures and independent external persons in the internal periodic review of disciplines or programmes.

Evidence: Where there is evidence to question either the effectiveness of the current assurance of quality and standards, or the institution's capacity to maintain quality and standards in the future, the judgement will indicate whether the concerns are limited to a small number of matters or are more widespread, and whether or not these matters place quality and/or standards at risk. If a discipline audit trail results in a request for specialist advice and the findings of the specialist advisers indicate that quality and/or standards on any programme are a cause for concern, this will be made clear.

Recommendations: A judgement of limited confidence is likely to lead to a small number of recommendations that are considered essential, as well as a number that are considered advisable and desirable.

Follow-up

Within three months of report publication: the institution is asked to submit an action plan to the Agency indicating how it intends to address the recommendations in the report, and to provide, subsequently, a progress report on how the action plan has been implemented. The audit is not formally signed off until the Agency is satisfied that the action plan has been implemented successfully, with a maximum time limit of 18 months. If at that point concerns remain about the effectiveness of the remedial action, the Agency conducts a further visit.

After three years: the Agency expects to make a brief visit to the institution to review progress since the audit and discuss the institution's intentions in respect of managing quality and standards over the three years until the next audit.


No confidence

A judgement of no confidence indicates that there is substantial evidence of serious and fundamental weaknesses in the institution's capacity both at institutional and discipline level to secure and maintain the quality of its programmes and standards of its awards.

Evidence: A judgement of no confidence will be reached either because of serious absences or flaws in the institution's procedures themselves or because of ineffectiveness in their management, and where either quality or standards can be seen to be at immediate risk, or there is serious doubt as to the institution's capacity to secure and maintain them in the future. A judgement of no confidence may also be reached where it can be demonstrated that the information made available to the public by the institution cannot be relied upon and can be shown to be inaccurate and/or misleading.

Recommendations: A judgement of no confidence will be accompanied by a significant number of recommendations that are considered essential, as well as a number that are considered advisable and desirable.

Follow-up

Within three months of report publication: the institution is asked to submit to the Agency an action plan, with implementation times within 18 months, indicating how it intends to address the recommendations in the report. Subsequently, it is asked to provide quarterly progress reports on how the identified weaknesses are being addressed. After 18 months, the Agency carries out a short follow-up visit to the institution to check progress. The audit is not formally signed off until the Agency is satisfied that the action plan has been implemented successfully. If after 18 months concerns remain about the effectiveness of the remedial action, the Agency may bring forward the date of the next audit.

After three years: the Agency expects to make a brief visit to the institution to review progress since the audit and discuss the institution's intentions in respect of managing quality and standards over the period until the next audit.


Annex H: Indicative report structure

The indicative report structure is set out below.

Preface
[A standard summary, common to all reports, of the institutional audit process and its possible outcomes]

Summary
[A summary intended primarily for the public, especially potential students, and to be made available separately from the rest of the report]

  • Introductory statement.
  • Statement of confidence.
  • Summary outcomes of discipline audit trails.
  • Reliability of information.
  • Use made of the academic infrastructure.
  • Features of good practice.
  • Recommendations for action by the institution.

The findings of the audit
[An overview of the findings of the audit taking an institutional perspective of: the institution's capacity to manage the quality of its programmes for the assurance and enhancement of quality; its capacity to underpin its programmes with effective learning support; and its capacity to manage effectively the security of the academic standards of its awards]

  • The effectiveness of institutional procedures for assuring the quality of programmes.
  • The effectiveness of institutional procedures for securing the standards of awards.
  • The effectiveness of institutional procedures for supporting learning.
  • The outcomes of the discipline audit trails.
  • The institution's use of the academic infrastructure.
  • The utility of the SED as an illustration of the institution's capacity to reflect upon its own strengths and limitations, and to act on these to enhance quality and standards.
  • Commentary on the institution's intentions for the enhancement of quality and standards.
  • The reliability of information.
  • Features of good practice [bulleted list].
  • Recommendations for action by the institution [bulleted list].

Main report

Section 1: Introduction

[A standard introduction to this audit]

  • The institution and its mission.
    [Size, style and mission of the institution - any special features - how these influenced the directions taken by the audit]
  • Collaborative provision.
    [A statement of the scale/complexity of collaborative provision - the way it is addressed in this audit (embedded or identified for separate audit of collaborative provision)]
  • Background information.
    [The information base for the audit, including the nature and status of any submission to the audit team by the student body]
  • The audit process.
    [The audit process as applied to this particular audit - schedule and sequence - selection of discipline audit trails and thematic enquiries - the documentary evidence base - the nature and status of any submission to the audit by the student body]
  • Developments since the previous academic quality audit.
    [Outline of key actions taken since the last audit and since the Agency's visit at the three-year mid-point in the cycle: indication of any major structural change that has taken place since the last audit and three year mid-point that will have a bearing on this audit; outline of key findings of Agency reviews since last audit: outline of matters emerging from any overseas audit; and outline of any matters emerging from PSB reports]

Section 2: The audit investigations: institutional processes

[This is a narrative that evaluates the impact on quality, learning support and standards of various institutional processes and features of quality management relating to all the institution's awards. In each sub-section the emphasis should be on the use made by the institution of the process or feature of quality management, and on its effectiveness. The following represents a core set of sub-sections that will be included. Other sub-sections may be added where appropriate - for example, to reflect the particular features of the institution, or where identified by the institution in its SED]

  • The institution's view as expressed in the SED.
  • The institution's framework for managing quality and standards, including collaborative provision.
  • The institution's intentions for the enhancement of quality and standards.
  • Internal approval, monitoring and review processes.
  • External participation in internal review processes.
  • External examiners and their reports.
  • External reference points.
  • Programme-level review and accreditation by external agencies.
  • Student representation at operational and institutional level.
  • Feedback from students, graduates and employers.
  • Progression and completion statistics.
  • Assurance of quality of teaching staff, appointment, appraisal and reward.
  • Assurance of quality of teaching through staff support and development.
  • Assurance of quality of teaching delivered through distributed and distance methods.
  • Learning support resources.
  • Academic guidance, support and supervision.
  • Personal support and guidance.
  • Collaborative provision.

Section 3: The audit investigations: discipline trails and thematic enquiries

  • Discipline audit trails.
  • Thematic enquiries.

Section 4: The audit investigations: published information

  • The students' experience of published information and other information available to them.
  • Reliability, accuracy and completeness of published information.


Annex I: Reference documents relevant to the institutional audit process

External reference points

When considering institutional management of quality and standards, audit teams draw upon a range of external reference points, including:

Code of practice for the assurance of academic quality and standards in higher education: Sections 1-10 (Quality Assurance Agency for Higher Education, 1999-2001) (http://www.qaa.ac.uk).

Guidelines for preparing programme specifications (Quality Assurance Agency for Higher Education, 2000) (http://www.qaa.ac.uk).

Guidelines on the Quality Assurance of Distance Learning (Quality Assurance Agency for Higher Education, 1999) (http://www.qaa.ac.uk).

Subject benchmark statements (Quality Assurance Agency for Higher Education, 2000-2002) (http://www.qaa.ac.uk).

The framework for higher education qualifications in England, Wales and Northern Ireland (Quality Assurance Agency for Higher Education, 2001) (http://www.qaa.ac.uk).

Indirectly funded partnerships: codes of practice for franchise and consortia arrangements (Higher Education Funding Council for England, 00/54, 2000) (http://www.hefce.ac.uk).

Information on quality and standards in higher education: Final report of the Task Group (Higher Education Funding Council for England, Report 02/15, 2002) (http://www.hefce.ac.uk).

Discipline categorisation

For the purposes of discipline categorisation and other calculations, the institutional audit process has as its main organising principle the Subject Groups defined within the Joint Academic Coding System (JACS) (http://www.hesa.ac.uk).

Related reference documents

Agency review processes related to the institutional audit process and referred to in this Handbook are described in detail in the Handbook for academic review (Quality Assurance Agency for Higher Education, 2000) (http://www.qaa.ac.uk).

The arrangements during the transitional period 2002-2005 are described in a note entitled Arrangements during the transitional period 2002-2005 for Higher Education Institutions in England.


Annex J: The Agency's operational principles and process standards

Background

1 The Agency's approach to undertaking institutional audits draws upon the practices and process standards developed and enhanced by its predecessor bodies. Since those bodies began their work, good practice in auditing (guided by published standards of auditing practice) and requirements relating to accountability and reporting, have developed considerably. The Agency recognises that some of the process standards it has observed in the past have been implicit rather than explicit, and that the institutional audit process should be underpinned by a more explicit statement on operational principles and process standards.

2 In developing its operational principles and process standards, the Agency has taken note of the principles underpinning the AA1000 series accountability standard and the Seven Principles of Public Life developed by the Nolan Committee.

Principles

3 The Agency seeks to observe and promote several general principles within both the strategic and operational levels of its work. The principles are:

  • Inclusiveness - taking into account the needs of all stakeholder groups and facilitating their participation in aspects of the Agency's work.
  • Openness - transparency in the work and methods of the Agency, to build trust and confidence among stakeholders, and to provide information about the Agency's work to the wider public.
  • Accountability - demonstrating that the Agency is using its resources to good effect and with probity; conducting its work with integrity and impartiality; and ensuring that stakeholders are able to depend on the information provided.
  • Timeliness - the need for regular, systematic and timely action in all reporting processes to support the decision-making of the Agency and its stakeholders.
  • Comparability - using experience drawn from within the Agency and other organisations as a means with which to inform future work.
  • Relevance - ensuring that the information provided by the Agency is useful to, and understood by, all stakeholders.

4 These principles have been used to develop explicit service standards for institutional audit, the details of which are published on the Agency's web site.

Quality assurance mechanisms

5 The Agency is committed to the regular monitoring and evaluation of its policies, procedures and processes, to ensure their ongoing credibility and to continuously improve its performance in response to the results. In respect of institutional audit, this commitment includes providing the opportunity for participants in the process, including students, to provide structured feedback on their experiences.

TopTop