Handbook for institutional audit: England

QAA 021 7/2002

PDF version (143KB)

Contents

• Introduction
• The aims and objectives of institutional audit
• The institutional audit process in summary
• How the process works
• Annex A: Outline of the institutional audit process
• Annex B: Guidelines for producing institutional self-evaluation documents
• Annex C: Guidelines for producing self-evaluation documents for discipline audit trails
• Annex D: Written submissions from students
• Annex E: Information
• Annex F: Selection and training of auditors, audit secretaries and specialist advisers
• Annex G: Criteria for confidence judgements, and the relationship between confidence judgements, recommendations and follow-up action
• Annex H: Indicative report structure
• Annex I: Reference documents relevant to the institutional audit process
• Annex J: The Agency's operational principles and process standards

---

How the process works

Preparation

24 An outline of the institutional audit process is provided in Annex A. The process begins around 10 months before the audit visit, when the Agency's Information Unit supplies the assistant director with a digest of the information set published by the institution about its management of quality and standards. The digest takes account of reports on the institution by the Agency and other relevant bodies within the six years preceding the audit, and is shared with the institution.

25 A preliminary meeting between the institution and the Agency takes place around nine months before the audit visit. The purpose of the meeting is to clarify the scope of the audit; to discuss the interactions between the institution, the Agency and the audit team; to ensure that the SEDs will be well-matched to the process of audit; to discuss any matters relating to both the published and internal information sets; and to confirm the basis for choosing discipline audit trails and areas for thematic enquiry. The meeting also includes an opportunity for discussion between the Agency and officers of the student representative body about the student contribution to the audit.

26 During the preliminary meeting, the Agency discusses with the institution any areas that are to be reviewed more fully than through the discipline audit trails (because the institution has requested a full subject review for its own purposes, or because a professional, statutory, regulatory or similar body requires the use of the subject review method as a basis for its own decisions about accreditation), and agrees a timetable. These reviews are not normally integrated chronologically with the institutional audit, but their findings are followed up by the institution and (if appropriate) the Agency, and their reports provide a major contribution to the discussions in the next audit round.

27 Following the preliminary meeting and drawing upon the information received from the Information Unit and the institution, the assistant director identifies a range of discipline areas from which the final selection of discipline audit trails will be made. The number of trails is determined on the basis of the size and breadth of the institution's provision, as measured by the number of JACS Subject Groups and the number of students. Further information on the selection of trails is provided below, paragraphs 40-42.

28 In the light of the initial identification of discipline audit trails, an audit team with appropriate expertise is appointed provisionally by the Agency. The institution is notified of the size of the team and the number of discipline audit trails around eight months before the audit visit.

Documentation and analysis

29 The institution is required to submit its initial documentation for the audit no later than 18 weeks before the audit visit. The initial documentation comprises the institutional SED and other documents that the institution wishes to provide for the audit team in advance of the briefing visit. If representatives of students within the institution wish to make a separate written submission to the team, that submission should also be sent to the Agency at this stage. Guidance on preparing institutional SEDs and student submissions is provided in Annexes B and D.

30 On receipt, the documentation submitted by the institution and its students is distributed by the Agency to the audit team. The team also receives an analysis of relevant data produced by the Information Unit in liaison with the assistant director. The analysis uses the institution's submission to augment the digest provided for the assistant director in advance of the preliminary meeting. On the basis of this information, the team is asked to consult (normally using electronic means) and to select, from the provisional selection made by the assistant director, the discipline areas that it intends to pursue during the audit. At this stage, the team also considers possible areas for thematic enquiry.

31 On the basis of the audit team's decisions, and not less than 14 weeks before the audit visit, the Agency confirms the membership of the team and provides the institution with a confirmed list of discipline audit trails. The documentation required to support the audit trails is described in more detail below, paragraph 44.

The briefing visit

32 The visit to the institution has two parts. The first part, the briefing visit, is held five weeks before the audit visit and lasts for a maximum of three days, of which a maximum of two days is spent at the institution. The purposes of the briefing visit are to permit the audit team to gather any additional (written or oral) information that it requires to clarify what it has already received; to consider its detailed lines of enquiry for the audit visit; to propose a programme for that visit; and to allocate particular responsibilities to individual team members. The assistant director accompanies the team throughout the visit.

33 The briefing visit is focused at the level of institutional management rather than individual disciplines. It has a standard structure and includes meetings with representatives of both the institution's staff (normally those who are involved in quality management at a senior level) and its students. The meetings with staff offer the institution an opportunity to bring the audit team up to date on institutional developments and changes since the institutional SED was submitted. The meeting with students offers a further opportunity for student representatives to draw the team's attention to matters of interest to the student body.

34 Following the briefing visit, the assistant director writes to the institution to confirm the programme for the audit visit (including any areas identified for thematic enquiry) and the illustrative documentation that the audit team would wish to be made available in advance of, or at the start of, the audit visit. The documentation may be drawn from the unpublished information set identified in HEFCE 02/15 or more widely, but is limited in all cases to no more than is necessary to inform the team's proposed enquiries.

The audit visit

35 For most institutions (with the exception of small specialist institutions) the audit visit extends over five working days, Monday to Friday. The detailed programme for each visit, based around meetings with staff and students, is decided by the audit team. Most visits include:

• opportunities for the team to read the documentation provided to support the audit, including external examiners' reports and documentation relating to internal reviews;
• exploration of the institution's approach to quality assurance;
• exploration of the relationship between institutional procedures and their operation at the programme or discipline level, giving particular attention to the effectiveness of internal reviews of programmes and awards;
• exploration of the way in which the institution is using the FHEQ, the Code of practice, and Subject benchmark statements;
• exploration of the chosen discipline audit trails and thematic enquiries, including targeted discussions and (in respect of audit trails) scrutiny of illustrative examples of assessed students' work. The conduct of the trails and enquiries is described in more detail below, paragraphs 43-54;
• exploration of the accuracy, completeness and reliability of the information published for students and others, with particular attention to programme specifications;
• exploration of the claims made for the quality of programmes and the actual achievements of students, focusing not only on academic outcomes, but also on the ways in which students are treated and their opportunities to learn optimised;
• during the closing stages, meetings with senior staff and, where necessary, staff from the discipline areas selected for trailing, to discuss any matters outstanding and to follow up any matters emerging from the audit trails.

36 On the final day of the audit visit, the audit team considers its findings at both the institutional and discipline levels in order to:

• decide on the confidence that it believes can reasonably be placed in the soundness of the institution's management of the quality of its programmes and the academic standards of its awards;
• decide on the reliance that it believes can reasonably be placed on the accuracy, integrity, completeness and frankness of the information that the institution publishes about the quality of its programmes and the standards of its awards;
• identify features of good practice in the management of quality and standards, or in the delivery of teaching and the facilitation of learning;
• agree recommendations, categorised in terms of importance.

The assistant director joins the team for this final part of the process.

37 The audit team also confirms on the final day any discipline areas on which it wishes to seek specialist advice, having notified the institution, where possible at the start of the penultimate day, of any areas in which this is likely. When specialist advice is to be sought, the team's findings, judgements and recommendations on the final day are provisional.

38 There is no oral report to the institution at the end of the visit, but a letter is sent to the head of the institution within two weeks, outlining the main findings of the audit and the likely recommendations in the draft report. If specialist advice is to be sought, a letter is sent to the institution confirming the remit of the specialist advisers, but the institution is not informed of the main findings of the audit until after the advisers have reported back to the audit team.

Discipline audit trails

39 Discipline audit trails have three principal purposes:

• they provide verification that the institution's quality assurance mechanisms are operating in the manner intended;
• they provide a window through which the audit team can consider aspects of what is actually being achieved by students and the effectiveness of the support offered to assist their learning;
• they provide a direct means of comparing the claims made by the institution for the accuracy, completeness and reliability of the information that it provides about quality and standards, with the experience of students and others who have actually used it.

40 The number of discipline audit trails for each audit is determined by the Agency on the basis of the size and breadth of an institution's provision, as measured by the number of JACS Subject Groups and the number of students. As a general guide, the trails are expected to represent some 10 per cent of the institution's higher education programmes in terms of full-time equivalent student numbers. During the transitional period, 2002-05, it is expected that most institutions (with the exception of small specialist institutions, for whom other arrangements will be made) will have between four and six trails.

41 The initial identification of possible discipline audit trails is made by the assistant director. The scope of each trail takes into account the format used by the institution in its own model for internal periodic review - for the purposes of the trails, a 'discipline' may cover a programme, a cluster of related programmes, a field of study, a department, or another unit of review. Care is taken to ensure that the interests of students in areas of small or very large provision are not overlooked. The final selection of trails is made by the audit team from within the range of possible trails identified by the assistant director.

42 There are several possible reasons for choosing a discipline for trailing. It may be chosen because:

• it offers a recent illustration of institutional processes for assuring the quality of programmes and the standards of awards;
• it appears to offer particularly interesting or innovative features;
• there is a lack of clarity in the institutional SED about particular aspects of the quality assurance arrangements, which might be better illustrated for the team through examination of a particular discipline;
• there are indications in other documentation (including reports on the institution by the Agency and other relevant bodies within the six years preceding the audit) of a possible or identified weakness;
• when taken together with the other disciplines selected, it enables the audit team to sample an appropriate range of the institution's provision.

Provision that is subject to another form of review by the Agency (see paragraph 26), or is to be reviewed separately under contract from another body, or is funded by the Teacher Training Agency, will not be identified for a trail.

43 Discipline audit trails are interspersed with more general enquiries during the course of the audit visit and involve two auditors, at least one of whom has current or recent experience in a discipline within the same JACS Subject Group. Normally around 25 per cent of auditor time during the visit is allocated to the trails.

44 The trails comprise five elements:

• a short SED, to be available to the audit team seven weeks before the audit visit. A recent internal review report (or similar) covering the area of the discipline audit trail, accompanied by the relevant programme specifications, is normally sufficient for this purpose. Guidance on the nature of discipline SEDs is provided in Annex C;
• provision of a limited amount of illustrative documentation, some of which may be requested for circulation to the team in advance of the visit. The documentation may be drawn from the unpublished information set identified in HEFCE 02/15 or more widely, but is limited in all cases to no more than is necessary to inform the team's proposed lines of enquiry;
• discussions between the team and staff and students (at discipline level) about the ways in which the institution's quality assurance policies are implemented and their perceived and actual effectiveness. The discussions focus upon a small number of specific topics identified by the team, but also provide an opportunity for staff and students to bring other matters to the team's attention. A small number of external participants in internal reviews may be asked to participate in the discussions;
• scrutiny of the accuracy, completeness and reliability of the information that the institution provides to potential students, employers and other stakeholders about the quality of its programmes and the standards of its awards. This includes discussions with staff and students about programme specifications;
• scrutiny of the relationship between the programmes offered and the FHEQ, relevant Subject benchmark statements, and relevant sections of the Code of practice. This includes discussions about the quality of teaching and learning and the standards achieved by students, and draws upon the primary evidence provided by a sample of students' work (coursework and examination scripts) that has already been assessed and is the same as, or similar to, the most recent sample provided by the institution for the relevant external examiner(s). A small number of those external examiners, and/or external participants in internal reviews may be asked to participate in the discussions.

45 Each audit trail results in a conclusion by the audit team about the extent to which the institution's quality assurance arrangements are operating in practice, at discipline level, in a way that ensures acceptable quality and standards. The normal expectation is that the evidence seen by the team will confirm the institution's comments in the relevant discipline SED. In the event that the discipline SED indicates a significant weakness in arrangements, the team will seek to satisfy itself that the institution is taking appropriate action to address that weakness.

46 In certain circumstances, the audit team may find itself unable to reach a conclusion within the context of the audit visit. Such circumstances are likely to involve:

• the identification of potentially excellent practice that the team is unable to confirm without advice from specialists in the relevant discipline; or
• the identification of significant weaknesses, including possible shortcomings in the effectiveness of the facilitation of student learning, that the team feels unable to confirm without advice from specialists in the relevant discipline; and/or
• the identification of significant apparent discrepancies between the institution's published information relating to the discipline and the findings of the team.

47 In these circumstances, and following consultation with the assistant director, the audit team informs the institution at the start of the penultimate day of the visit (providing the trail is largely complete by that time) that it is unlikely to be able to reach a conclusion without a second opinion from specialist advisers. This period of notice provides the institution with an opportunity to supply the team with further information before the final day of the visit. If, on that final day, the team confirms its intention to seek specialist advice, a team of at least two specialist advisers is asked to make a separate visit to the discipline area as soon as possible. The team will not make a negative summative judgement on a discipline without reference to specialist advisers.

48 The remit of the specialist advisers is to undertake further study of the discipline area, looking in depth at particular aspects indicated by the audit team. Their work includes scrutiny of primary evidence such as assessed student work, and involves meetings with students and staff, and possibly with external examiners. Where their specialist advice has been sought on possible shortcomings in the effectiveness of the facilitation of student learning, their work also includes direct scrutiny of the interaction between academic staff and students. Their findings are not reported separately but are shared with the audit team so as to inform the judgements made by the team in its final report. The drafting of the report proceeds while the specialist advisers undertake their work, but the draft is not submitted to the institution until the team has considered their findings.

49 If, during the course of their work, the specialist advisers identify potential areas for concern beyond their discipline remit, the audit team may wish to conduct further discussions with the institution.

50 The findings of the specialist advisers may result in a recommendation from the audit team that there should be a full review within the discipline concerned. Such a recommendation may be made when the findings of the specialist advisers:

• indicate that it is not possible to reach a conclusion about the health of the discipline without further, detailed scrutiny; and/or
• indicate that there is reason for serious concern that the quality of programmes and/or the standards of awards are at risk.

51 Under these circumstances, the Agency may conduct a separate review under the procedures described in the Handbook for academic review.

Thematic enquiries

52 Thematic enquiries are explorations of the way in which aspects of the institution's quality assurance procedures work across the institution as a whole. They may be undertaken if an audit team considers that an aspect of an institution's management of quality and standards is particularly interesting or requires checking across several disciplines. They may be linked to consideration of specific sections of the Code of practice.

53 Confirmation of the areas for thematic enquiry takes place at the briefing visit and the institution is notified of any areas selected shortly afterwards, as part of the programme for the audit visit.

54 Evidence in respect of thematic enquiries may be obtained both through discipline audit trails and through the audit team's discussions with staff and students at institutional level. If, during the course of a thematic enquiry, the audit team identifies questions at a discipline level on which further advice is needed, it may seek the views of specialist advisers under the arrangements described above, paragraphs 44-51.

Use of reference points

55 When considering the institution's management of quality and standards, the audit team draws upon a range of external reference points, including the FHEQ, Subject benchmark statements and the Code of practice (see Annex I). In so doing, it is not seeking evidence of compliance, but rather for evidence that the institution has considered the purpose of the reference points, has reflected on its own practices in the relevant areas, and has taken, or is taking, any necessary steps to ensure that appropriate changes are being introduced:

• in respect of the FHEQ, the team considers the institution's procedures for relating its programmes and awards to the appropriate level of the FHEQ, drawing upon the discipline audit trails for more detailed information and evidence of practice;
• in respect of the Code of practice, the team does not seek information about adherence on a precept-by-precept basis. It expects to see a statement in the institutional SED about how the intentions of the precepts have been addressed, and to discuss during its visits any key changes that the institution has made to its practices and any areas that have caused particular difficulty. During discipline audit trails and thematic enquiries, the team may request evidence in support of the statement in the institutional SED. In respect of collaborative provision, its discussions may include reference to HEFCE's Indirectly funded partnerships: codes of practice for franchise and consortia arrangements (2000);
• in respect of Subject benchmark statements, the team enquires into the way in which the statements have been taken into account when establishing and/or reviewing programmes and awards, and may request evidence of practice during discipline audit trails and thematic enquiries. The Agency views the statements as authoritative reference points, but not as definitive regulatory criteria for individual programmes or awards.

Judgements and reports

56 The audit results in a report published by the Agency. The concluding section of the report sets out the audit team's judgement on:

• the confidence that can reasonably be placed in the soundness of the institution's present and likely future management of the quality of its programmes and the academic standards of its awards.

The judgement is based on a number of factors, including the extent and degree to which the team concludes that quality and standards are managed successfully, with reference to the institution's individual situation, context and mission, as well as to external reference points; and the team's direct scrutiny of academic standards through primary evidence.

57 The judgement provides one of three expressions of confidence - 'broad confidence', 'limited confidence' or 'no confidence' - the detailed criteria for which are set out in Annex G. The statement of confidence is, in essence, a judgement of probability: it cannot be unconditional. In general terms, where the audit team judges that the institution is managing quality and standards soundly and effectively and that its future capacity for maintaining quality and standards appears good, broad confidence is expressed. Where the team has doubts, either about the current assurance of quality and standards, or about the institution's capacity to maintain quality and standards in the future, it expresses limited confidence. Very occasionally, a team may make a judgement of no confidence in an institution. The team is required to indicate clearly the areas of concern that have given rise to any limitation of confidence and the reasons for its judgement.

58 The concluding section of the report also sets out the audit team's judgement on:

• the reliance that can reasonably be placed on the accuracy, integrity, completeness and frankness of the information that the institution publishes about the quality of its programmes and the standards of its awards.

This judgement takes into account the team's findings in respect of the provision it has considered in the discipline audit trails, augmented where necessary by the advice provided by specialist advisers. It contributes to the confidence judgement. The Agency is aware that institutions will need time to meet all of the requirements of HEFCE 02/15 and will provide advice as appropriate to audit teams visiting institutions early in the audit cycle.

59 In making these judgements, the audit team gives particular attention to the Agency's expectations in two key areas. The first expectation is that the institution is making strong and scrupulous use of independent external examiners in summative assessment procedures. The second is that a similar use is made of independent external persons in the internal periodic review of disciplines or programmes. The team is unable to make a judgement of broad confidence in an institution if either of these elements is seriously deficient.

60 The two judgements are accompanied by recommendations for consideration by the institution, categorised in order of priority:

• 'essential' recommendations refer to important matters that the audit team believes are currently putting quality and/or standards at risk and which require urgent corrective action;
• 'advisable' recommendations refer to matters that the audit team believes have the potential to put quality and/or standards at risk and require preventive, or less urgent, corrective action;
• 'desirable' recommendations refer to matters that the audit team believes have the potential to enhance quality and/or further secure standards.

61 The concluding section of the report may also highlight features of good practice in the management of quality and standards at institutional level and within the disciplines selected for audit trails.

62 The report provides comment on other matters, including the characteristics, strengths and limitations of the institution's internal quality assurance methods, and the quality of programmes and standards of awards achieved in practice, drawing upon the findings of the discipline audit trails. A summary statement on each of the trails is provided in the concluding section of the report. The report also identifies any area where the audit team considers there is good reason for a full review at the discipline level to be carried out, or where it considers that an action plan at either the discipline or institutional level should be implemented.

63 The draft report is prepared and submitted to the institution as soon as possible following the audit visit, normally within eight weeks. Its production is co-ordinated by the assistant director and its format and contents follow a standard structure (see Annex H). The institution is asked to provide the Agency, within four weeks of receipt of the draft report, with corrections of errors of fact. The final report is prepared in the light of the institution's response.

64 As the published report is intended to provide information of use to both a lay and professional readership, it includes a summary intended primarily for the public, especially potential students, which is available separately from the rest of the report. In addition, the institution is invited to provide a brief statement to be published as an appendix to the report. The statement provides an opportunity for the institution to report on developments since the audit visit, particularly in respect of actions taken or proposed to address the recommendations of the audit team.

65 The normal expectation is that the report is published within 20 weeks of the audit visit. This period may be extended if advice is sought from specialist advisers.

Sign-off and follow-up

66 The audit is completed when it is formally signed off. Where the report makes a statement of broad confidence, the audit is signed off on report publication. A brief enquiry is made by the Agency through correspondence with the institution after one year on the way in which the institution has responded to the report.

67 Where the report makes a statement of limited confidence, the report is published, but there is a programme of follow-up action. The Agency requires an action plan from the institution within three months of the report's publication and, subsequently, a progress report on how the action plan has been implemented. The audit is not formally signed off until the Agency is satisfied that the plan has been implemented successfully, with a maximum time limit of 18 months. If at that point concerns remain about the effectiveness of the remedial action, the Agency conducts a further visit.

68 Where the report makes a statement of no confidence, the report is published, but the programme of follow-up action includes the requirement that the institution submits an action plan to the Agency within three months of the report's publication, and quarterly progress reports thereafter on how the identified weaknesses are being addressed. After 18 months, the Agency carries out a short follow-up visit to the institution to check progress. The audit is not formally signed off until the Agency is satisfied that the action plan has been implemented successfully. If after 18 months concerns remain about the effectiveness of the remedial action, the Agency may bring forward the date of the next audit.

69 Where the report includes a recommendation that there should be a full review within a specific discipline, the review is conducted by the Agency under the procedures described in the Handbook for academic review. The procedures followed in the event that the review results in an expression of limited confidence in academic standards are those described in the Handbook for academic review, and include a further, formal review within one calendar year and, ultimately, a potential risk to funding.

70 Three years after the audit, the Agency expects to make a brief visit to the institution to review progress since the audit and to discuss the institution's intentions in respect of managing quality and standards over the three years until the next audit. In preparation for the visit, the Agency reads all relevant internal review reports produced by the institution in the three years since the audit. If any reports raise matters of serious concern that the institution does not appear to be addressing satisfactorily, the Agency may bring forward the date of the next audit.

71 A summary of the relationship between the audit team's judgements and recommendations, and the follow-up action required by the Agency, is provided in Annex G.

Audit administration and institutional contacts

72 The administration of the process takes place in accordance with an operational manual produced by the Agency. Responsibility for the co-ordination of the audit rests with the assistant director, but the judgements and recommendations resulting from the audit are made by the audit team. However, it is the responsibility of the assistant director to test that the team's findings are supported by adequate and identifiable evidence, and that the audit report provides information in a succinct and readily accessible form. To this end the Agency retains editorial responsibility for the final text of the report.

73 Every effort is made to ensure that a close and constructive working relationship is established with institutions and actively maintained beyond the specific requirements of the audit and related activities. Each institution is invited to nominate a correspondent to liaise with designated staff of the Agency on a continuing basis. The liaison process is separate from the process of audit management and is conducted by a different member of the Agency's staff.

74 The Agency endeavours to protect the quality of the audit process through the adoption of explicit operational principles and service standards (see Annex J) and quality assurance mechanisms. The latter include the opportunity for participants in the process, including students, to provide structured feedback on their experiences.

Complaints and representations

75 Complaints about the conduct of the audit and representations against the judgements made by the audit team are considered by the Agency in accordance with the formal procedures published on its web site (http://www.qaa.ac.uk).

Next > Annexes