Handbook for institutional audit: England

Arrangements during the transitional period 2002-2005 for Higher Education Institutions in England

Introduction

1. The purpose of this note is to provide guidance on the operational arrangements for the transitional period, 2002-2005, and describe the method for the developmental discipline-level engagements.

2. The programme of subject reviews previously proposed by the Agency for individual institutions from 2002 onwards, and set out in the Handbook for academic review published in 2000, will not now take place. However, the Higher Education Funding Council for England (HEFCE) requires that, during the transitional period 2002-03 and 2003-04, those institutions that are still awaiting their first audit (for this purpose, the year in which an audit takes place is excluded from the calculation) should have some interactions with the Agency at the level of the discipline. The benefits to the institution of having these interactions include: the opportunities to test, develop and refine internal review processes; the generation of up-to-date qualitative information that will serve both internal processes and institutional audit; and the opportunity to have first hand experience (through the involvement of nominees from the institution, addressed in paragraphs 11 and 18 below) which might assist further with the institution's arrangements to identify, enhance and disseminate good practice.

3. The majority of these discipline-level interactions will be developmental engagements. The main purpose of these engagements is to provide an opportunity for institutions to test, in co-operation with the Agency, the strength of their internal review procedures at the level of the discipline or programme, and the robustness of the evidence they use in those procedures. In the interests of enhancement, these will lead to threshold judgements on standards and the quality of learning opportunities in the discipline and, where appropriate, matters for further consideration by the institution. They will employ a standardised method, derived from the principles and methods which will be adopted in the discipline audit trails that will accompany future institutional audits (please refer to paragraphs 39-51 in the Handbook for institutional audit: England).

4. A smaller number of discipline-level interactions will take the form of subject review, using the method described in the Agency's Handbook for academic review. This sets out the opportunities for applying the new method with flexibility and with a minimum of additional burden on institutions that have robust internal review and reporting processes. The method will also be applied giving due emphasis to the importance attached to the review process making a contribution to the continuing improvement of academic standards and the quality of learning opportunities within the subject. Further guidance on the part that subject reviews can play in the enhancement of learning and teaching will be provided for those institutions undertaking them.

5. Reports of subject reviews and developmental discipline-level engagements will be submitted to the institution and to HEFCE and their findings will be used as part of the evidence base that informs a subsequent institutional audit. Subject review reports will be published. The reports derived from developmental discipline-level engagements will not be published.

6. Discussions will be held with individual institutions to agree a programme of discipline-level interactions in the transitional period, informed by agreement on the timing of the institutional audit and the criteria for the eligibility for the different types of discipline-level interactions.

Subject Review

7. A small number of discipline-level interactions will take the form of subject reviews, using the method described in the Handbook for academic review. The 42 units of review listed in Annex K of that Handbook will be applied to subject reviews. However, when determining the scope of reviews, account may be taken of structure and pattern of provision and their preferred lines for internal reporting. Between 2002 and 2005, subject reviews will be carried out in institutions that, under the 1995-2001 subject review programme, received:

either two or more subject reviews with the profile totalling 17 points or less;
or two or more subject profiles containing two or more Grades 2.

8. In these institutions, the provision that is eligible for review will be:

subject areas not previously reviewed by the Agency or HEFCE, from which at least one complete cohort has graduated;
subject areas which, under the method for the assessment of the quality of higher education applying prior to 1995, received a 'satisfactory' judgement from HEFCE based on the self-assessment report, but were not visited;
all subject areas which received three or more Grades 2 in the post-1995 review programme.

9. Subject review using the method described in the Handbook for academic review may also apply where:

the institution requests one or more full subject reviews for its own purposes. For example, it may be that an institution seeking degree awarding powers or university title might wish to have the fullest possible external evidence of high quality and standards, and therefore take the initiative to request the subject review method;
the relevant Professional, Statutory or Regulatory Body (PSRB) requires that the subject review method be used as a basis for its decisions about accreditation. That is a matter for the PSRB to decide.

10. The arrangements for subject review will include discussions with the institution concerning the scope of the review and the availability of the outcomes of internal reviews. It is not the intention to revisit any subject areas that received a Grade 1 in the post-1995 review programme, because they will have already been revisited and action will have been taken either to improve or to close the programmes concerned. A schedule of subject reviews will be agreed with each institution that comes under the criteria set out in paragraphs 7 and 8 above, when the schedule of institutional audits has been agreed. The method of review will be that described in the Handbook for academic review. If the institution presents a record of continuing improvement, for example in a recent continuation audit, to a subject eligible under paragraphs 7, 8 and 9 above, the Agency will be willing to consider applying the subject review process with a lighter touch in line with the Handbook for academic review.

11. The value of the role of the subject review facilitator to the review process is generally acknowledged. Although the nomination of a facilitator is voluntary, institutions will be encouraged to take advantage of this opportunity during the transitional period.

12. Guidelines for producing self-evaluation documents, programme specifications, the role of the subject review facilitator, and on the availability of supporting documents will be found in annexes C, D, E, F and H of the Handbook for academic review. Institutions may find it helpful to take up the invitation contained in the Handbook for academic review to draw upon existing internal review reports and other relevant evidence in preparing its self-evaluation.

Developmental discipline-level engagements during 2002-03 and 2003-04

Eligibility and selection

13. Institutions whose institutional audits are not scheduled for 2002-03, and which are not required to undertake subject reviews during 2002-05 (see paragraphs 7-10 above), will undertake a limited programme of developmental discipline-level engagements. All these institutions will be expected to have at least one engagement, and not more than two, in each of the years before they have their institutional audit. Institutions may put forward a proposal for up to four developmental discipline-level engagements in the period commencing 2002 up to a year prior to their scheduled institutional audit. Institutions scheduled for institutional audit in 2003-04 will be expected to have one or two discipline-level developmental engagements in 2002-03. Institutions scheduled for institutional audit in 2004-05 will be expected to have one or two developmental engagements in 2002-03 and one or two in 2003-04. Using the Subject Groups defined in the new Joint Academic Coding System (JACS) (http://www.hesa.ac.uk) for reference, a programme or group of programmes within a discipline area will be identified for each engagement.

14. Discipline areas that will have priority for a developmental engagement are those which include programmes that:

were not previously reviewed by HEFCE or the Agency, and from which at least one complete cohort has graduated; and
under the HEFCE quality assessment method applying prior to 1995, were included in subjects that received a 'satisfactory' judgement from HEFCE on the basis of the self-assessment report, but were not visited.

15. The disciplines selected for these developmental engagements will be agreed during the course of preliminary discussions between the Agency and the institution. It is not intended that all discipline areas that meet the criteria for an engagement should have one. Where no discipline can be identified that meets either of the two criteria above, the discipline will be chosen by the Agency in consultation with the institution. In practice, a discipline area will normally be a programme of study or a cluster of related programmes leading to a named award or a cluster of related named awards.

The process

16. The process of the developmental engagement will, so far as possible, reflect the principles which will apply to institutional audit and discipline audit trails, in order that they can be used developmentally during the transitional period to assist institutions to refine their internal procedures and to prepare for institutional audit. The process will employ the academic infrastructure documents, including appropriate uses of the external reference points, and capture the students'view.

17. The developmental engagement will begin with the preparation of a short self-evaluation document (SED) on quality and standards. The guidelines for producing a discipline-level self-evaluation report are provided in Annex C of the Handbook for institutional audit: England. The students' representative body - normally the Students' Union, or its equivalent - has the opportunity to make a written submission in advance of the visit. There will be a period of preparation by the team, using the SED prepared by the institution as the main source of information. The Agency's Information Unit will provide the team with an analytical digest of information. An initial discussion with the institution may be advantageous. During the preparation stage, members of the team will be expected to communicate freely, preferably by email, on their preliminary views of matters arising from their reading of the SED. The period of preparation will be followed by a two-day visit of either two consecutive or two single days to the institution. Following the visit, the team will produce a report which will be submitted to the institution and the HEFCE. The report of a developmental engagement will make two threshold-based judgements, expressing confidence, or otherwise, in the academic standards set and achieved in the discipline, and in the quality of the learning opportunities available to students. Reports of the developmental engagements will not be published, but their findings will be used as part of the evidence base that informs the subsequent institutional audit. If, however, the engagement finds evidence of serious cause for concern, the report will recommend that a subject review as described in the Handbook for academic review is undertaken.

Team composition

18. The composition of the teams for discipline-level engagements will be in keeping with their developmental nature and in support of the linkages between the discipline and the institutions' internal processes. It is envisaged that most teams will combine internal and external membership, together with specialist expertise in the discipline and the institutional perspective. Each team will consist of a coordinator, a specialist and an institutional auditor appointed by the Agency and may also include a nominee of the institution. If an institution chooses not to nominate a member of the team, no extra member will be provided by the Agency. Institutions can nominate any employee they wish; while some may want to nominate a colleague who is or has recently been a specialist in the discipline area, others may prefer a member who represents the institutional perspective. In either case, the internal member of the team is expected to apply their knowledge of the institution's policies, systems and procedures for quality assurance. The coordinator will have a particular role in preparing the engagement, coordinating the process and finalising the report. Each member of the team may offer more than one area of expertise so that, for example, the institutional nominee may contribute specialist knowledge or an institutional perspective or both, and the coordinator and the institutional auditor may have expertise in the discipline. The JACS codes, the self-evaluation and preliminary discussions between the Agency and the institution will be used to determine appropriate expertise within the discipline areas. All team members are expected to play a full and equal part in the engagement. The members appointed by the Agency will be selected against criteria that reflect the process of developmental discipline-level engagements, and they will have received training. The Agency will also provide training for the institutional members.

Self-evaluation

19. Self-evaluation is central to a developmental engagement. The institution will be invited to provide a document or a set of documents that presents a summary of the internal review processes and the outcomes of the internal review in the discipline. In addition, it will supply the relevant programme specifications. A recent internal report on the review of a programme or cluster of programmes within the discipline might well be sufficient to serve. If it so wishes, for example should the internal documentation for any reason be unsuitable for this purpose, the institution may prepare a self-evaluation document (SED). Institutions will note the advantages of ensuring that their internal processes are capable of generating evaluative reports that can serve, with minor adaptation, both for developmental engagements and for the discipline-level audit trails of institutional audits. The institution may wish to consult Annex C of the Handbook for institutional audit: England and the proposed framework for the developmental engagement report in paragraph 28 below to consider an appropriate framework and format.

20. Although no strict word count will be imposed, as a guide to its brevity a set of papers presenting self-evaluation derived directly from internal review, and offering an introduction or overview and index, should be no more than approximately 40 pages excluding the programme specifications. If employing a freestanding SED prepared especially for the engagement, a guide of 3,000 words maximum excluding programme specifications should be applied. The set of papers or SED should serve all the programmes within the discipline. The emphasis should be on the evaluation of the effectiveness of the curricula and arrangements for assessment in supporting the stated aims and intended learning outcomes, and of the student achievement of the appropriate academic standards, and an evaluation of the learning opportunities offered to students to support their learning and achievements. Description of the programme(s) should be the minimum necessary for the team to be able to understand the background of the self-evaluation. It is likely that the programme specifications appended to the papers or the SED will cover most or all of the descriptive material that will be needed.

21. Institutions are encouraged to make as much use as possible of summaries of evidence available from internal review processes and other sources, including structured comments from present and past students, to form the basis of the self-evaluation. Institutions need not restrict themselves to the information set described in Information on quality and standards in higher education: Final report of the Task Group (HEFCE 02/15) - the Cooke Report. Equally, it is understood that in the early part of the transitional arrangements, the full range of the information set may not be available.

Supporting documentation

22. The method proposed for the conduct of the developmental engagement places proper responsibility on institutional staff for the provision of evidence to support the self-evaluation. This principle is fundamental both to institutional audit and to the discipline-level engagements. Supporting documents which the team might expect to have access could include:

programme approval (validation) and review reports;
programme or subject handbooks;
learner support materials, including module or unit guides;
student handbooks;
records of staff-student liaison committees, or equivalent;
assessment criteria and guidance to markers;
samples of students' work;
examination board minutes;
external examiners' reports (typically for the last three years);
student feedback summaries;
outcomes of consultations and other engagements with students;
recruitment and progression data, including into employment;
staff development documents relating to the provision;
PSRB accreditation reports, if relevant.

23. There will be no requirement for a documentation 'base-room' of a type familiar to institutions with the previous style of Agency review, as all these documents should be to hand within institutions. The team will request documents only when required. Institutions may wish to consider granting electronic access to records (eg through its intranet) to the team. The Agency's Information Unit will provide the team with the institution's published information set, as described in HEFCE 02/15.

The visit

24. The visit of the team to the institution will take the form of a developmental engagement with a series of discussions between the team and staff engaged in the teaching and management of the programme(s), and between the team and a representative group, or groups, of students. The visit might be two consecutive days or two separate days, by agreement with the institution. Discussions will normally focus on specific issues derived from the self-evaluation and its analysis, and from issues identified during the initial discussions with the institution. Institutions will recognise that the quality of their self-evaluation and the reporting that arises from internal review processes will have a strong influence on the nature and coverage of discussions during the visit. As with discipline audit trails and thematic enquiries in institutional audits, the focus of these developmental engagements will be on the institution's own procedures, and its evaluation of them, for setting, monitoring, maintaining and improving quality and standards, with an emphasis on the outcomes - the quality of programmes as experienced by students and the academic standards they achieve. This is a key feature of commonality between the developmental discipline-level engagements and institutional audits.

25. During the visit, the team will gather evidence, documentary and oral, with some reference to primary evidence, to enable it to produce its report (see below, paragraph 28). The team will examine the basis of information about the quality and standards of programmes provided to potential students, employers and other stakeholders, including the completeness, accuracy and usefulness of the programme specifications. The approach taken by the team will vary in detail from one engagement to another, but teams are likely to draw upon aspects of the aide-mémoire for subject review, set out in Annex E of the Handbook for academic review. Annex E is, however, designed to support full subject review, and developmental engagement teams will take a 'lighter' and more selective approach, focusing on outcome standards, the quality of the learning opportunities and the means of achieving and assuring them.

Judgements

26. Using the self-evaluation documentation as the prime source of information, the team will evaluate the quality of learning opportunities and the outcomes in terms of the academic standards achieved by students for the programme(s) involved in the developmental engagement. The team's evaluation will be in terms of a general overview rather than very detailed scrutiny. The outcome of developmental engagements will not overturn or replace existing published outcomes derived from earlier subject reviews. Judgements will take the form of information that the institution can benefit from, and incorporate statements of:

'confidence', or otherwise, in the academic standards set and achieved, for the programme(s) involved in the engagement. The judgement focuses on whether the learning outcomes of the programme(s) are appropriate in content and academic level for the named award(s), and whether actual student achievement is generally consistent with the intended outcomes;
'confidence', or otherwise, in the quality of the learning opportunities that support students in achieving the academic standards of the award(s) to which the programme(s) lead. Any need for differentiation in this confidence statement between the aspects and the programmes covered under the 'quality of learning opportunities' heading will be reflected in the report's conclusions and the recommendations for further development (see below, paragraph 28).

27. In coming to its judgements, the team will make use of oral evidence gained through the visit, and documentary sources of evidence, particularly internal review reports or the SED, submissions from students, external examiners' reports, student and/or programme handbooks and other curricular documents, and samples of student work. Where the team identifies matters relating to the strength of the internal review processes, the self-evaluation reports and the robustness of the evidence base that in its view deserve further consideration by the institution, the report will make recommendations. Where such documents are not available, or their form or content do not allow a satisfactory evaluation to be made, or where there are other causes for concern, the team will report such matters to the Agency. A decision as to whether a subject review is needed will then be made.

Reports

28. The team will be expected to produce a report of no more than 2,000 words on the outcomes of the developmental engagement. The report will normally adopt a standard framework:

an introduction on the process of developmental engagement at discipline level;
the programme(s) covered by the developmental engagement;
a statement of overall educational aims of the programme(s);
an evaluation of the quality of the learning opportunities provided by the programme(s);
an evaluation of the effectiveness of the programme(s) in achieving the intended outcomes and the extent to which the students achieve the programme(s) aims and intended learning outcomes;
the conclusions reached and the judgements made;
the strength of the internal review processes, the suitability of the self-evaluation documentation and the robustness of the evidence base together with any recommendations for further development.

29. A key function of the report is that it should be useful to the institution. The detailed structure of reports may vary within the broad framework set out above to suit local circumstances.

30. Reports of developmental discipline-level engagements will be submitted to the institution and to HEFCE. They will not be published. Recommendations for further development will be aimed at the institutional audience. Their purpose, in keeping with the developmental nature of the engagement, will be to offer the views of the team on how the institution might consider enhancing its current practice, or extending existing good practice, in the quality of provision in the discipline and in securing its academic standards.