The University of Bolton
Institutional Audit

NOVEMBER 2005

RG 216 11/05

Findings

163 An institutional audit of the University of Bolton (the University) was undertaken during the week 21 to 25 November 2005. The purpose of the audit was to provide public information on the quality of the University's programmes of study and on the discharge of its responsibility as a UK degree-awarding body. As part of the audit process, according to protocols agreed with Higher Education Funding Council for England (HEFCE), Standing Conference of Principals (SCOP) and Universities UK (UUK), three audit trails were selected for scrutiny at the level of an academic discipline (DATs). This section of the report summarises the findings of the audit. It concludes by identifying features of good practice that emerged from the audit, and recommendations to the University for enhancing current practice.

The effectiveness of institutional procedures for assuring the quality of programmes

164 The University's procedures for approving, monitoring and reviewing programmes are set out in the comprehensive web-based Quality Assurance Manual (QAM). The audit team considered that, in general, the QAM, along with the guidance given to staff on implementing the procedures contained in the QAM, was a feature of good practice.

165 The Validation Handbook is part of the QAM, and details the two-stage process of programme approval and validation, and also clearly sets out the definitions and procedures for major and minor modifications to programmes. In most instances new programme proposals arise from departmental plans, and are tested against their fit with the University's strategic aims. The second stage involves the establishment of a validation panel, the external membership of which is vetted by the Pro-Vice Chancellor (PVC) (Academic Development and Research) to ensure that there is an appropriate level of external practitioner involvement. The Validation Handbook offers little advice on the appointment of external practitioners to validation panels, in contrast to the high quality guidance it provides on other matters.

166 The University operates a four part annual monitoring process involving Programme Quality and Enhancement Plans (PQEPs), Data Analysis Report (DARs) and Subject Annual Self-Evaluation Reports (SASERs), the timetable for which is set by Academic Board (AB). The second part of the process is the production in January of a DAR which is based upon centrally produced annual monitoring statistics. Degree classifications are analysed by entry qualifications, gender, age and ethnicity and some trend analyses are available. Similar breakdowns of the data can be provided for other statistics at the request of the department. The system used measures retention in a uniform way across all courses, enables trend analyses (up to six years) and is compatible with the 'projected learning outcomes' measure produced by HEFCE. Of particular note is the production and processing of data based on both 'leaving cohorts' and 'starting cohorts'. The former allows monitoring of the performance of part-time students in a well founded manner, avoiding the problem of falsely high failure rates when students continue to accrue results over a longer period of time than the norm or move from full-time to part-time registration. The audit team read a number of annual monitoring reports (AMRs) and concluded that it was an effective if somewhat over-complicated process, and that the University may wish to make more explicit the rationale for multiple reports. The team considered that the ability to generate and process reliable data relating to retention, progression and achievement for all students in this way, and the guidance provided for staff on the interpretation of those data was a feature of good practice.

167 The University operates a five/six year cycle of periodic review, the details of which are set out in the Guidelines for Internal Subject/Programme Review, which is part of the QAM. Central to the reviews are the production by the course team of a reflective self-evaluation document (RSED). The reviews are conducted over a period of time and may include observation of course committees and observation of validation events, and as such enable additional depth of scrutiny to occur than in a review of shorter duration. Successful reviews result in revalidation of the programmes included. The SED acknowledged that there had been some slippage in the current cycle of review, but added that extra staff development had been introduced to enable programme teams to better prepare for periodic review. Reports emanating from periodic reviews are in a format so that they can be posted directly to the Teaching Quality Information (TQI) website. The audit team read a number of such reports and considered that while they covered the relevant quality assurance matters, they were also somewhat terse, especially when compared to recent validation reports. The University may wish to consider ways of making the deliberations of the review panels more visible within the reports.

168 The University has recently introduced a system of annual audits of a sample of programmes which are document-based small scale reviews. The audit team read a number of these reports and formed the view that they provided a well-focused and constructive critique of an area of provision.

169 Programmes which are delivered in collaboration with a partner institution or involve an e-learning aspect are largely subject to the same procedures of approval, annual monitoring and periodic review as mainstream programmes. One exception is the extra requirement that a third of the course material of new programmes involving a distance or e-learning element should be available to the validation panel. The audit team considered this additional requirement to be a feature of good practice.

170 The SED stated that the University has 'a variety of systems in place for securing feedback from stakeholders, and the best practice is very effective in providing information leading to improvement and enhancement'. The audit team concurred with the variability of practice in the application of the systems. While the team appreciate that the University is endeavouring to raise performance in certain areas, it would advise it to carefully consider how it can ensure that departments are maximising feedback from students.

171 Overall the audit team concluded that the University's procedures for assuring the quality of its programmes were robust. Through its reading of a number of internal reports and tracking of their consideration through the University's various committees the team considered that the procedures reflected the practice contained in the Code of practice, Section 7: Programme approval, annual monitoring and review and that they enabled AB to be confident about the quality of the programmes that the University provides.

The effectiveness of institutional procedures for securing the standards of awards

172 The University uses a number of mechanisms to secure the standards of its awards including external reference points, monitoring progression and completion statistics and through the use of external subject specialists in its validation, review and assessment processes. The audit team saw extensive evidence of the University's involvement of external subject specialists in validation and review processes. Nominations of externals in these processes are similar to those for external examiners, and it was clear to the team that the University takes careful note of and responds appropriately to the comments of external advisers. However, the team considered that given the University's commitment to vocationally related programmes that it might provide better guidance to departments on the selection of external practitioners for validation panels.

173 Operation of the external examiners system is set out in the Nomination, Appointment and Duties of External Examiners. At both undergraduate and postgraduate level the University operates two levels of examination boards: pathway boards examine results from particular modules while departmental assessment boards make progression and award decisions. It is a University requirement that there is at least one external examiner for each programme and that one 'senior' external examiner has responsibility for commenting on overall standards and assessment processes. As far as possible externals also cover collaborative programmes.

174 AQSU scrutinises departmental nominations of external examiners. New appointees receive a standard set of documents, and in order to ensure consistency a member of AQSU attends the department organised induction. External examiners present their report on a standard pro forma via email to the Vice Chancellor, care of the Head of AQSU. The forms are designed so that the material uploaded to the TQI site is immediately available in a form, which provides appropriate information to potential students and other stakeholders. A second report is required from the senior external examiner, and this includes the components required for publication on the TQI website. At departmental level external examiners' reports are formally addressed during annual monitoring, while the head of department can ask course leaders to respond more urgently to matters raised. Heads of department summarise key points arising and report to standing panels, which in turn report to Academic Quality Development Commitee (AQDC). AQDC gains a separate perspective via an Annual Overview Report produced by the Head of the AQSU, who also attaches comments to each report prior to it is sent to the head of department. External examiners are asked on the report pro forma to state whether previous comments have been noted and adequately responded to. The audit team read a number of external examiners' reports and tracked their passage through relevant departmental and University committees. They also read recent Annual Overview Reports, and concluded that the University's procedures and processes relating to external examiners' and their reports operates effectively at both departmental and University levels. The team noted as good practice the induction of external examiners and the University procedures for considering their reports.

175 Overall the audit team found that the University makes good use of external body reports, management information and external subject specialists in its procedures for setting and securing the standard of its awards.

The effectiveness of institutional procedures for supporting learning

176 Induction of new staff is a University requirement. However, the audit team learnt that the nature and effectiveness of induction of new academic staff varied between departments. However, the team noted that the Human Resources Department is addressing this matter by interviewing all new staff after completing their induction in order to ascertain the effectiveness of induction

177 The main locus of staff support and development is at the departmental level, and each department is required to have a staff Communication, Appraisal, Responsiveness and Evaluation (CARE) plan. Part of the development opportunities for new staff without sufficient teaching experience is the expectation of completing the University's own Certificate in Teaching and Support for Learning (CTSL). The University has recognised a number of problems inherent in the programme and are in the process of revising the curriculum content. The audit team also noted that departments are inconsistent in their operation of the University's system of peer observation.

178 As part of its overall strategy the University has placed significant emphasis on the development of distributed and distance learning methods of programme delivery. To enable staff to increasingly use its VLE in their teaching the University has formed eLab to provide technical support and appointed a Professor of e-learning to provide pedagogical guidance regarding e-learning approaches.

179 The audit team learnt of students' general satisfaction with the level of learning resources provided. The University has thorough processes in place for identifying resource needs for new programmes, and on-going provision of resources is undertaken through a number of mechanisms including through annual monitoring and periodic review, and by central services frequent evaluation of the resources they offer.

180 The Dean of Students has overall responsibility for the development, implementation and monitoring of strategies and procedures related to students support services including academic and personal support and guidance. The audit team considered that these services were generally appropriate to the profile of the student body reflecting the numbers of part-time and mature students, and those that were studying on vocationally relevant programmes. In particular, the team noted as a feature of good practice the University's commitment to providing a warm welcome and on-going support for international students.

181 The audit team learnt of many examples at the departmental level of good practice in the implementation of the University's personal tutoring policy, including offering support to part-time students and those undertaking placements. However, the team learnt that this was not a universal picture. Support for postgraduate research students was generally reported to the team as being of a high quality. One related aspect that detracted from this view was the lack of certainty among senior staff and in some of the University's internal documents about the opportunities for postgraduate research students to undertake teaching. The team strongly encourage the University to adopt a single approach to this matter, and in doing so take account of expectations contained in recent publications of the research councils.

182 The audit team concluded that in general the University has established effective procedures to support student learning. The team learnt of numerous examples of staff commitment to support student learning, and considered that this resulted in part from the University's effective policies and procedures in the appointment and support of staff.

Outcomes of discipline audit trails

Electronics

183 From its discussions with students and staff, and its study of students' assessed work, the audit team formed the view that the standard of student achievement in the above named awards offered by the computing and electronic technology programme area within the Department of Computing and Electronic Technology was appropriate to the titles of the awards and their location within the FHEQ. At undergraduate level BEng (Hons), BEng, HND and HNC in Electronic and Computer Engineering form a coherent programme suite with different entry and exit points and availability in both full and part-time modes. The programme specifications are referenced to the Subject benchmark statement for engineering. In addition, the team found that the standards of student achievement in the two part-time, e-learning, distance MSc programmes, MSc Advanced Microelectronics and MSc Electronic Product Development were also appropriate to the titles of the awards and their location within The framework for higher education qualifications in England, Wales and Northern Ireland (FHEQ).

184 The audit team came to the view that the quality of the learning opportunities was suitable for provision leading to the awards listed above. In particular the team considered that the quality of material and support offered to students on the e-learning master's provision to be particularly noteworthy. However, the Department might wish to consider if it is making the best possible use of the annual monitoring cycle, especially module and programme evaluation questionnaires.

Mathematics

185 From its discussions with students and staff, and its study of students' assessed work, the audit team formed the view that the standard of student achievement in the BA/BSc (Hon) Mathematics was appropriate to the titles of the awards and their location within the FHEQ. There was no reference within the programme specifications to the relevant subject benchmark statement, although the links were clearly drawn in the learning outcomes section of the discipline self-evaluation (DSED).

186 The students met by the audit team considered the teaching and support provided by departmental staff to be excellent. The team learnt that there was a high dependence on informal means of communication with students and providing academic support and guidance. Nonetheless, based on its scrutiny of the material made available to it, and from meetings with staff and students, the team considered that overall the quality of learning opportunities were suitable for the programmes leading to the above named awards in Mathematics.

Psychology

187 From its discussions with students and staff, and its study of students' assessed work, the audit team formed the view that the standard of student achievement in the BSc (Hons) Criminological and Forensic Psychology, MSc Psychology, and MSc Critical Psychology, was appropriate to the titles of the awards and their location within the FHEQ. Although the programme specifications did not make formal reference to the subject benchmark statement extensive use had been made of the requirements of the British Psychological Society, in particular with respect to the design of the core curriculum.

188 From scrutiny of the material made available to it, and from meetings with staff and students the audit team concluded that the quality of learning opportunities were suitable for the programmes leading to the named awards in psychology. In particular the team noted the department's careful approach to providing personal tutors for its students and the importance attached by students to the Science Resource Centre in aiding their learning.

The use made by the institution of the Academic Infrastructure

189 Overall it was clear to the audit team that the University had responded to the various sections of the Academic Infrastructure in a coherent and appropriate way. New or updated sections of the Code of practice for the assurance of academic quality and standards in higher education (Code of practice) are initially considered by the relevant management post holder. The University's existing procedures are reviewed against the Code, and any recommendations for change are considered in detail by AQDC and AB. The audit team learnt of a number of changes to the University's procedures resulting from publication of the Code and concluded that the University had approached the Code in a positive manner, and regarded them as a set of guidelines of good practice in the sector.

190 The University already had in place formal level descriptors for sometime preceding publication of the FHEQ and thus the University considered that it was relatively unproblematic for course teams to map their intended learning outcomes onto the FHEQ. Explicit reference to the FHEQ is a requirement of the validation and periodic review processes, and external examiners' in their reports are required to specify whether standards achieved by students are at the appropriate level.

191 Validation and periodic review processes have been revised so that they necessitate reference to subject benchmark statements, and new programmes cannot be approved without the production of a programme specification. The SED claimed that nearly all programmes now have a programme specification. Some of the programme specifications seen by the team did contain reference to subject benchmark statements and a mapping of pathway learning outcomes onto the levels of the course. The audit team did, however, note some variation in the quality of the specifications.

The utility of the SED as an illustration of the institution's capacity to reflect upon its own strengths and limitations, and to act on these to enhance quality and standards

192 The audit team found the self-evaluation document (SED) to be a well-organised and comprehensive document, which provided a good basis for the audit. The structure of the SED reflected the structure of the institutional audit report template. Each section outlined the relevant University quality assurance arrangements, provided examples of good practice and issues and intended enhancements. In most sections there was a clear statement of overall evaluation of the relevant aspect of the quality assurance framework. It was also comprehensively referenced, and as such the team found the SED to reflect the principles which the University claims underpins its approach to quality assurance, namely openness to internal and external sources of critical comment and maturity to behave as an academic community capable of reflection and self-criticism.

Commentary on the Institution's intentions for the enhancement of quality and standards

193 During the audit the audit team learnt of the extent of progress on many of the University's planned enhancements of quality and standards. The team considered that overall the University's plans for enhancement were appropriate, and that these plans provided it with confidence that the University was an institution committed to development and improvement the student experience.

Reliability of information

194 During their meetings with students the audit team heard that in almost all cases they were very satisfied with the information available to them. Prospectus and pre-course material had been received in a timely fashion and was accurate. Overseas students reported no problem in accessing course detail. All students had received student handbooks and module guides, and members of an e-Learning programme, all of whom were mature and returning to formal learning after a gap, praised both the availability and the usefulness of on-line induction material as well as the quality of the material provided to them on-line. During its reading of a large number of University publications and internal documents and websites the team found that the material was accurate, complete and frank.

195 The University has generally been punctual in ensuring that it has met the expectations set out in the requirements for publishing TQI set out in HEFCE 02/15 and HEFCE 03/51. Hence, its Learning and Teaching Strategy, external examiner reports and feedback from recent graduates are available on the HERO site. The audit team noted that the format used for external examiners' reports ensured that the summaries provided on the HERO site did full justice to those reports. At the time of the audit visit summaries of those internal reviews completed since the deadline for TQI had either been submitted for uploading or were awaiting approval by AB. Based on the evidence that the team had available to it, it concluded that the University is meeting the requirements of the HEFCE 03/51, and that published University information was reliable, accurate and complete.

Features of good practice

196 Of the features of good practice noted in the course of the audit, the audit team noted the following in particular.

the comprehensiveness of the web-based Quality Assurance Manual and associated documents and the guidance given to staff on implementing the procedures that these contain (paragraph 22);
implementation of the University's Assessment Process Handbook (paragraph 32);
the developments in e-learning and the validation and quality control of e-learning materials (paragraphs 39 and 150);
the ability to generate and process reliable data relating to retention, progression and achievement for all students and the guidance provided for staff on the interpretation of those data (paragraphs 46 and 47);
the induction of external examiners and the University procedures for considering their reports (paragraphs 62-67); and
the proactive approach taken to the identification of support requirements for international students (paragraph 122).

Recommendations for action

The audit team considers it advisable for the University to:

review the extent to which departmental implementation of its policies and procedures is sufficiently consistent, in particular those relating to module evaluation and personal tutoring (paragraphs 30, 86, 50, 95, 103, and 140)
ensure that it continues to reflect creatively on the mechanisms for attaining student representation and involvement, especially at boards of studies and senior University committees (paragraphs 84 and 145).

The audit team considers it desirable for the University to:

attain greater consistency of routine industrial or employer liaison in vocationally relevant programmes (paragraphs 38; 89)
develop an editorial policy for publishing material through the University website and a process for ensuring the implementation of that policy (paragraph 159).

Appendix

The University of Bolton's response to the audit report

The University welcomes the findings of the audit report and the confidence it places in our management of the quality and standards of our academic awards together with the quality of the student experience. We also welcome the fact that the three discipline audit trails undertaken by the audit team, in Electronic Engineering, Mathematics and Psychology, confirmed the appropriateness of the academic standards achieved by our students and the suitability of the quality of learning opportunities available to students.

Furthermore, we note with great pleasure, the six features of good practice identified by the audit team.

There are no recommendations for action deemed to be essential. Our response to the two advisable and two desirable recommendations are as follows:

With regard to the consistency of departmental implementation of University policies and procedures relating to module evaluation, the Academic Quality and Development Committee and the Pro-Vice Chancellor (Academic) will review practice and require full implementation of University policy by all University Departments. On the related point of consistency of personal tutoring arrangements, the Learner Experience Committee already has this issue under active review.
With regard to the recommendation to reflect creatively on mechanisms for attaining student representation and involvement especially at senior committees, we will use our successful joint project on Student Representation with the Students' Union to focus on improving representation and participation at departmental and University levels.
With reference to attaining greater consistency of routine industrial or employer liaison in vocationally relevant programmes, we believe that the University already achieves high standards in this respect and that we employ a range of proactive methods to ensure industrial and employer involvement. The recommendation that this might be improved by greater use of formal liaison panels at programme level (para 89 of the report) is one we will consider seriously. However, we would not regard it as necessarily the most effective or only indicator of employer input to ensuring the vocational relevance of our programmes.
Finally, in response to the recommendation to develop an editorial policy for publishing material through the University website and ensuring its implementation, the recently appointed Pro-Vice Chancellor with specific responsibility for internal and external communications is already making improvements to the editorial control and quality of the University website and will ensure compliance with a revised editorial policy.

In conclusion, the University takes seriously the conclusions of the audit report and finds a wealth of useful peer commentary in its contents which will contribute to our own critical review processes. As we declared in our self-assessment document, two cardinal principles of our quality and standards procedures are 'openness to a wide range of internal and external sources of critical comment' and 'self-accountability and responsibility'. It is in this spirit that we both welcome the audit reports findings and will respond actively to them as part of our process of continuous review and quality enhancement.

ISBN 1 84482 472 1

The University of Bolton Institutional Audit