204 An institutional audit of the University was undertaken during the week commencing 10 November 2003. The purpose of the audit was to provide public information on the quality of the University's programmes of study and on the discharge of its responsibility as a UK degree-awarding body. As part of the audit process, according to protocols agreed with HEFCE, SCOP and UUK, four DATs were selected for scrutiny. This section of the report of the audit summarises the findings of the audit. It concludes by identifying features of good practice that emerged from the audit, and recommendations to the University for enhancing current practice. With regard to collaborative provision, this report focuses on the University's relationship with the College. The rest of the University's collaborative provision will be the subject of a separate audit.
The effectiveness of institutional procedures for assuring the quality of programmes
205 The University has established a CARP to which staff are nominated. Groups to undertake approval, monitoring, and review processes are appointed from the Panel to either CARTs or AMTs. The course approval process consists of two stages of procedures which involve both academic and resourcing committees at school and University level. It results in a report which goes to a subcommittee of the APC, the QASC, which makes recommendations to Senate. Guidelines for all processes are provided in Handbooks. External input is provided by appointed 'external experts' primarily by written submission.
206 Annual Monitoring Reports are written to a common template by designated members of staff. A standard set of supporting documentation is required which includes external examiners reports and the responses to them. When approved at school level these are submitted to an AMT convened from the CARP. Feedback is given to schools and a written summary report is made by one of four AMTs to the QASC. The audit team concluded that the process for dealing with the reports was sound but that the structure of the report lacked cohesion.
207 Each programme is reviewed on a six-yearly cycle by CARTs put together as for course approval. There is also a more informal process of mid-term review for approval of programme specifications only. Course teams provide a comprehensive set of documentation which includes a written critical appraisal. After the review meeting a report is submitted to the QASC which in turn reports to the APC. The audit team concluded this review process was well informed and robust. There was concern over the form and consistency of the external experts' input into the programme approval process. External experts are only required to provide written input to this process and are not required to attend discussions; the team considered that the University may be missing an opportunity to make best use of the external input to the process. External experts are appointed for three years.
208 Overall, the audit team concluded that processes were lengthy and complicated. There are some concerns about the role of Senate, the format and consistency of the external input and the limited number of staff involved.
209 Student feedback is elicited through unit level questionnaires, stage questionnaires and through SSLCs. The outcomes from stage questionnaires are part of the supporting information for Annual Monitoring Report but the input from unit questionnaires is variable. The SED identified the return rates on stage questionnaires in particular as an ongoing difficulty. Attempts to share good practice in this area have been made.
210 There is no university-wide requirement for schools to seek feedback from graduates or from employers of graduates on the quality of programmes. Some schools have advisory boards drawn from their employment sector which provide informal opportunities for constructive interaction in this area. No structured formal mechanisms exist for obtaining feedback on the quality of programmes from graduates.
211 There is significant distance-learning provision in some schools and collaborative provision figures noticeably in the University's overall undergraduate numbers. The University's policy is that all such provision should be subject to the same procedures as it applies to its internal provision except for the College, to which different but related procedures apply. In the audit team's view the current policy does not recognise the inherent increased risk for the College as a partner institution. There are also significant concerns about the ability of the University to oversee effectively provision in the College leading to a University award.
212 The SED stated that the main quality-related committees together form an important function for assuring quality and standards of awards. It goes on to say that the University recognises that the reporting structure and membership require further qualification to avoid duplication and assure effectiveness. It also says that it believes its Annual Monitoring Review to be effective.
213 The audit team concluded that the quality assurance processes and the associated committee structure had developed gradually over the years to a point where they were now overly complex, hierarchical, and multilayered. Successive changes have been made in response to external developments to the point where the quality assurance arrangements no longer best serve the University's need to have effective, efficient, timely, appropriate and inclusive processes. Given the mutual decision not to merge with the College in the foreseeable future, the team consider it necessary and timely for the University to conduct a complete holistic review of its quality assurance structures and processes paying particular attention to the arrangements with the College and the quality assurance of the provision delivered there that leads to a University award.
The effectiveness of institutional procedures for securing the standards of awards
214 In 1997-98 the University introduced a regulatory framework for its modular scheme. Since then, a tension has emerged between ensuring uniformity across the institution and affording sufficient flexibility to accommodate local needs. This tension was apparent in the attempts by the University to implement institution-wide criteria for the marking and grading of assessed work. The audit team formed the view that this issue will require firm executive action if the assurance of consistent standards is to be achieved within a reasonable timescale.
215 The University distinguishes between the roles of unit and course external examiners. All newly-appointed external examiners are sent a comprehensive information pack and they are invited to attend an induction event. The annual reports of external examiners are considered by the Pro-Vice-Chancellor (Learning and Teaching) and copied to the appropriate departments. Departments are required to report on progress made on issues that have been raised by external examiners in their Annual Monitoring Reports and, once approved, copies of these should be sent to external examiners. External examiner reports are also analysed by the ASSU which on an annual basis prepares summaries for undergraduate and postgraduate provision.
216 The SED stated that external examiners play a key role as arbiters of the standards of programmes leading to awards of the University and, to this end, particular care has been taken in the development of the University's procedures for external examining including the processing of external examiners' reports. The University's view was confirmed by the audit team's enquiries. It was evident that the summary reports are considered thoroughly by senior committees, and discussion within these committees has resulted in a number of appropriate resolutions for further action. Action taken by the University includes a strengthening of its procedures for ensuring that external examiners receive copies of the relevant AMRs. The team viewed this as good practice.
217 Student progression and completion data are issued by the ASSU to schools, and the audit team was furnished with examples of the use of this information in the course of the Annual Monitoring Review process. Although the data set available to staff is reasonably comprehensive, it relates to undergraduate or postgraduate provision within an area rather than to individual courses, although this can be provided by ASSU if required. The University is addressing this issue and it is also seeking to resolve the occasional disparities that arise between centrally-recorded and locally derived information. In general, the team found that there are effective arrangements to provide statistical data for consideration by course teams, but that the University might wish to consider the desirability of furnishing its senior committees with statistical analyses of student progression and completion across the full range of the University's provision, including that which is offered in partnership with other organisations.
The effectiveness of institutional procedures for supporting learning
218 Operating within LSS the University's library and IT facilities are generally well appreciated by students. The library offers extended opening hours and works closely with the academic areas while many of the 369 computer workstations maintained by IT Services are accessible 24-hours a day.
219 The SED provided a comprehensive description of provision in this area and explained the role that these services play in supporting the University's aims and objectives. In recent years there has been substantial investment in the provision of information in electronic format and undergraduates in particular expressed a preference for accessing information in this way. Annual service reviews form an essential part of the annual planning process for the library and IT Services.
220 The phased merger of Management Information Services with LSS will become increasingly important as student information systems need to be more closely integrated with the VLEs system that the University is currently installing. At present it is being left to schools and departments to articulate a clear and strategic objective for the use of a VLE and the University needs to do more to develop and promote a top-level strategic vision of the role that these systems will play in the University of the future. There is a similar lack of any clear strategy for the provision of learning resources in support of the University's mission to widen participation.
221 The variability in the way that the tutorial system operates across different schools and departments in the University was commented on in the 1998 Quality Audit report and the audit team found evidence which showed that the situation has changed little in the intervening period. There appears to be no institutional overview of personal tutorial support and, so far as the team have been able to tell, there is no centrally produced guidance or University CoP on this subject. In its SED the University stated that a review of the effectiveness and expectations of the personal tutor system will need to be carried out in the context of the Excellence Plus Strategy. Excellence Plus is aimed primarily at student self-development and key skills acquisition and has so far been piloted in four of the eight schools. It is intended that the use of Progress Files should be incorporated as an integral part of the personal tutor system. The team recommends that the University should expedite the review of the personal tutor system.
222 Academic support for students engaged in postgraduate research is provided through the appointment of supervisors and, in most cases, there are two members of staff associated with each student. Graduate students who met with members of the audit team spoke highly of the standard of supervision they had experienced.
223 Details of the University's central student welfare services were included in the SED under the heading of Learning Support Resources. This provided an accurate description of the arrangements for personal support and guidance and where relevant the precepts of the Code of practice have been adequately addressed.
224 The University achieved IiP status in March 2003 and the priority it attaches to the effective management of staff and performance is reflected in its Human Resource Strategy. Arrangements are in place for the induction and integration of new staff, all of whom are allocated a mentor for the duration of their probation. Staff conveyed to the audit team that the guidance they had received when they joined the University had been particularly helpful and that mentors had played a valuable role which often extended beyond probation. The University has a formal appraisal scheme for all staff and in recent years this has been transformed from a rather patchy exercise to near complete coverage. The University also promotes recognition of teaching excellence through its annual Chancellor's Awards for Distinguished Teaching with nominations made by students and then supported by staff colleagues.
225 Staff training and development plays a key role in realising the University's strategic objectives by endeavouring to create a workforce able to deliver high-quality teaching and research. The ASSU in conjunction with the TQEG runs a Learning Support Seminar series, and a joint initiative with the College has led to the development of an impressive joint catalogue of training and development events open to members of both institutions. The SED noted that peer observation of teaching had been embedded in University practice and the audit team was able to confirm this with staff.
226 The audit team was impressed by the University's approach to the assurance of quality of teaching staff, through appointment, appraisal and reward. The team also considered that the level and quality of staff development that was offered in the University and the ways in which it is planned, coordinated and delivered was worthy of note.
Outcomes of discipline audit trails
Clinical Sciences
227 The Clinical Sciences DAT reviewed the undergraduate provision of the Foundation year in Clinical Sciences/Medicine (1 year full-time) and BSc (Hons) Clinical Sciences (3 years full-time). The audit team formed the view that the standard of student achievement was appropriate to the title of the award and its location within the FHEQ. Assessment took account of University guidance and was managed with consistency across the programme. The learning opportunities offer a seamless transfer of students to the Medical School of the University of Leeds. Programme specifications are made available to students and are utilised by the Course Management Committee in the annual monitoring process. They could be enhanced by identifying the levels at which the graduate outcomes are achieved and inclusion of all relevant subject benchmarks for each of the identified routes within the award.
228 The course team is committed to the continuous review and enhancement of the programme and demonstrated evidence of their responsiveness to student feedback including actions resulting from the review of module and stage evaluations submitted by students. Module handbooks inform new students of actions taken as a result of previous students' experiences of the module. Students commented favourably on the course team's approachability and its availability in a variety of supportive roles, noting in particular the benefits of regular meeting with their personal tutors. The audit team concluded that the quality of learning opportunities offered is suitable for this multidisciplinary course and located correctly within the FHEQ.
Engineering
229 From its study of students' assessed work, and from discussions with students and staff, the audit team formed the view that the standard of student achievement in the BEng programmes in Mechanical Engineering, Mechanical and Automotive Engineering and Medical Engineering, and the MEng programmes in Mechanical Engineering and Medical Engineering, was appropriate to the titles of the awards and their location within the FHEQ. The programme specifications relate to the Subject benchmark statement for engineering and provide concise information about the programmes in a style that is oriented towards the needs of prospective students.
230 The School has had difficulties with the level of recruitment onto its BEng and MEng programmes and the audit team were concerned about the low progression rates for the programmes that formed the subject of the DAT. Tutorial support and the feedback of comments on assessed work were reported by students to be variable and the School is expecting to address the former through implementation of the University's Excellence Plus Strategy.
231 The School of Engineering, Design and Technology underwent a major restructuring exercise in 2002 although according to reports from external examiners this was achieved without having any detrimental impact on the academic performance of students. As part of this exercise the University invested in new laboratories and teaching space and a substantial injection of funds from industry provided a state-of-the-art CAD suite. Students commented favourably on the quality of the computing facilities and being able to access them into the evening.
232 The audit team concluded that the quality of learning opportunities provided for students was suitable for programmes of study leading to the award of BEng (Hons) and MEng degrees and that these awards are correctly located within the FHEQ.
Management
233 From its study of students' assessed work, and from discussion with students and staff, the audit team formed the view that the standard of student achievement in the MBA programme, the BSc programmes in Accounting and Finance, BSc Marketing, and the BA in Business Studies and Law offered jointly with the College, was appropriate to the titles of their awards and their location within the FHEQ. Programme specifications set out clearly the aims, objectives and learning outcomes of the programmes together with details of how these are to be achieved and the programme delivered. The specifications reflect the relevant subject benchmark statements.
234 Evaluation of the provision by both undergraduates and graduates is very favourable. They appreciated the quality of the environment and the learning resources available to them. Students commented on the good communications which marked their relationships with staff, the quality of feedback they received and their involvement in School affairs. Students on the joint programme were complimentary about their learning experiences but noted that further coordination between the two providing institutions could be beneficial. The audit team concluded that the quality of learning opportunities provided for students, including graduate students, was suitable.
Peace Studies
235 The Department of Peace Studies offers five undergraduate awards and four masters programmes. The standards and content of the assessed work were appropriate to the titles of the awards and their location within the FHEQ, and the high standard of teaching, assessment and student achievement within the Department was confirmed by external examiners' reports. There are effective arrangements for the moderation of assessed work, and students commented favourably on the department's assessment procedures.
236 The information provided for students is detailed and informative, and there is clear evidence that the Department succeeds in its attempts to provide a safe and supportive learning environment that caters for the needs of students recruited with a wide range of entry qualifications and from diverse national and cultural backgrounds. Students commented very favourably on the quality of teaching, which they described as 'leading edge'. The audit team considered that the resources and environment provided by the Department afforded full opportunities for students to achieve the requirements of the awards for which they were registered.
237 Programme specifications were supplied for each of the Department's undergraduate programmes, and a single specification was provided for the postgraduate provision. These are designed to serve as an aid to programme and quality management, and as a source of information for students. The programme specifications are placed on the Department's web page and students informed the audit team that they were valued as an initial source of the information needed for module selection and as a means of determining programme outcomes and their future career prospects.
238 The Department has established effective arrangements to enable students to contribute to the monitoring and development of its provision. These include the appointment of a Student Liaison Assistant and a variety of formal arrangements for eliciting student feedback. Students reported that the informality of their relationships with staff, and the fact that effective and appropriate action is taken in response to student feedback, created a culture which encouraged their active participation in the evaluation of courses.
The use made by the institution of the academic infrastructure
239 The SED identified improved externality as one important feature of the University's approach to the maintenance of academic quality and standards, and stated that is has started to integrate the elements of the national academic infrastructure into its own quality procedures. The ASSU has performed a key role in both evaluating the University's arrangements against the precepts of the Code of practice and in ensuring adherence to any institutional requirements though the medium of its own codes of practice and Quality Assurance Handbook. The evidence available to the audit team indicated that Senate committees had considered the recommendations of the ASSU, and of the various working groups that had been established as each section of the Code was published. This, however, has tended to focus in the procedural implications of the Code. In any future review of its quality strategy, the University may find it helpful to take a longer-term view by considering the more fundamental issues which inform the continuing development of the Code and to ensure that these issues are debated in full by the relevant committees of Senate.
240 The University is currently reviewing the recommendations of a working group on the production of programme specifications, and it is by this means that the University is addressing the implications of the FHEQ. This process has been supported by the early dissemination throughout the University of work within the sector on level descriptors. While there was some variation in the level of detail provided by programme specifications, the majority were both comprehensive and written in an accessible style. The intended purposes of programme specifications are to support the course approval process and to provide information for students, and the audit team was able to confirm the value of the University's programme specifications in both respects.
The utility of the SED as an illustration of the institution's capacity to reflect upon its own strengths and limitations, and to act on these to enhance quality and standards
241 The SED prepared for the audit gave a clear description of the University arrangements for assuring the quality of the provision and securing the standards of its awards. It reflected on aspects of the provision indicating areas of good practice and areas of difficulty. The audit team was able to explore these during the visit.
242 At the time of the visit some of the intended changes indicated within the SED had not yet been implemented. The overall lack of a strategic emphasis within the document with little reference to the development, implementation and monitoring of central policies for quality assurance processes and consistency of academic standards was noted by the audit team and reflected the current situation at the University.
243 The SED reflected the planned merger with the College which was anticipated at the time of its submission. However, this was no longer the case at the time of the visit. The audit team was guided by the SED, but scheduled further detailed discussion with staff at both institutions to review the University's current approach to its relationship with the College.
Commentary on the institution's intentions for the enhancement of quality and standards
244 The SED discussed the enhancement of quality and standards by describing various initiatives the institution intends to pursue. These included reviews of:
- the effectiveness of how the University listens to the student voice;
- the effectiveness and expectations of the personal tutor system;
- the effectiveness of the arrangements for quality related committees and the system of course approval and review;
- the usefulness and effectiveness of the approach to programme specifications; and
- the implementation of the national academic infrastructure.
The audit team discussed these proposed initiatives during the visit and staff the team met were not always aware of how these intentions were to be progressed.
245 The lack of a more strategic approach to enhancement may help to explain why personal tutoring, for example, has not received prompt and effective attention. This issue was first raised in an Annual Monitoring overview in 1997-98 and also identified as an issue in the 1998 Quality Audit report.
246 The role of quality-related committees is discussed in the main body of the report. In the specific area of teaching quality enhancement a more strategic approach has been adopted with the introduction of a Chancellor's Award for Distinguished Teaching and a well-established Teaching Quality Enhancement Group. In the audit team's view both these initiatives make an effective contribution to the recognition and enhancement of teaching quality.
Reliability of information
247 The SED detailed the University's overall position in relation to both the published information set and information required to be available within all higher education institutions by HEFCE 02/15. The audit team found that the University was making progress towards fulfilling its responsibilities in this matter.
248 The University currently publishes information about admissions, progression, award and achievement on the intranet and will release this information into the public arena once assured of its accuracy. The audit team has concerns about the proposed facility to allow public users to manipulate the data, because of the possibility that individual students within small cohorts could be identified.
249 The review documentation and discussion in meetings confirmed that the current published information matches the reality of the provision. With the exception of gaining information from graduates and employers, the University plans to publish data at, or above, the national standards.
Features of good practice
250 The following features of good practice were noted:
i the University's commitment to the regional community (paragraphs 7, 11 and 137);
ii the value of the University's programme specifications in supporting course approval processes and in providing information for students (paragraphs 58 and 240);
ii the University's efforts to combine major course reviews with those required by PSRBs, (paragraph 60);
iii the induction and mentoring of new staff (paragraph 78);
iv the joint training and staff development initiative with the College (paragraphs 86 and 91); and
v the University's commitment to widening participation and to catering for the needs of a diverse student body (paragraphs 137 and 153).
Recommendations for action
251 Recommendations for essential action:
i on the basis of the evidence relating to the partnership with the College, that the University reviews and modifies monitoring processes to ensure effective oversight of its collaborative provision and secures appropriate and formal agreements with collaborative partners (paragraphs 127, 132 and 133).
252 Recommendations for action that is advisable:
i without delay, progress the work to define assessment levels to ensure consistent standards across the University (paragraph 32);
ii without delay, initiate a review of the strategy and structures for the management of quality and standards (paragraphs 42, 46 and 52);
iii review the effectiveness of the structures and processes for annual monitoring of academic provision (paragraphs 45 and 46);
iv in collaboration with the student body, develop effective and transparent arrangements for student participation in all appropriate quality assurance processes (paragraph 65); and
v ensure that the current review of the tutorial system delivers an effective and appropriate level of support across the University (paragraph 108).
253 Recommendations for action that is desirable:
vi consider how the extent to which students feel they are informed of the outcomes of the feedback they provide and the manner in which it is employed could be improved (paragraph 70); and
vii consider furnishing either APC or QASC with statistical analyses of student progression and completion across the full range of the University's provision, including that which is offered in partnership with other organisations (paragraph 74).
Appendix
The University's response to the audit report
The University welcomes the opportunity to provide a statement on developments since the audit visit in response to the recommendations outlined in the report. Following the significant decision made before the audit visit to suspend the proposed merger with the College, the University and Bradford College have established a 'Continuing Collaboration Strategy Group' to address relevant quality related issues.
The Group has prepared an action plan that addresses the issues subsequently confirmed by the audit team. Following on from the regular discussions held with Bradford College during the merger process, it has been agreed to formally align Bradford College's quality assurance arrangements for validated programmes with the University's own rigorous procedures for course approval, monitoring and review for which 'broad confidence' has been affirmed by the audit team.
The draft revised Agency Code of practice; Section 2: Collaborative provision, flexible and distributed learning [including e-learning] when finalised will provide the University with an opportunity to review current practice and management of all our collaborative provision against the precepts through normal procedures developed to respond effectively to external requirements.
A University consultation, initiated before the audit visit, on the implications of the publication and consistent implementation of clear criteria for the marking and grading of assessments has resulted in the preparation of a series of informed recommendations presented to the Academic Policy Committee for further discussion and subsequent approval and implementation across the University.
As stated in our self-evaluation document (and instigated before the audit visit), we continue to review the effectiveness of our quality strategy and arrangements for quality assurance. Although we do not possess a written quality assurance strategy, we consider that quality is an integrated component of everything we do, rather than perceived as a separate entity. However, we recognise the need to reflect on the merits of this belief. The annual review of our course approval, monitoring and review panel provides an opportunity to consider the effectiveness of our quality assurance processes and we have regular meetings of Associate Deans, and CART Chairs, all of which make recommendations to the Academic Policy Committee.
Since the audit visit we have completed the annual monitoring review process to assure and enhance the quality of all our undergraduate and postgraduate taught provision. In our self-evaluation document we reflected on some difficulties that we experienced during the first year of the implementation of a new format for annual monitoring reports. The quality of the annual monitoring reports this year has improved as a result of increased guidance and support. We acknowledge that there is room for improvement and will work with academic departments to further enhance the monitoring process as appropriate.
We continue with our pro-active approach to support the implementation of the teaching quality information data set with further development of a web based facility for academic departments to access the published information.
Finally, we welcome the findings of the discipline audit trails which confirm that our quality assurance systems and processes are working effectively at the programme level.
ISBN 1 84482 043 2
Top