Findings
180 An institutional audit of the University was undertaken during the week 11 to 15 October 2004. The purpose of the audit was to provide public information on the quality of the University's programmes of study and on the discharge of its responsibility as a UK degree-awarding body. As part of the audit process, according to protocols agreed with HEFCE, SCOP and UUK, four DATs were selected for scrutiny. This section of the report of the audit summarises the findings of the audit. It concludes by identifying features of good practice that emerged from the audit, and recommendations to the University for enhancing current practice.
The effectiveness of institutional procedures for assuring the quality of programmes
181 The framework for managing quality is based on a division between faculties, who are responsible for operational management of validation, periodic review and annual monitoring, and the institution, which has overall strategic responsibility for the design of quality management processes, the annual quality audit process, student surveys and internal academic audit. The institutional responsibility for quality for taught programmes is formally delegated by AB to the QSC.
182 Proposals for new programmes are considered within faculties before being considered at institutional level. PDC has to be assured that student numbers, staff and physical resources are available, that the programme is compatible with the University's Strategic Plan and that the proposed programme does not duplicate provision in other faculties. The proposal is then passed to a validation panel, containing members external to the University as well as University staff from outside the school proposing the programme. The audit team considered that the process, for the most part, was consistent with the Code of practice and was both robust and thorough.
183 The University operates a system of annual and periodic review of programmes. Annual review of programmes produce action documents which outline actions to be taken in the present year, as well as reviewing the results of previous action plans. The input material is based on external examiner reports, student surveys at module and programme level, student intakes and student achievement. The audit team concluded that these reports were valuable and formed an effective mechanism that could highlight areas of good practice in addition to identifying concerns. The University may wish to consider how data on student achievement at module level may be incorporated into PSAADs in order to increase the effectiveness of the annual review process. PSAADs are submitted by programme boards of study to school directors, but not formally approved by either the director or by school management teams. The University may wish to consider whether it would be more appropriate to require formal approval of PSAADs at a level above the boards of study.
184 Periodic review of programmes is carried out at five-yearly intervals, although it is possible for such a review to be brought forward if either the school running the programme wishes to make major changes to the programme or the institution has major concerns about the programme. The periodic review of programmes is well established and follows the precepts of the Code of practice. In a similar approach to that seen in the validation of new programme proposals, a panel that includes members external to the University and University staff external to the school running the programme examine a SED, previous years PSAADs and background evidence about resources, staffing, student survey data and the reasoning behind any proposed changes to the programme. The team considered that the process is consistent with the Code and is both robust and thorough. A feature of the process that the University may wish to consider is the final approval of the report of the programme review panel by the FQC, rather than at the institutional level by QSC.
185 IAA provides a mechanism within the University for ensuring consistency across faculties and service areas. Each year a topic is selected as the basis of the audit and all faculties are audited in the same year. Members of audit panels, with one exception, are University staff external to the faculty being audited. The audit team considered the process as very thorough and gave the opportunity to monitor the development of faculties within a strong institutional framework. The team regarded IAA as a valuable addition to the University's quality processes.
186 Feedback from students, expressed largely in a quantitative form, is regarded by the University as a key performance indicator. This feedback is collected at module and at institutional level, and is used in the annual review process of programmes in addition to its use at institutional level to inform priorities in the provision of learning resources. The University is concerned about the low level of response to questionnaires, especially as response rates have decreased with the replacement of paper questionnaires by electronic ones. The audit team concluded that the module level feedback provided a useful mechanism for monitoring the student experience and that institutional level feedback procedures adopted for 2003-04 would be likely to improve response rates and data validity. If this improvement occurs the team considered that the student experience at institutional level would also be usefully monitored.
187 Surveys of alumni, conducted three years after graduation, and of employers gave results of varying value. Some schools had formal industrial advisory panels, while others had close links through the relevant PSRB. In still other cases, such as knowledge transfer partnerships and direct links with major local employers, formal links were also seen. However, there were no systematic collection of survey data from alumni and employers across the institution, the data collected being, in the view of the institution, mostly informal and indirect. The audit team concluded that employer input to programmes with PSRB involvement and in areas where employers could be readily identified was very effective. They were unable to form an opinion on the effectiveness of the informal and indirect employer input to programmes.
188 DL programmes form a small part (62 FTE) of the University's portfolio of programmes. The procedures that assure quality in these programmes follow the precepts of the Code of practice, Section 2 and also follow the University's procedures that assure the quality of all their programmes, such as validation, annual review and periodic review. Materials produced for DL are reviewed both internally and externally.
189 Much more significant is the use of the Off-Campus Gateway, an on-line support system which gives access to all off-campus services and resources. This service was introduced in April 2003 and has played an increasing role in the University's activities. Meetings with students confirmed that they were aware of the on-line support available, used the system and were positive about the quality of the on-line access and support available to them. Staff were positive about the developments in the field of e-learning and saw, together with the students, this as a valuable addition to the programmes available at the University. The audit team concluded that DL and distributed learning methods are being developed in line with good practice and that sound methods of assuring quality are in place.
190 Since the continuation audit in 1999, the University has striven to find a balance between an operational structure with reliance on a highly complex network of committees and a centrally focused management function. The current committee structure has developed in a way that focuses lines of communication between individual faculties and the institution very much through the executive committee structure, and in which the scope and focus of agendas and minutes varies distinctly between the academic, deliberative, committees and SMG and its subcommittees. It may be that the University has allowed the recording and reporting functions of its committee structure to lose clarity, which has a potential to adversely affect academic input into deliberation on matters of quality and standards. It was also unclear to the audit team whether the reporting line to the AB was robust enough to allow the AB to be assured of quality and standards of programmes and to allow it to fulfil its function of representing the views of the wider academic community on a range of academic matters at all levels of the University. The University may wish to re-evaluate its movement in this direction and to consider ways in which the operation of its central committees may be further enhanced.
The effectiveness of institutional procedures for securing the standards of awards
191 In securing the standards of its awards the University relies on its external examiners and on its procedures for programme approval, annual monitoring and periodic review. Terms of reference and criteria for appointment of external examiners and other external advisers are clearly set out in the University's regulations. The policies and practices of the external examiner system align with the precepts of the Code of practice, Section 4.
192 Programme approval and periodic review processes employ external representation, with two advisers one of whom will be from a professional body or employer organisation where appropriate. External advisers are specifically required to examine the programme's aims, outcomes, content and assessment in the context of the SBS and the FHEQ, to ensure that the proposal has taken account of these and focuses on academic standards. Advisers are also asked to make comparisons with similar provision at other HE institutions. There are clear procedures for responding to the comments of external advisers.
193 A large number of the programmes delivered at the University are accredited by PSRBs, and such accreditations help to secure the appropriateness and standards of the University's awards. The outcomes of accreditation or annual monitoring by PSRBs are incorporated into annual and periodic review processes at school level, and reports are also received directly by QSC for consideration at university level. Procedures for responding to such reports have been revised to ensure prompt responses to reports whose arrival does not coincide with the normal annual quality assurance cycle.
194 The University has recognised problems with student data management and accuracy for some years. A new student record system, OSS, was in the process of being implemented fully at the time of the audit visit. When fully operational this system will integrate information on admissions, enrolment, assessment, progression and awards and will enable staff at all levels to have access to current data. While the information available at the time of audit was limited, the system has the potential to provide a full range of student data for use in the analysis of assessment, progression and awards.
195 The primary role of external examiners is to ensure that justice is done to the individual student and that the standard of University awards is maintained, and each examiner is required to comment specifically on standards. External examiner reports are submitted electronically, and are considered at both programme and institutional levels. Programme teams respond directly to examiners but do not report upwards on actions taken until the next periodic review process. QSC also considers a digest of external examiner reports, and a generic university-level response is sent to each examiner after appropriate institutional actions have been taken.
196 External examiners have commended many good practices, and confirm that standards are comparable with, or exceed, those at institutions with which they are familiar. The audit team noted prompt written responses by the relevant departments to matters raised by external examiners.
197 The audit team was satisfied that broad confidence can be placed in the University's present and future capacity to manage effectively the academic standard of its awards. This conclusion was based on the strong systems for considering and responding to the views of external examiners, external advisers and PSRBs, and on the anticipated full implementation of the OSS student record system for the analysis of progression and achievement data.
The effectiveness of institutional procedures for supporting learning
198 The University in its SED, indicated that change management allied to strategic and operational delivery requirements has been a key aspect of the HR strategy.
199 The audit team found evidence within the project groups established to implement the strategy to support this statement. In addition, senior staff clearly indicated to the team that the adoption of the EFQM excellence model had led to a change in leadership style and the development of more effective team working within the University. The audit team found little evidence of knowledge of the EFQM excellence model at school and programme level.
200 There are clear procedures for appointment, appraisal and reward. Staff development is supported through the annual PDPR and the audit team found strong evidence for its effectiveness at both personal and corporate levels. There was clear evidence of integration between the HR Strategy and the LTA Strategy linking to the programmes offered by both the CSD and LDU. The audit team found support from across the University for the range, appropriateness and value of the training opportunities available. The team considered that the coordinated and integrated approach to staff development within the University and its management at individual, school and faculty levels was an example of good practice.
201 The University is aware of the need to develop increased access to training opportunities for part-time and teaching assistant staff and the audit team encourages the University to continue improving the access to staff development activities for these categories of staff.
202 The audit team found evidence of a number of effective activities, throughout the University, which enable good practice in relation to LTA to be disseminated. While the University recognises that more needs to be done to spread dissemination strategies throughout the University, the team was of the view that the University considered the dissemination of good practice seriously and were making good progress in this area.
203 The University is a large and complex institution, which comprises a number of campuses across Liverpool. In this context, the University's stated purpose to 'create and support opportunities for successful participation by under-represented groups ' presents a particular set of challenges in terms of the effectiveness of procedures for supporting learning.
204 The University provides a wide range of support facilities for learners at institutional, faculty, school and programme levels. Institutionally, these operate largely through a converged library and computing support service in three LRCs as well as through the Off-Campus Gateway. In keeping with the University's commitment to devolution, faculties provide some specialist learning support and a learning materials fund, based on student numbers, is administered through LIS on behalf of the schools.
205 The audit team confirmed the University's view that LIS and CIS work effectively together to manage the learning support resources. There was also evidence that students were appreciative of the CWIS, the provision of 24-hour access at some LRCs and access to the University network from halls of residence. The team found evidence of some variability in the level of computing facilities at different LRCs and encourages the University to continue its efforts in monitoring the level of provision across all its campuses to ensure parity.
206 The introduction of the VLE has been well planned and put into operation. It was clear to the audit team that the VLE has gained a good deal of support from staff and students and that it is beginning to be used in innovative and interactive ways. The team encourages the University to closely monitor targets for the further implementation of the VLE across programmes and to seek ways of disseminating good practice in its use as a support for other forms of learning.
207 A variety of arrangements are in place for academic and personal guidance and support. The University Campus Centres are well utilised by students, forming an administrative focus for most of the students who talked to the team. Fewer students used the Campus Centres for specialist welfare support, and the University might wish to consider ways of ensuring that these services are more consistently taken up by students.
208 There is no single system of personal tutoring across the University. Instead the audit team found a variety of ways in which schools and programmes support their students. The support system operated by the RGS, was regarded by the team as particularly effective. In talking to students at both undergraduate and postgraduate level, the team concluded that support systems are generally seen as effective by students and that they are aware of how to access them.
209 Overall, the findings of the audit indicate that the University provides a full and varied range of support and guidance services. Students spoke warmly of their relationships with teaching staff and the quality of personal and academic support at programme level was a positive feature of their experience at the University.
Outcomes of discipline audit trails
Biomolecular sciences
210 Programme specifications for the programmes that were the subject of this DAT set out appropriate educational aims and learning outcomes. In general, the programme specifications match the expectations of the Academic Infrastructure. External examiner reports seen by the audit team were almost universally positive, confirming the conformity of programme content to SBSs and parity of standards with those found elsewhere. One of the degree programmes is accredited by the IBMS, and related programme validation, monitoring and external examiner reports are satisfactory.
211 The School complies with University procedures in the annual and periodic monitoring of programmes. Annual monitoring reports for 2002-03, before the OSS student records system came into operation, lacked statistical data and were descriptive rather than evaluative. However, draft annual reports for 2003-04 seen by the audit team were much fuller and contained progression data and an account of the University student satisfaction survey. There are some concerns about student progression from level 1 to level 2, but these are being actively addressed.
212 Students of the School confirmed to the audit team that their needs for induction and for academic and pastoral support are being met. They spoke enthusiastically about the effectiveness of the feedback which the staff provide at module level in bringing about improvements. Students were also very supportive of the use of the VLE as a tool in supporting their studies. Student support mechanisms are strong and external examiners and IBMS accreditors are positive about quality and standards. The School is taking action designed to improve progression rates and to improve its use of the formal annual review process.
213 Overall, the audit team found that the quality of learning opportunities is suitable for the programmes of study leading to the named awards and that those awards were appropriately situated within the FHEQ.
Electrical and electronic engineering
214 Programme specifications at undergraduate level met the expectations of the SBS and the FHEQ, and provided information to students on the intended learning outcomes. At postgraduate level the programme specification met the expectations of the FHEQ and provided information to students on the intended learning outcomes.
215 In meeting with the audit team students expressed satisfaction with the academic support and learning resources given by the School and University. They expressed reservations about the quality of some feedback given on coursework, as well as the accuracy and fullness of information given on some postgraduate programmes. Annual and periodic review in addition to discussions within the programme team and School were used to identify areas of concern and form the basis of action plans to remedy these areas.
216 Overall, the audit team found that the quality of learning opportunities is suitable for the programmes of study leading to the named awards and that those awards were appropriately situated within the FHEQ.
Food science
217 The programme specification for the undergraduate degree included appropriate aims and learning outcomes clearly linked to teaching, learning and assessment and specifically referenced to the relevant benchmark statement. The learning outcomes were appropriate for its location within the FHEQ. The programme specification for the CPD course was less specific but indicated learning outcomes appropriate for its location within the FHEQ. The staff informed the audit team that they were preparing revised programme specifications to a University template.
218 Student evaluation of the programmes was positive and students expressed satisfaction with nature and extent of the support and learning resources available to them. They felt sufficiently informed about assessment criteria, that feedback was constructive and effective, and that they had appropriate mechanisms for representation.
219 Overall, the audit team found that the quality of learning opportunities is suitable for the programmes of study leading to the named awards and that those awards were appropriately situated within the FHEQ.
Sociology
220 The programme specifications for the sociology programmes set appropriate educational aims and learning outcomes. These outcomes are clearly linked to the learning and teaching styles employed by the programme team and to the types of assessment that students experience. Effective use is made of the SBS for sociology to inform undergraduate programmes. The external examiner reports read by the audit team showed a considerable level of positive comment on the programmes in sociology.
221 Students who met the audit team were very enthusiastic about the level and type of support received from the sociology staff. They were also positive about the opportunities they are afforded to comment on the programme and they were satisfied that their concerns are taken into account and fed back to them. The year-long tutorial module at level 1 was seen by students and staff as a positive support mechanism and the way in which personal tutoring is undertaken was clearly of benefit to students. Measures have been introduced to address the drop-out rate at level 1. Learning support through the LRCs, including library and computing facilities were regarded positively by students while the introduction and development of the VLE as a supplement to face-to-face teaching was also appreciated.
222 Overall, the audit team found that the quality of learning opportunities is suitable for the programmes of study leading to the named awards and that those awards were appropriately situated within the FHEQ.
The use made by the institution of the Academic Infrastructure
223 The University has made full use of the Academic Infrastructure as a reference point for policies and procedures. As each part of the Academic Infrastructure has become available, the University has taken appropriate steps to ensure that it has been addressed.
224 All of the University's awards are described by levels which are consistent with those of the FHEQ. Common academic regulations are used across the University's programmes to ensure consistency. Programme specifications have been routinely adopted by the University with a standard information content. The University has more recently also adopted a standard template for its programme specifications and a rolling programme of publication in the standard template is under way from 2004. All of the University's programmes are validated and periodically reviewed with reference to the appropriate SBSs and these are also clearly indicated in the programme specifications.
225 As each section of the Code of practice was published, the University established clear mechanisms for reviewing and, where necessary, revising policies and procedures so that they are consistent with all parts of the Code. The audit team concluded that this systematic approach had been beneficial, for example, with consistent practices across the University relating to assessment of students, external examining amongst others.
The utility of the SED as an illustration of the institution's capacity to reflect upon its own strengths and limitations, and to act on these to enhance quality and standards
226 The SED provided a clear overview of the structure and processes the University has in place to manage quality and standards. It both described, and in most subsections, evaluated, a wide range of the University's quality management enhancement QME systems and was thoroughly underpinned by, and carefully cross-referenced to, a range of supporting documentation. The SED provided a valuable summary of developments since the 1999 audit and described responses to the report of that visit and to other external quality assurance evaluations that were informative and helpful to the audit team.
227 The SED reflected the University's ability to manage change and adapt procedures in the light of new University priorities and external pressures, including reviewing management, academic and committee arrangements and responding positively to the national Academic Infrastructure. The SED also accurately represented the University's strong emphasis on the primacy of the executive and management structures of the University and the use of management models, notably the EFQM excellence model, in most aspects of QME rather than AB and deliberative committees. The audit team considered that, although the SED described many of the University's policies, procedures and practices, it might have gone further in describing overarching strategies, for example for communications. However, overall the SED provided a sound review that acknowledged areas for further development and in most respects corresponded with the main findings of the team. It provided confidence that the University had capacity for self-reflection and the ability to develop appropriate responses.
Commentary on the institution's intentions for the enhancement of quality and standards
228 The University noted a range of mechanisms by which enhancement occurs at university, faculty, school and individual level. The SED described adherence to the EFQM as an important driver for development of management and organisational systems and a means of identifying enhancement priorities. The University's LTA Strategy, 2002-05 provides a clear framework for strategic enhancement in relation to the academic operation of the University at all levels. The SED described an infrastructure to support the strategy, which included a central LDU and various key posts in faculties that support the delivery of the strategy in a variety of ways. These included supporting new initiatives in LTA, a variety of publications and guides and a range of staff development activities. The the activities RGS was noted by the University as a focus for both the support of research students and also for staff development. The University provided the audit team with detailed information on a range of activities designed to support individual staff, including formal qualifications in teaching and learning in HE for newly appointed inexperienced academic staff. The SED also described the ways in which quality assurance procedures, for example annual programme monitoring and periodic programme review identified good practice for dissemination.
229 From the wide range of evidence available to it, the audit team concluded that the University did indeed have in place a comprehensive range of enhancement activities that formed a coherent whole. Importantly there was a demonstrable commitment to the activities that gave the team confidence that would continue into the future. Some examples that the team would single out are; the broad, integrated approach to staff development, spanning the CSD, the LDU, schools and faculties, the role of the RGS in supporting both research students and staff who supervise research students; the support for both staff and students in the strategic implementation of the University's chosen VLE. Overall, the University's enhancement strategy is well matched to its broad strategic objectives.
Reliability of information
230 Staff informed the audit team that the University had made good progress in meeting the requirements of HEFCE 03/51. At the time of the audit the University had certain elements such as the Teaching and Learning strategy and the Employer Needs published. Other qualitative information such as external examiner summaries were actively being progressed. The team was satisfied that the University had engaged with the need to produce accurate information as required by HEFCE 03/51 and that the University had made good progress towards meeting the requirements within the designated timeframes.
231 The University has had difficulties with regard to student data management and accuracy which have not yet been fully resolved. The OSS system has the potential to resolve many problems in relation to data sets, however the audit team advise the University to continue its development with some urgency, including a strategy to ensure the accuracy, consistency and completeness of the data.
232 During the course of the audit, the team found several examples of inconsistency within published information, one of which had caused significant concerns to an overseas student. While recognising that the University has been, and continues to, develop procedures for ensuring ownership of published information, the audit team recommend that the University develop an Information Management Strategy, including appropriate authority for 'signing off' published material to ensure consistency of published information across all processes and at all levels of operation, as a matter of urgency.
Features of good practice
233 The audit team identified the following areas of good practice within the University:
- the broad integrated approach to staff development, spanning the CSD, the LDU, schools and faculties (paragraphs 93-97)
- the role and operation of the RGS (paragraphs 117-121)
- the strategic implementation and development of, and support for, the VLE (paragraphs 108-112)
- the rigorous IAA process (paragraphs 45-48).
Recommendations for action
234 The audit team also recommends that the University should consider further action in a number of areas to ensure that the academic quality and standards of the awards it offers are maintained.
The team advises the University to:
- further develop committee structures to clarify lines of communication and to enhance its ability to reflect upon the management of quality and standards at institutional level (paragraphs 29-31)
- ensure that the reporting of quality processes at all levels of operation ensures accountability from each level to the next (paragraphs 44, 142)
- implement strategies to ensure the consistency of published information across all processes and at all levels of operation (paragraphs 22, 55, 177-8)
- develop strategies to ensure that management information systems are fully exploited in order to achieve oversight of issues related to the management of quality and standards at institutional level (paragraphs 34, 43, 83-87, 130-1, 154).
It would be desirable for the University to:
- more clearly develop and articulate its strategic approach to the management of quality and standards, moving beyond operational aspects of the framework (paragraphs 29-31)
- further enhance staff engagement, at all levels, with academic planning and development (paragraph 30).
Appendix
Liverpool John Moores University's response to the audit report
The University welcomes the outcome of the Institutional Audit, that confirms both the quality and standards of the academic programmes and awards and places confidence in the processes which assure the future management and enhancement of the provision at Liverpool John Moores University.
The University was pleased that areas of good practice were highlighted by the audit team in the text of the report including: the broad integrated approach to staff development, spanning the Centre for Staff Development, the Learning Development Unit, schools and faculties; the role and operation of the graduate school; the strategic implementation and development of, and support for, the virtual learning environment; and the rigorous internal academic audit process.
In terms of addressing the recommendations for action, the University is committed to keeping its committee structure under review in order to clarify its implementation and to enhance the University's ability to reflect upon the management of quality and standards at institutional level. In reviewing and developing its new framework for quality management, the University is keen to strengthen accountability from each level to the next within the framework and to develop a more strategic approach to the management of quality and standards. Through a programme of linked development projects, the University is seeking to ensure the consistency of published information across all processes and at all levels of operation; and that management information systems are fully exploited in order to achieve oversight of issues related to the management of quality and standards at institutional level.
In conclusion, the University appreciates the constructive and professional approach taken by the audit team, the findings of which have consolidated the University's approach to quality management and plans for future development.
ISBN 1 84482 228 1
Top
QAA updates
QAAtweets
QAAtube