Introduction
1 The Quality Assurance Agency for Higher Education (the Agency) is a UK organisation that seeks to promote public confidence that the quality of provision and standards of awards in higher education (HE) are being safeguarded and enhanced. It provides public information about quality and standards in HE to meet the needs of students, employers and the funders of HE. One of the Agency's activities is to carry out quality audits of collaborative links between UK HE institutions and their partner organisations in other countries. In the spring and early summer of 2003, the Agency audited selected partnership links between UK HE institutions and institutions in Italy. The purpose of the audits was to provide information on the way in which the UK institutions were maintaining academic standards and quality of education in their partnerships.
The process of audit of overseas partnership links
2 In July 2002, the Agency invited all UK HE institutions to provide information on their collaborative partnerships in a range of overseas countries. Using this information, the Agency approached a number of institutions which had indicated that they had established collaborative links with partner institutions in Italy. Following discussion, a variety of collaborative partnerships was selected for scrutiny. Each of the UK institutions whose collaborative link had been selected for the audit provided a Commentary describing the way in which the partnership operated, and discussing the effectiveness of the means by which the UK institution assured quality and standards in the link. In addition, each institution was asked, as part of its Commentary, to make reference to the extent to which the link was representative of its procedures and practice in all its overseas collaborative activity, or specific to the partnership being audited. Institutions were also invited, in their Commentaries, to make reference to the ways in which their arrangements met the expectations of the Code of practice for the assurance of academic quality and standards in higher education (Code of practice), Section 2: Collaborative provision (1999) published by the Agency, which took full effect in August 2000.
3 In the spring of 2003, audit visits were made to each UK institution to discuss its arrangements in the light of the Commentary. In May 2003, an audit team visited the partner institutions in Italy to gain further insight into the experience of students and staff, and to supplement the view formed by the team from the institution's Commentary and from the UK visit. During the visits to institutions in Italy, further documentation about the partnerships was made available to the team, and discussions were conducted with key members of staff, lecturers and students. The team for this audit comprised Professor G Chesters, Professor A Gale, and Dr L H Roberts. The UK and overseas audit exercise was coordinated for the Agency by Dr P J A Findlay and Mrs S Patterson, Assistant Directors, Reviews Group. The Agency is particularly grateful to the UK institutions and their partners in Italy for the willing cooperation provided to the team.
4 This report describes the audit of the collaborative link between the University of Wales (the University) and John Cabot University, Rome (JCU). In accordance with its Federal Plan, the University has established a central Validation Board (VB) and Unit (UWVU) to manage its validation work. The audit was conducted on the basis of visits by the audit team to the University to meet staff associated with the operation of the link and to JCU in Rome. The team also scrutinised documentary evidence made available by the UWVU. On 19 March 2003 the team met the Head of Validation Services and the Academic Secretary. The team also had a videoconference with the moderators for the courses. The team visited JCU on 27 May 2003 and met senior staff associated with the link and a group of students.
5 The University's collaborative arrangements with the Institución Empresarial Europea, Madrid, Spain and the Escuela de Negocios Caixavigo, Vigo, Spain were the subject of an audit by the Higher Education Quality Council (HEQC) in December 1996. In March 1999 the Agency reported on the University's collaborative arrangement with Portobello College, Republic of Ireland and in November 2000 published a report on an audit of the University's link with the Know How Group of Colleges in Spain. The University operates a number of franchised arrangements some of which have also been the subject of audits by the Agency.
The background to the collaborative partnership
6 John Cabot International College was founded in 1972 as an American liberal arts college, becoming a university in 1990 after it had been incorporated by the Delaware Department of Education and given the power to award degrees. It has recently been accredited by the American Middle States Commission on Higher Education for an initial period of five years. It is also authorised by the Italian Ministry of Public Instruction to operate as a USA Higher Education Institution in Rome, although its degrees, in common with most other foreign degrees, are not recognised by the Italian Government. Among its corporate goals are:
- to achieve an international reputation as a distinguished institution of HE;
- to be known for the teaching excellence, the scholarly achievements and the community service of its faculty;
- to produce graduates who are sought by, and successful in, the international communities of business, public service and academia.
7 The current population is some 450 students with about a third being drawn from the USA, a third from Italy and a third from the rest of Europe. JCU is managed by a Board of Trustees with regular oversight undertaken by Academic Council and a number of subcommittees.
8 The Commentary described the collaborative provision as 'validated degree schemes, each of three years' duration' which lead to the awards of: BA (Hons) Business Administration; BA (Hons) Political Science; and BA (Hons) International Affairs. The first students were registered on the programmes in October 2000 and, through the use of accredited prior learning, graduated in September 2002. At the end of the course of prescribed study, successful students are awarded both a JCU degree and a separate degree from the University. At the time of the audit there were 48 students in total registered on the programmes (26, 2 and 20 respectively). The language of assessment and delivery is English.
9 The University is formally responsible for securing the quality and standards of awards made through its collaborative provision, with responsibility for the detailed oversight delegated to the VB and to the UWVU. The VB reports to the University's Academic Board, and has three subcommittees: a Standing Executive Committee (SEC); a Planning Advisory Group (PAG); and a Health Studies Committee which oversee designated areas of the Board's activities; at the time of the audit, the terms of reference of the VB were under review. SEC is responsible for implementing the VB's detailed quality assurance work including, scrutiny of nominations for external examiners, reports from validated centres, reports from external examiners, and reports from moderators (see paragraphs 27 to 29). PAG monitors the operation of the UWVU. An imminent change to the University's procedures is that, in future, the Academic Board will receive internal quality reports on collaborative links. The UWVU provides annual reports to the University Council which, in the view of the audit team from the examples presented to it, appeared to be more descriptive than analytical or evaluative.
10 The UWVU is a central University organisation one of whose objectives, in conjunction with the VB, is 'to provide an international validation service across all subject boundaries by drawing on the University's pool of academic expertise and excellence'. The Head of Validation Services leads the unit, which consists of 10 staff. The UWVU aims to work to a philosophy which demands 'excellence by focusing on quality assurance and appraisal mechanisms alongside regard for appropriate course content, whilst maintaining a workable and flexible approach to potential partners'. It has oversight of validated programmes in over 60 institutions throughout the world with some 9,000 students enrolled in programmes in more than 20 countries. Much of the responsibility for the detailed implementation of the University's validation policies and procedures lies with the Head of Validation Services. The UWVU was the subject of an internal quality assurance audit in November 2002, the conclusions of which were generally positive, with some suggestions for enhancement. Successive internal reviews of the operation of the UWVU have recognised the potential vulnerability of the Unit's operation through inadequate staffing, and steps have therefore been taken to augment its establishment. At the time of the audit team's visit a further internal review of the UWVU was underway, focussing on benefits and attendant risks of the Unit's activities.
11 The audit team formed the view that the UWVU was an autonomous unit under able leadership. Any potential for isolation from the academic community is mitigated to a significant extent by the key operational involvement of moderators, who are academic staff from the University's constituent institutions who take responsibility for one or more collaborative arrangement (see paragraphs 27 to 29). Staff of the UWVU also maintain regular, in some cases daily, contact with academic colleagues within and outside the University.
12 The Commentary identified the JCU link as typical of the University's collaborative arrangements and the audit team was satisfied that the manner in which the degree schemes at JCU had been validated, monitored and developed was broadly representative of the University's approach. Reference to earlier Agency and HEQC audits of the University's collaborative activity overseas confirmed this view as well as demonstrating the University's responsiveness to external reports on its linkages. In one significant way the JCU link is not typical of the University's validated arrangements in that students receive two awards, one of the University and one of JCU. The team was informed that such arrangements were becoming more frequent but were still unusual.
13 The claim in the Commentary that the collaboration with JCU was pursued 'in accordance with the definitions set out in the QAA Code of practice' seemed to the audit team to be broadly and consistently substantiated by the evidence. The internal Audit of the Work of the Validation Unit (reporting in November 2002) specifically checked alignment of processes with the Code of practice and found broad adherence. Where there was a disjunction, for example, seeking external examiner views on subject benchmarking, The framework for higher education qualifications in England, Wales and Northern Ireland (FHEQ) and programme specifications, the report made recommendations for action. The team recognises the awareness of the Code of practice within the UWVU and shares the general conclusion reached by the internal audit.
14 Detailed procedures governing the operation of validated partnerships are set out in the regularly revised University of Wales Validation Handbook of Quality Assurance Policies and Procedures (the Handbook), which contains 41 sections and appendices setting out a Code of Practice and details of relevant processes including: validation; initial vetting of new centres; post validation follow-up; course monitoring mechanisms; annual monitoring; upgrading of degree schemes; amendments to existing schemes; withdrawal of validation; and quinquennial reviews. The audit team commended the thoroughness of the Handbook, as well as the good practice of sending an updated version and other relevant documentation to collaborative partners on an annual basis (see also paragraph 34). The Code of practice, Section 2: Collaborative provision is included in the Handbook indicating that the University expects its partners, its moderators and all involved in collaborative arrangements to adhere to the relevant precepts and guidance.
The establishment and management of the collaborative partnership
15 After an initial contact between the University and JCU in September 1998, there was a period of some 15 months during which JCU's Academic Dean liaised with colleagues at the University about the precise requirements for validation and prepared draft submissions for commentary by the moderators designate. A preliminary visit by UWVU staff was undertaken to JCU in November 1998, reporting to the Validation Board in February 1999. Upon receipt of comprehensive documentation from JCU, a validation exercise was planned and undertaken in May 2000. A panel of assessors, chaired by the Deputy Chair of the Validation Board and including two professors external to the University, paid a formal two-day visit to JCU in late May 2000; the panel also included the moderators designate. The University may wish to consider the extent to which it is reasonable that moderators designate, who have been involved in advising the potential partner, including reading early drafts of the submission documentation, be expected to maintain the requisite objectivity in the formal approval process.
16 While it was clear from the validation report that appropriate attention was paid to the structure, content and philosophy of the three programmes, the Commentary provided little information on the initial approval of JCU as a partner institution of the University. The recent internal audit suggested that the initial vetting form for potential partnerships be redrafted to require more information on compatibility of mission between the University and its prospective partners. The Handbook had a section entitled 'Initial Vetting of New Centres' but included little detail on formal investigation of the standing or financial probity of the potential partner or consultation with those who might provide such information, for example the British Council or other HE institutions. The Initial Vetting section of the Handbook stated that 'the University [would] seek appropriate information to ensure that the proposed partner [was] a financially stable institution with effective and adequate management and administrative systems ...' but it was not clear to the audit team how such information was gathered or analysed. At its meeting in February 1999, the VB considered the report from the Head of Validation Services, written in November 1998, on the preliminary visit to JCU which included some basic financial data about the then prospective partner. As a general point, with no specific reference to the collaborative link with JCU, the University may wish to consider paying more specific and formal attention to the financial health of prospective partners as a way of mitigating risk.
17 The Commentary did not include information on the policy of the Italian Government on recognition of the awards and there was no reference to local recognition and the status of graduates in either the validation or follow-up reports. The audit team heard from colleagues at JCU that the Italian Government appeared to be reluctant to recognise both the British and the American awards. Subsequent to the audit the team was informed that the University through the VB was undertaking detailed work in cooperation with the European Commission to seek recognition of UK degrees across the EU.
18 The validation report also made no reference to the fact that students following the approved University course at JCU would receive two awards on successful completion. It appeared to the audit team that the point of principle of allowing the same credits to contribute to the award of distinct and separate degrees was also not given specific consideration by the VB in discussion of the detail of the report. The team's discussions with the Head of Validation Services suggested that this significant general matter of principle was now due to receive consideration by the VB.
19 The validation process does not require any formal, systematic comparison of content or standards of programmes with University collaborative programmes of the same title offered in different locations around the world; the University informed the Agency that the reference points used by the Validation Board in this context included the FHEQ, the Code of practice, subject benchmark statements, programme specifications and detailed learning outcomes. While accepting the strength of this position, the audit team would wish the University to consider how it might develop its approval processes for validated programmes so that the issue of comparability is given more prominence.
20 The panel of assessors recommended validation subject to two conditions, the first of which stressed the need to revise the degrees' structures and content 'with particular emphasis on levels 1 and 2', and the second of which related to preparation of definitive course documents 'covering levels 1 to 3'. A post-validation visit resulted in a further report to the SEC in April 2001, at which stage progress towards fulfilling the conditions was deemed to be 'very good'.
21 The audit team saw the text of the current Agreement (signed in October 2002) between the partners but not the original. The Commentary did not explain why a new Agreement was necessary after the programme had been operating for only two years, given that provision for review was related to a quinquennial process. There appeared to have been a major change in the basis of the academic approval subsequent to the original validation, namely the move from validation of a three-year programme to validation of only two years of study. The Student Handbook referred to 'the two year program', and the publicity produced by JCU and authorised by the UWVU referred to the need for students registered for awards of the University to pay validation-related fees for each of the last two academic years. Given the 'particular emphasis on levels 1 and 2' in the first condition at validation and the requirement that definitive course documents be produced 'covering levels 1 to 3' in the second condition, it appeared unusual to the audit team that what had initially been a validation of all three years of the degrees was modified significantly within such a short time (June 2002). The team was unable to find any explanation for this change apart from the implications of a decision to move from the original weighting for classification of assessment stages from 10:40:50 for the three years to a 0:40:60 model. Nonetheless, counting only years two and three towards degree classification would not necessarily entail a decision not to validate the first year of a programme, designed around meeting the relevant Agency subject benchmarks through standard three-year undergraduate programmes of study. One moderator expressed the view that the approach to year one was 'fairly relaxed', acknowledging a 'generous' attitude. The team was unable to establish the extent of control exercised by the University over the content of the courses followed by JCU students before registering for the University's awards; the articulation between JCU modules and the validated modules at levels 2 and 3 was not explicit in the October 2002 Agreement.
22 The audit team identified a related concern in that the students registered for the University award were taught in common with those students following the more broadly designed liberal arts curriculum for the JCU award. The University may wish to consider how it can assure itself that the shared delivery provides the level of intellectual demand and rigour appropriate to a UK honours degree award.
23 The major change in the collaborative arrangement was not mentioned in the Commentary and, from the evidence available to it, the audit team was unable to establish whether it had been approved by the VB or any of its subcommittees. There was also no reference to the revision in the reports from moderators or external examiners or in the JCU Annual Report. The Handbook outlined very clearly the process by which amendments to validated schemes of study were approved. What was described as Procedure B concerned 'amendment to a scheme's syllabus and/or assessment criteria which contributes or affects more than 30% of the entire scheme (or more than 30% of the modules/credits contributing towards the final degree classification)'. Procedure A concerned amendments that affected less than 30 per cent of the scheme; in such cases the VB was simply informed of such amendments through the moderator's report or the partner's annual report or, if required, a report by the Head of Validation Services. Procedure B required a written request for approval, the constitution of a panel of assessors and a written panel report containing recommendations for VB approval. The team found no evidence of either procedure being followed although it would appear that both the change in assessment weighting and the change to a two-year validation fall within the definition of Procedure B. The team took the view that a change from a three-year validation to a two-year validation should, at the very least, have been subject to formal approval by the VB.
24 The written Agreement between the partners was comprehensive, including definitions, term and duration, entry requirements and awards, payments, conduct of examinations, quality assurance procedures, information to be provided by JCU (both to staff and students), appeals, termination, clauses on promotional material, indemnity, jurisdiction and exclusion of third parties. It affirmed in several ways the University's right and responsibility to protect the standards of its awards.
25 The University demonstrated that it had control over the certificates issued to students on successful completion of their course. It prepares all the certificates in house and sends them to JCU. The sample certificate seen by the audit team met the requirements of the relevant precepts of the Code of practice.
26 Control over the accuracy of publicity and marketing materials is exercised by the UWVU, empowered by the clause in the standard agreement which stipulated that 'the University shall have absolute discretion as to the contents of any statements, advertisements or on other promotional material.'. The Head of Validation Services contacts course directors in partner institutions requesting completion of a pro forma on publicity materials at the start of the each academic year. There is a follow-up letter if they fail to submit their returns. The concern to adhere to the relevant precepts of the Code of practice was clear, and was reinforced by the recent audit of the UWVU which recommended an immediate examination of partners' web sites to check on material on validated programmes lodged there.
Quality of learning opportunities and student support
27 A crucial element in the University's system for the management of its collaborative arrangements is the role of the moderator, a member of the University's academic staff from, and identified by, an appropriate department in one of its constituent institutions. In the case of the JCU link, the three programmes are covered by two moderators from the University of Wales, Bangor, one with several years of broad experience as moderator (having been responsible simultaneously for several validated links), the other relatively new to the role. Moderators from across the University meet annually to compare experiences, exchange good practice and to be updated on policy and procedural modifications. The December 2002 meeting, for example, addressed the innovative Moderators' Intranet (which allows peer communication), the need to follow up action after the Annual Report from the validated centres, the emerging University on-line resource for use by students at all validated centres, the updated version nine of the Handbook, and an overview of issues arising from all external examiner and moderator reports. The audit team commended the University's efforts to ensure consistency of approach, to disseminate information and to share best practice among its moderators.
28 The expectations placed upon moderators were set out in detail in the Handbook. They are charged with a quality enhancement and programme development role and report to the VB on comparability of academic standards and adequacy of resource. Specific duties relate to assisting with the preparation of draft examination papers, a process which proved somewhat problematic initially as JCU staff were accustomed to preparing examination papers closer to the point of assessment than the University's schedule required; attendance with a monitoring brief at examination boards; submission of an annual report; compilation of reports on two visits each year to the partner institution (to include meetings with teaching teams and students) and, the maintenance of close and regular contact with the partner institution. At JCU, moderators have been involved in staff development workshops and other less formal developmental activities. Moderators' performance is assessed at the time of the quinquennial review of validated programmes, although appointments have been terminated outside the quinquennial review process when problems have occurred; the audit team heard that a formal procedure for early termination of moderator appointments in the event of unsatisfactory performance would be submitted to the VB for approval.
29 The way in which the moderators for the link work with colleagues in Rome is appreciated by JCU. It became clear to the audit team that the academic interaction had been both rich and mutually challenging at times, as curriculum and assessment practices were progressively modified to dovetail with expectations of UK undergraduate degrees. The team saw and heard evidence that the moderator's role had been primarily developmental, reserving the more judgemental and evaluative aspects to the external examiners' role. In this context, the team noted that the moderators' responsibility to the VB was to report on the equivalence of academic standards: the moderators for the link were aware of the need to maintain the balance between the developmental and the standards-related aspects of their role but had not encountered any difficulties in this area in the link with JCU. The team saw evidence in various reports and interchanges to support the claim in the Commentary that the interaction between JCU, moderators and external examiners was satisfactory.
30 General communication at institutional level between the UWVU and JCU was commendable. Liaison at subject level rests entirely upon the moderators, as the 'parent' department at the University of Wales, Bangor, plays no part otherwise in the day-to-day or year-to-year liaison. There have been preliminary discussions between JCU and one of the moderators about staff exchanges.
31 The processes for ongoing quality assurance were clearly explained in the Handbook, including approval of draft examination papers, examination boards, reports of moderators and external examiners, and annual college and course review forms. The Handbook also contained excellent guidance on the conduct of quinquennial reviews. The Annual College and Course Review forms, essentially, and often referred to as, the annual report have detailed pro forma covering admissions policy, entry qualifications of students, wastage rates, target numbers, summary of student feedback, staff changes, resources changes, minutes of course monitoring committees, list of modules, proposed changes, assessment criteria and weightings, response to external examiner and moderator comments, conduct of examinations and analysis of results. Annual reports go to SEC, whose members divide into reading pairs to scrutinise reports from validated centres with which they have no direct connection and any recommendations are referred back to the partner institution with a specific invitation to respond on each recommendation. These responses are then considered by the moderators and external examiners, and a report sent to the VB as necessary.
32 JCU had produced Annual Reports for the University for 2000-01, and 2001-02. Both reports included, on the whole, appropriate and sometimes extensive information; they confirmed that the UWVU had access to JCU's responses to matters raised by external examiner and moderator reports. The audit team identified items in the annual reports that might have been expected, but did not appear to have elicited comment from the SEC readers. The University might, with benefit, keep under review the approach to the current scrutiny of annual reports that it receives from validated centres in the context of its effectiveness in identifying good practice and matters for further action.
33 At the time of the audit, the VB was considering a proposal that the internal system of Joint Boards of Studies be extended to include validated centres. The audit team, agreeing with the 1994 HEQC Audit report which commended such boards as providing 'a valued means for academic dialogue and support', supported this proposal as a forum for discussion of matters beyond the formal remit of an examination board and outside the moderator system.
34 The UWVU provides partner institutions with annually updated guidance documentation including: a User Guide, Academic Regulations, Standing Orders, Quality Assurance Handbook, Examination Board Agendas, Directions to Candidates, and Unfair Practice, Verification and Appeals and Student Complaints procedures. This mechanism promotes consistency of approach across the University's collaborative links. There was also evidence of frequent contact by email between JCU staff, the moderators and staff of the UWVU.
35 The main information channel for students is through the administrative and academic staff at JCU. The Head of Validation Services writes to all new students, giving details of the University and inviting queries. Students met by the audit team did not recall any direct contact with the University, including receipt of the 'welcome' letter or meetings with University staff, and did not know any colleagues who had had such contact; they perceived themselves as JCU students and the University as largely a regulatory presence. The team saw evidence that meetings between moderators and students without JCU staff present did take place but were unable to reconcile this with the students' unawareness of these meetings, the existence, purpose and outcomes of which did not appear to have been appreciated; the University may wish to consider how it could publicise these meetings more effectively. Students whom the team met expressed a wish to meet University staff and to be more strongly identified with the University. The team formed the view that the University might wish to consider how, together with JCU, it might more effectively promote itself as a substantive institution to its future alumni on the validated courses.
36 The Commentary did not include specific reference to the University's policy on staffing and staff development in its validated linkages. The Agreement governing the operation of the programmes requires that 'employment of teaching staff shall be established only subject to the written prior approval of the University'. Curricula vitae (CV) of staff were included in the original validation document and those of new appointments are submitted to the University with annual reports. The audit team was informed that the moderators also received, for comment and confirmation, CVs for candidates for appointment. The moderators meet the teaching staff on a six-monthly basis. Staff development is a key element in the role of the moderators who are required to report on such activity to the VB.
37 Although there was no requirement in either the Agreement or the validation report for staff development there has been significant staff development activity at JCU in support of the validated programmes; for example, moderators have provided guidance and support on entry requirements, assessment criteria and classification of degrees. The easy professional and collegial relationship between the moderators and the staff at JCU facilitates this type of exchange.
The assurance of the standards of awards
38 The Academic Dean at JCU has oversight of the day-to-day management of the programme and provision of student support by the course directors. JCU administers the process of selection and admission, with a requirement to report on the qualifications of the intake; one of the moderators has been involved in consideration of special cases. Registration records for students are held in the UWVU, while JCU holds student files. Matters of student progression are overseen by JCU.
39 The University's Regulations provide the overall framework within which the validated programmes operate. The Handbook included detailed procedures designed to assure the standards of the awards, for example, guidance on assessing and examining students and generic criteria for degree classifications; this latter information is also made available to students in the Student Handbook. It was not clear to the audit team that these generic criteria had been applied to particular learning outcomes, either at programme or module level. The planned production of programme specifications will make this explicit, since the Handbook's advice is that assessment details should relate 'to the learning outcomes of each module and the degree scheme overall'. There is also a requirement that programme specifications demonstrate that awards conform to the relevant UK subject benchmark statements. JCU staff were aware of programme specifications and subject benchmarks and acknowledged their value. To assist the University in reinforcing confidence in the standards of its awards, the team recommends that programme specifications be put in place as a matter of priority with consideration given to the necessity for associated modification to procedures that assure the standard of awards. The team noted that the need for more detailed specification of aims, objectives and learning outcomes tied to methods of assessment was identified in the original validation report.
40 Both the moderators and the external examiners have a role in assuring comparability of standards, based on their general experience, and are required to report on comparability. Systematic calibration of student performance against achievements on similar programmes both within the University's directly delivered provision and within its validated provision appeared not to be a part of the approach to standards assurance. In the light of the University's recent decision to disband its Subject Panels, which would be a possible forum for consideration of such comparison of standards, it might be timely to explore the potential for a systematic and statistical approach to confirmation of consistency in the judgements of student achievement across the University's validated provision.
41 The Commentary signalled a number of areas for improvement in the operation of the collaborative link, all of which related to standards: the 'need to refine further the existing conversion scale employed for transposing US grade point average results onto the UK degree classification scale'; 'further encouraging JCU's internal examiners to use a broad marking range and not to 'bunch' students together'; and the need to introduce 'elements (for example, a dissertation within the Business Administration degree) which will clearly demonstrate students' academic achievement, at the required honours level, in the degrees' final year'. Although the validation report did refer to the 'vital' need to look at the way in which methods of assessment should reflect 'the academic gradient of the students' study', progress in this area seemed to the audit team to have been slow, given that students had already graduated from the degree programmes. The report also noted an agreement that 'a table of equivalents might be necessary in order to reconcile [the nature of the University of Wales degree classification system] with JCU's existing system'. The team formed the view that a series of systematic and pragmatic solutions had been adopted in order to allow students on the courses to be assessed. The team endorsed the need, as expressed in the Commentary, for further refinement of the conversion scale to reconcile differences between UK and US approaches was required to establish clearly the standards of the awards.
42 Procedures for the appointment of external examiners are rigorous; duties, explanation of role in validated arrangements and reporting requirements are all clearly defined. The University encourages the use of two external examiners for each degree course, one who has UK experience and expertise and another who is familiar with local HE systems. The JCU link has benefited from the appointment of two Italian professors to serve as co-external examiners. Although the SEC makes the appointments formally, in practice JCU made nominations that were accepted after a check against established criteria. External examiners' reports are processed through the annual reporting procedure; moderators play a key role in ensuring that due attention is paid to external examiners' comments and are also required to take a view on them.
Conclusions
43 The collaborative arrangement between the University of Wales (the University) and John Cabot University, Rome (JCU) provides for the validation of three programmes of study leading to the awards of: BA (Hons) Business Administration; BA (Hons) Political Science; BA (Hons) International Affairs. The collaboration has been in operation since the academic year 2000-01. In accordance with its Federal Plan, the University has established a central Validation Board (VB) and Unit (UWVU) to manage its validation work.
44 The UWVU, under the leadership of the Head of Validation Services, is both efficient and effective in providing guidance and support for the operation of the University's validated provision and in ensuring that policy and procedural requirements are met. The responsibilities of the VB in relation to the University's collaborative arrangements are clearly defined and exercised. The location of management responsibility beyond the VB for policy and oversight of the University's collaborative provision and for the work of the UWVU were not clear from the Commentary or the evidence base for the visit. A diagrammatic representation of the relationship of the VB and its subcommittees and the membership of senior staff of the University provided subsequent to the audit visits established greater clarity in this respect. The University may wish to consider whether it might include similar details of those responsibilities at senior management level in its policy and procedural documentation in relation to the maintenance of quality and standards in the University's validated arrangements.
45 The University provided a Commentary which described the partnership but was brief and schematic and its concision did not do justice to the good practice identified by the audit. The approach of the Commentary mirrored the overall style of University reports and minutes seen by the audit team which, in its view, did not represent the extent of the analysis, reflection and self-evaluation which discussion with staff and students confirmed as taking place. In its review of the principles and practice of its validation policy, the University should take account of the need for demonstrable evaluation and analysis of the operation of its validated provision.
46 The Commentary identified the JCU link as typical of the University's collaborative arrangements and the audit team was satisfied that the manner in which the degree schemes at JCU had been validated, monitored and developed was broadly representative of the University's approach.
47 The framework for the operation of the University's validated arrangements is set out in a regularly updated University of Wales Validation Handbook of Quality Assurance Policies and Procedures (the Handbook) which is clear and comprehensive, particularly the reporting pro forma. There was a clear commitment to adherence to the relevant precepts of the Code of practice.
48 A significant and effective feature of the University's approach to collaborative arrangements is the use of moderators for support of the linkages. Annual meetings allow moderators to exchange experience and promote the dissemination of good and consistent practice. The moderators for the link with JCU are clearly committed and enthusiastic about the partnership and provided proper advice and support to their colleagues at JCU in the development, approval and implementation of the collaborative provision.
49 The Handbook gives great attention to assessment policy and procedures. Current systems in this area are heavily reliant upon external examiners and moderators to ensure and confirm comparability of standards. The moderators are required to ensure that due consideration is given to reports from external examiners. The planned definition of programme specifications will assist in establishing the requisite transparency in matching assessment criteria and the achievement of learning outcomes and in obviating any potential for conflict between the developmental and standards-related aspects of the role of the moderator.
50 While the original validation was for a full three years of study, the basis for the arrangement appears subsequently to have changed to a recognition of year one of the JCU programme as equivalent to the first year of an award of the University, with validation being restricted to the final two years of the courses. This change in the arrangement was reflected in a revised Agreement for the operation of the provision but did not appear to have been formally approved by, or on behalf of, any University body with the appropriate authority. The University must move swiftly to clarify the way in which study at JCU articulates with the two years of the University programmes that lead to the validated awards. This issue is tied to the need for the University to assure itself that the joint delivery of the programmes provides sufficient intellectual challenge for the UK honours level awards for which it has responsibility. In this context, the impending consideration by the VB of the question of provision for students to graduate with two awards from collaborating institutions based on double counting of credit-bearing modules is welcomed.
51 Earlier audits by the Agency and the HEQC expressed confidence in the University's arrangement for the stewardship of quality and standards in its validated provision. On the basis of the evidence gathered with regard to the University's collaborative arrangement with JCU, the conclusion of the audit is that, once action is taken as a matter of priority to remedy the issues identified in this audit, it will be possible to express similar confidence in the University's stewardship of quality and standards in its collaborative provision both overall and in relation to the partnership with JCU.
Appendix A
University of Wales' response to the recommendations of the Agency's report of the collaborative partnership with John Cabot University, Rome
he Audit Report of the University's collaborative partnership with John Cabot University, Rome (JCU) will receive appropriate consideration by key University bodies including the Academic Board and Validation Board. Particular attention will be given to the Report's recommendations for specific action to be taken with respect to improving the collaboration with JCU and where more general consideration of policy issues is advised.
With respect to particular matters highlighted within the Report, the University would wish to offer the following comments:
Sections 12 and 18:
The Validation Board has instigated the process of reviewing the possibility of students gaining an additional degree qualification for undertaking the same scheme of study. In undertaking its review of the matter the Validation Board will gauge the extent and nature of the practice and will take cognisance of the relevant sections of the revised version of the Code of practice, Section 2: Collaborative provision.
Sections 16 and 19:
The matters of procedure raised in these sections will receive due consideration by the Validation Board.
Sections 21, 22 and 23, 39, 41 and 50:
A meeting between the course moderators, Head of Validation Services and senior representatives of JCU has been convened for late March 2004 to discuss the matters highlighted in these sections of the Report. A report of these discussions, together with appropriate recommendations, will be made to the Validation Board.
The University is committed to implementing any appropriate measures which lend to the further strengthening of its procedures for offering its provision on a collaborative basis.
Appendix B
Students registered on the programmes
|
Total Students |
Year 1 |
Year 2 |
Year 3 |
||
|
John Cabot University |
BA Business Administration |
56 |
- |
12 |
16 |
|
BA International Affairs |
- |
14 |
11 |
||
|
BA Political Science |
- |
2 |
1 |
||
Top
QAA updates
QAAtweets
QAAtube