Preface
Quality Assurance of Overseas Collaborative Provision
The Quality Assurance Agency for Higher Education (QAA) was established in 1997 to promote public confidence that the quality of provision and standards of awards in higher education offered by UK institutions are being safeguarded and enhanced. As part of its activity, QAA undertakes regular academic quality audits of individual institutions to review the operation and effectiveness of arrangements for assuring quality and standards. A brief guide to the various quality assurance processes operating in UK higher education is available from QAA on request.
Quality audits also cover the arrangements which institutions use to assure the quality and standards of their awards and programmes offered in collaboration with other partners, both within and outside the UK. As part of this process, QAA audit teams visit overseas partners of UK institutions so that the same enquiries can be made about how the quality and standards of UK awards and programmes offered to students outside the UK are safeguarded, as are made of UK-based provision. This initiative is designed to help provide enhanced confidence in the work of British universities and colleges operating outside the UK.
QAA's audits of overseas partnerships use as their main reference point the Code of Practice for Overseas Collaborative Provision in Higher Education 1996 (2nd edition), which was published by the former Higher Education Quality Council (HEQC) and which QAA has endorsed for the purpose of the current programme of visits. This offers guidance on good practice, and a framework within which institutions can review and consider their current and future activities. The Code has been widely welcomed by universities and colleges. While UK institutions participating in the programme have not been narrowly 'measured' against the Code (which is not intended to be a definitive check list), their experience of using it, and the findings from the overseas visits in general, will contribute to its revision and further development.
This report is published following an overseas audit programme to Israel, undertaken in the spring of 1998 which looked at collaborative arrangements between several UK institutions and their overseas partners. It should be read in conjunction with QAA's (and, where appropriate, the former HEQC's) published audit report(s) on the UK university or college concerned, details of which can be found in this report.
The UK universities and colleges which participated in this overseas audit programme did so voluntarily and with the agreement of their overseas partners. Their collaborative links cover between them a range of programmes and subjects, levels of award and different forms of institutional partnership, involving a mix of partners from small, privately funded organisations to large, publicly funded universities.
Introduction
1 This is the report of an audit carried out by the Quality Assurance
Agency for Higher Education (QAA), of the quality assurance arrangements
relating to an association between the Edinburgh Business School (EBS),
which is the Graduate School of Business of Heriot-Watt University (also
referred to as 'the University' or 'Heriot-Watt') and Ramat-Gan College
(also referred to as 'the College') for the purpose of offering opportunities
to study for an award of the University, in Israel, specifically the MBA
by distance learning.
2 The audit of this association was one of seven undertaken in the spring of 1998, covering a range of activities which UK higher education institutions have established in association with institutions in Israel. In every case, visits were made to both the UK and Israeli institutions. Further information about quality audit and QAA is contained in the preface at the front of this report.
3 The general quality assurance arrangements of the University were the subject of an academic quality audit by the Higher Education Quality Council in 1993. Following a further audit, a report was published in 1995 which gave a more detailed consideration of the University's collaborative provision.
4 The Agency is grateful to the staff and students of Ramat-Gan College for their assistance and their co-operation with the audit team.
The audit process
5 Prior to the audit visit, the University provided QAA with briefing
documents, comprising a description of the University's process for the
validation and delivery of award-bearing courses overseas. The audit team
requested a number of supplementary documents concerned with the relationship
including the University's Quality Handbook; the relevant memoranda
of agreement; documentation on the initial approval and subsequent review
of the MBA; the most recent annual monitoring report or equivalent; and
any report on the suitability of the College for a closer relationship
and different role. The University was helpful in allowing the team sight
of an internal audit document on the work of the Edinburgh Business School,
which was, at the time of the overseas audit, under consideration by senior
committees.
6 One auditor, together with the audit secretary and the Assistant Director co-ordinating the audit, visited Heriot-Watt University in February 1998. Discussions took place with a number of University staff, including the Vice-Principal of the Quality Unit, the Project Director for the relationship with Ramat-Gan, and teaching staff responsible for the core and elective distance learning materials and the assessment of student work. Unfortunately, owing to his indisposition, it was not possible to meet the Director of the Edinburgh Business School. The University requested that liaison for the visit should be conducted directly by QAA with Ramat-Gan College rather than through its own offices. The team visited the College itself on 1 March and briefly on 9 March 1998, holding discussions with about 15 people, including the Head of the College, the Head of the Centre for Advanced Studies, teaching staff (including internal examiners) and students. After their return from Israel, members of the team briefly met staff of the University's agent, Financial Times Management, on 29 May 1998. A further meeting, involving the full audit team, QAA officers and members of the University's staff, was held in November 1998 to discuss the draft report.
7 The audit team comprised Dr S Jackson, Professor D J Murray, Professor D R Webb, auditors and Mr D C Attwood, audit secretary. The audit was co-ordinated for QAA by Miss A R Hynes, Assistant Director in the Institutional Review Directorate, who accompanied the team to Israel.
The audit context - Israel
8 The period of preparation for the audit and subsequent visit to
Israel coincided with discussion of changes to the law governing higher
education in Israel. An amendment to the law was enacted in February 1998.
QAA understood that the amendment made to the law further empowered the
Israeli Council for Higher Education (CHE) and altered what had been an
advisory relationship with regard to the educational activities of foreign
universities in Israel into one of a regulatory authority. It was explained
that these changes were designed to address internal concerns about the
activities of non-Israeli universities which were offering higher educational
provision within Israel. The period to which this audit was addressed was
prior to the implementation of this amending legislation.
9 Prior to February 1998, institutions conferring degrees on the basis of study undertaken at a partner institution in Israel could, at their discretion and in respect of certain forms of provision, seek a form of approval from the CHE. In the group of institutions considered, practice on this matter varied. Therefore, reflecting on paragraph 10.3 of HEQC's Code of Practice for Overseas Collaborative Provision, 1996, which states that '...students and applicants should be clearly informed whether or not the award for which they wish to register has been given official recognition...', the audit team based its enquiries around the position taken by each institution. It asked what statement, if any, was being made by the UK university about approval by the CHE or other government department or agency; whether the university monitored claims made by a partner institution; and whether any claims made accorded with a reasonable interpretation of a form of approval secured from the Council (see below, paragraph 52).
Matters given consideration in the audit
The University context
10 Edinburgh Business School (EBS) originated in 1995, bringing together the former Heriot-Watt University Business School and the Esmée Fairbairn Research Centre. EBS is one of the constituent departments of the University's Faculty of Economics and Social Studies. On 1 August 1997, EBS became a company limited by guarantee, with the University as guarantor. It has an independent chairman and a board, on which the University and EBS are represented. Notwithstanding its new legal status, academic issues remain under the governance of the University, with specific responsibilities carried by the Faculty of Economics and Social Studies and the Heriot-Watt University Senate.
11 EBS offers an MBA degree of Heriot-Watt University, which can be studied through a variety of programmes or routes. One of these is the distance learning mode which was first launched in 1993. The University states that this programme is based on 'study materials that are fully self-sufficient and there is no requirement for tutor or student contact'. Other routes to the award are by full-time study, part-time study and through what is termed by EBS a 'consortium MBA'. With the exception of the consortium route (which has content tailored to the needs of participating partner organisations), other routes have common course content and assessment. Following an amendment to the regulations in December 1997, distance learning students are now able to transfer to other modes of study.
12 The number of distance learning students enrolling on the modules which make up the MBA programme has increased rapidly, particularly in the past two years. In 1997 there were 6,547 matriculated students participating worldwide in the distance learning route. Information supplied to the audit team by the University suggested that in early 1998 there were 'roughly 900 - 1,000 students' undertaking the EBS Distance Learning MBA course via the College in Israel. By December 1998 the number had reached 1,275 students.
13 In 1989 the University entered into an agreement with Pitman Publishing concerning the Distance Learning MBA degree. The agreement related to the publishing, marketing and sale of the English language version of the Distance Learning MBA course materials and to the University's obligations to prepare the written materials, update these materials, maintain student records, conduct examinations and issue certificates to successful students. The agreement between the University and Pitman Publishing includes no provision relating to local agents or distributors. The University made no stipulations in the agreement concerning local agents and imposed no obligations on Pitman Publishing relating to them. In practice, Pitman Publishing (and from 1 January 1998, Financial Times Management) acting at its own discretion, has entered into agreements with local agents or distributors (the term 'distributor' is used interchangeably with 'agent') who promote the Distance Learning MBA in their areas, sell the course materials, and recruit students. Some of these agents offer their own live tuition to support the courses further. Ramat-Gan College has been one of these agents. Financial Times Management informed the audit team that it acts to verify the financial adequacy and logistical capacity of local agents, but that it does not deem it to be its responsibility to warrant the standard of any tuition that is offered by an agent.
The background to the association
14 Ramat-Gan College was established in 1989, in co-operation with the town of Ramat-Gan, a suburb of Tel Aviv. It has some 6,000 students registered on various courses leading to awards of other institutions. The College provides what is, in essence, a tutorial function for degrees taken elsewhere, including the BA degree of the Israeli Open University. There is also what the College described as 'co-operation in the delivery of legal education with the Hebrew University in Jerusalem', and it offers studies in accounting which prepare students for examinations leading to the award of Certified Public Accountant. The Heriot-Watt University Distance Learning MBA is delivered under the auspices of the College's Centre for Advanced Studies, although the audit team was told that this had no separate organisational existence within the College. The first cohort of MBA students was admitted in January 1994. At this point there was no contractual agreement linking the College with Heriot-Watt. Contractually the College acted as an agent of Pitman Publishing and, as noted above, the agreement between the University and Pitman Publishing contained no provisions relating to agents.
15 In November 1996 the University entered into a direct agreement with Ramat-Gan College and, in January 1997 made consequential changes to the terms of its previous agreement with Pitman Publishing.
The nature of the 1996-97 agreements
16 The agreement of November 1996 was between three parties: the University, the agent of the authors of the MBA course material, and Ramat-Gan College. The agreement covered a range of matters of which four are particularly relevant to this present audit.
First, the agreement related to the translation of MBA course materials into the Hebrew language, to their publication and to the rights and obligations of the several parties in these publications. Among the provisions was a stipulation that the translation and publication of the courses in Hebrew would be the sole responsibility of Ramat-Gan College but that in exercising this responsibility the College would conform to the academic standards and public reputation of the authors of the MBA Courses and of the University.
Secondly, the agreement provides for the University and the Authors' Agent to make available to the College new, updated or revised materials 'according to the requirements of the market in consultation with the Publishers (ie the College) who will have them translated into Hebrew'.
Thirdly, the agreement relates to the promotion of the MBA degree and the admission and registration of students. The agreement provides for the College to be solely responsible for the promotion of the courses forming the MBA package as translated into Hebrew, but that the College should conform to the academic standards and public reputation of the Authors and of the University. The University's prior approval was required for all publicity and promotional material which used the name or crest of the University. The agreement also states the University's sole responsibility for registering and admitting students to the MBA programme.
Fourthly, the agreement relates to examinations taken in the Hebrew language. The College was to provide facilities and invigilators (although the British Council, rather than the College, has, in practice, been used for this purpose - see below, paragraph 37), the conduct of examinations was to be in accordance with the University's regulations and 'the marking and grading of examinations and remuneration thereof shall be consistent with the practices existing at Heriot-Watt University'. At the same time the agreement affirms the University's sole responsibility for, among other matters, the setting, invigilation and marking of examinations and for decisions on student performance, but provides that the University can delegate the administration of its authority in this and other, unspecified, areas.
17 In parallel with this agreement between the University and the College, the University entered into a further and consequential agreement with Pitman Publishing. Under this, MBA courses in English would continue to be provided through the agent of Pitman Publishing in Israel, where these had not yet been translated into Hebrew, and also for any students in Israel who wished to continue to study in English. It was also provided that Pitman Publishing would report on, and continue to monitor, the production quality of the Hebrew version of the course. Otherwise Pitman Publishing 'will not be responsible for the marketing strategies or any of the activities of Ramat-Gan College'.
18 The University informed the audit team that it did not regard the relationship with Ramat-Gan College as involving a formal collaborative partnership. What it recognised was a primarily commercial relationship between Pitman Publishing and, subsequently Financial Times Management, and its agent, and the University saw this as having been extended under what it regarded as largely a publishing agreement relating to Hebrew translations. The University stated 'we do not have a partnership with Ramat-Gan College'. It maintains that it has no responsibilities in respect of students registered at the College, other than those which it has for any of its distance learning students.
19 The audit team accepted that the formal agreement between the University and the College did not contain any provisions governing the teaching which the College provided in support of the courses. Nevertheless it concluded that the agreements which the University had entered into with the College, and with Pitman Publishing, contained explicit and also implied undertakings and requirements about academic activities which were to be discharged by the College on behalf of the University. This applied in respect of the academic standard of the Hebrew language version of the courses; in respect of the quality in publicity, promotion and the recruitment of students; and, most significantly, in respect of academic activities which had a direct bearing on the operation of the University's assessment processes. The content of the formal agreements made it difficult for the team to accept the University's view that it did not have any formal academic relationship with the College.
20 The audit team was told, in the briefing documentation supplied to it, that Heriot-Watt University 'takes a caveat emptor approach to live tuition offerings and price charged', which includes provision through the College. The team noted that the formal basis for such a statement was that EBS had contracted with Pitman Publishing/Times Financial Management. EBS therefore regarded itself as being contractually at one remove from immediate involvement in, or having responsibility for, local arrangements for supplementary instruction. Consistent with this approach, EBS states in the Handbook issued to students that '(it) should be noted that such tuition is entirely independent of Edinburgh Business School'. At the same time the team noted that in its MBA Prospectus the University had included a flyer which listed its agents for the Distance Learning MBA, stating that these agents would 'be able to offer you a high level of support and services'. The team considered that this might reasonably be taken to imply some form of endorsement by the University as to the quality of the local agents' provision. Whatever might be the supposed legal arrangement between EBS and those organisations and individuals, in the view of the team, a reasonable reading of the publicity material would suggest they were acting in support of the University's programme and were doing so with the University's knowledge and some form of approval. The University has pointed out that 'Financial Times Management does monitor the services to students and has, on occasion, terminated agreements with Agents'. The University did not accept that if this was appropriate practice in relation to agents as a consequence of the original, 1989, agreement between Heriot-Watt and Pitman Publishing, there was a parallel obligation on the University to monitor services to students at Ramat-Gan following the agreements of November 1996 and January 1997.
21 With regard to further non-contractual links, the audit team noted the front cover of the Ramat-Gan College MBA Handbook which is in Hebrew. Translated this reads 'The Centre for Advanced Studies, Ramat-Gan College. Master of Business Administration on behalf of Heriot-Watt University, Edinburgh'. This Handbook also includes a reference to the fact that Ramat-Gan College students are registered with Heriot-Watt University, with an example of a University registration card being given. The Handbook also includes photographs of members of the Heriot-Watt University teaching staff. The team noted that the College appoints, on behalf of the University, examination markers who are publicly described as 'internal examiners' of Heriot-Watt University. The team considered that, taken together, these various signifiers created an impression of a wider association between the College and the University and of endorsement, formal or informal, by the University of the links between the two institutions. In particular there was no clear distinction apparent in practice between those activities of the College which were conducted directly on behalf of the University and those activities which the College provided on its own authority and for which the University bore no responsibility.
22 The audit team was told by both the College and the University that all publicity material was approved by the University. This is, in any event, a requirement of the memorandum of agreement between the University, the authors' agent and the College, although it appeared from conversations with staff at EBS that this had not been the practice so far as the Hebrew version of the Handbook was concerned. The team also learnt that, notwithstanding the caveat emptor principle referred to above, the University stated that it would be concerned if anything at Ramat-Gan were to jeopardise Heriot-Watt's reputation. The team further heard, during the course of its discussions, of visits to the College by senior staff from EBS, and other informal interchanges, including interchanges between the College and the University over matters to do with the academic content of course material and the adequacy of assessment items, with the observation being offered to the team by one member of the University that 'it's not entirely a hands-off relationship'. The University has subsequently denied strongly that most of these interactions have occurred, or, where they have occurred, constitute, or are indicative of, any collaborative relationship between itself and the College.
23 In the light of the complex arrangements described in the preceding
paragraphs, the audit team considers that the University's description
of its association with Ramat-Gan does not fully reflect the actual nature
of the relationship, which appears to go well beyond the extremely limited
form which was put before the team. This is not to deny the validity of
what the University believes it is doing, but simply to point out to the
University that there may be a danger of misunderstandings on the part
of many of those involved in the association, including students, if the
relationship between EBS and the College is in practice different from
that which is publicly stated to be the case. In consequence, the University
may be inadvertently unaware of the potential pitfalls of the present arrangements,
particularly the possibility that they could put its standards at risk.
Because of this danger, the team strongly recommends that the University
review the existing arrangement in order to ensure that it fully meets
the University's usual expectations and requirements for all aspects of
the academic quality and standards of its programmes and awards.
Initial validation and approval processes
24 The full-time and part-time MBA was originally approved by the University's Senate on 27 November 1984, with the distance learning route being approved by Senate on 6 March 1990, following a recommendation from the Faculty Board of Economics and Social Studies.
25 The audit team considered two aspects to the initial approval and validation of what is specifically offered in Israel: first, the approval of a link with the College through which the programme would be offered or supported; and, secondly, the approval of the study materials as made available and supported by the College. As already noted, the view of the University, as formally declared, is that the College is not a partner institution, and that the relationship with Heriot-Watt University is not therefore contained within the procedures that exist for collaborative provision. The team was told that the relationship was a commercial one. Until the revised agreement between the University and Pitman Publishing in January 1997, the appropriateness of the College, its physical setting and facilities, its ability to provide academic support, and its management and financial soundness, as with any other agent of Pitman Publishing supporting the MBA Distance Learning Programme, was not, formally at least, something with which EBS (or the University) needed to be concerned.
26 The audit team was informed through internal quality assurance material of the University (see above, paragraph 5) that in recent years Pitman Publishing/ Financial Times Management had tended to select agents who could provide a long-term partnership arrangement in which student services were provided in addition to the distribution of material. The University itself has further noted that this implies, in practice, a measure of devolution of responsibility to Financial Times Management for the establishment of what are, in effect, course delivery links, and has expressed concern that EBS is unable to monitor the performance of these agents. The team considers that these concerns are important ones and the University may wish to consider the desirability of ensuring that its judgement about the need to monitor the performance of its agents leads to appropriate action.
27 As already noted, in addition to an indirect relationship with the the College through Financial Times Management, the University formed in November 1996 a direct contractual relationship with the College (see above, paragraph 15). Under this particular contractual relationship, the University has added further elements characteristic of an academic partnership. The University did not regard this development of the partnership as requiring any validation or approval. In the light of the audit team's conclusion that this partnership has - or has developed - a significant number of features of collaborative provision, the University may wish to consider how it should now be applying its policy on the initial approval and validation of collaborative partnerships to this particular institutional link.
28 Students study for the MBA degree using materials that are described
as being self-sufficient, and student performance is assessed solely on
the basis of unseen examinations. At the College, students study using
learning materials translated into Hebrew. The audit team was told that
these are word for word translations. Although available for distribution
outside Israel, this study material is, in effect, specific to students
in Israel. Tutors at the the College told the team that while, for the
most part, the study materials had a general relevance to business environments
internationally, there were parts which were not accurate to business and
its specific environment in Israel. Tutors on the programme stated that
acting as professional academic teachers, and not on the basis of any stipulated
duties, they saw it as their responsibility to their students to inform
them about places in the study materials which, while appropriate to the
USA or UK, were not correct in the Israeli situation. At the same time,
tutors had to emphasise to students that they would be examined on the
information contained in the University's study materials and not on the
basis of what applied in Israel. The team considered that in producing
a version of course materials for students in Israel, the University should
have followed a procedure for evaluating and validating the study materials
and the way they were to be supported and assessed in the Israeli situation.
While it may not be possible to adapt the worldwide EBS material to every
country (although this itself raises a question of the validity of a programme
of this sort which does not give any attention to the local context in
which its students are living and working), given that the EBS has recently
extended to China the form of arrangement it is using in Israel, the question
of the extent to which locally-customised versions of the worldwide MBA
by distance learning should require some other modifications to the form
of the degree programme would appear to be worth asking. The University
may wish to consider the advisability of reviewing its procedures for validating
and approving its study materials to take account of these situations.
Language of instruction and assessment
29 The language of instruction and assessment for the programme as devised and delivered by Ramat-Gan is normally Hebrew, although it is also possible for students to be assessed in English (see below, paragraphs 39 and 40). The implications for the quality management of the Distance Learning MBA programme as a result of it being delivered in Hebrew are important and are considered throughout this report.
Arrangements for the admission of students
30 In its briefing documents, the University explained that the College
insisted on students undertaking the Hebrew version of the MBA having a
first degree or equivalent as a condition for registering for MBA tuition
at the College. This is a locally-imposed requirement, as the University
does not itself have any requirement of this sort for admission to the
MBA Distance Learning programme.
Arrangements for programme monitoring and review
31 The audit team sought to identify the arrangements that the University had implemented to monitor the distance learning provision it offered, and specifically the mode for students studying through the College.
32 The University describes its arrangements for the assessment of quality in the second edition of its Quality Manual, produced in December 1997. In this, reference is made to annual course reviews and departmental audits, as well as to external means of quality assurance such as Teaching Quality Assessment. Internal reviews are managed by the University's Quality Unit, which reports through the Executive and Deans Committee to Senate. These are the latest arrangements that the University has for programme monitoring and review, and supersede those which were described in previous HEQC reports. The Quality Unit also undertakes internal quality audits, a process that dates from 1991 when such activities were carried out under the auspices of the then Academic Audit Unit.
33 The audit team noted, from documentation made available to it, that in October 1995, Senate had received a summary report from the Postgraduate Studies Committee on a quinquennial review of the MBA by distance learning. In the course of this summary, it was noted that the Committee had agreed several points, the most significant for the purposes of the present QAA audit being that: 'arrangements for courses of study operated in foreign languages should be approved by the Faculty Board of Economic and Social Studies and the Senate and all examination scripts written in a foreign language should be returned to the University and a sample translation should be undertaken in the UK'; and that 'significant amendments to course packages should be approved by Faculty Boards, the Postgraduate Studies Committee and the Senate'. The Committee further noted that 'given the increasing scale of the participants...it was important to demonstrate that the strict academic quality controls were being maintained'.
34 Although the relevant memorandum of agreement between the University and Ramat-Gan College was signed on 26 November 1996, some 13 months after Senate had received the Postgraduate Studies Committee's report, the audit team was unable to locate any reference to the Senate requirements having been met in advance of the signing, which would have ratified, or at least reviewed, something that had already begun. The University may therefore wish to consider how it might meet its own requirements and resolve the apparently anomalous position of the MBA as delivered through the medium of Hebrew.
35 The audit team noted that the University's Quality Manual states '(a)udits of academic and non-academic departments will proceed in the future but their form and frequency may be influenced by developments on the national quality scene...'. In connection with this, the team was provided with the report of a recent internal academic audit of EBS. This was the first time that EBS had been subject to review. Amongst a number of positive and negative comments in this largely commendable report, indirect reference was made to the provision in Israel, including the observation that the University was not satisfied that a rigorous system was in place for the review of MBA programmes within EBS (see above, paragraph 33). Although, owing to the timing of the present audit, it was not possible for the team to be told what action the University planned to take as a result of the internal analysis, the team noted that a number of its own concerns appeared to be shared and identified by the University's internal process.
Student assessment and standards
36 The prospectus for the MBA states that 'all students, irrespective of study route, complete the same rigorous final examinations', with distance learning students taking these examinations in a variety of centres around the world. The University considers the quality of the programme to be primarily assured through what it sees as the demanding and robust nature of the examination. The Student Handbook for the MBA programme reiterates this by stating that '(t)he ultimate test of a student's worth remains the final examination'.
37 In the case of the Ramat-Gan students, the University contracts with the British Council to organise the examinations, provide invigilation, and despatch examination scripts to specified recipients. The audit team was told that the British Council in Tel Aviv is recognised by Heriot-Watt as one of its international examination bases. The British Council administers the examination protocols according to instructions from EBS. Examination papers are produced at Heriot-Watt University, the internal examiner drafting a paper which is typed within EBS. The paper is reviewed and proof read by the internal examiner and, once it has been approved by the appointed external examiner, the text is sent for translation into Hebrew by an independent translator. The ratified combined English and Hebrew papers are sent from EBS to the British Council Offices in Tel Aviv. The team learnt that initially there had been problems with the accuracy of the Hebrew translation of the examination paper. The understanding is that any potential ambiguity between the two texts is resolved by reference to the English version. The team noted that this meant that Hebrew-speaking candidates were asked to refer to a definitive version of a question in a language other than that in which they had been taught or in which they were being examined. The University has indicated that 'there was a quality check on the translation from the English to the Hebrew. This was performed by translating the Hebrew back into English and comparing the result with the original. The result was in all cases satisfactory'.
38 Examinations are supervised by invigilators recruited by the British Council. The audit team learnt of the measures adopted by the British Council to prevent impersonation, cheating or other departures from good practice in the conduct of examination sittings. Initially (that is, in December 1995, June 1996 and December 1996) examinations were conducted at the College, as arranged between Heriot-Watt and the College, but under the supervision of the British Council. The inadequacy of the physical environment at the College for unseen examinations caused use of this venue to be discontinued. The intervention of the programme's external examiner, who visited the College at that time, apparently contributed to the University reviewing the location of the examinations. The University's MBA examinations are the same, whatever the study route and wherever they are taken in the world, and there is no preparation of separate papers to deal with the difficulties of administering examinations globally in real time. The team concluded that this meant that Ramat-Gan College students would be sitting identical examination papers to people who had already sat the paper some hours earlier and others who were yet to do so. With the very large numbers of students registered on the degree throughout the world, the potential for the disclosure of examination questions appeared to exist. Since the papers offer no choice of questions, the unseen character of the examination paper is particularly important. The University has reported that all examination papers are returned to EBS whether or not they have been used and 'are therefore not available for circulation or transmission in any form'. The University has also reported that it has 'undertaken several extensive studies over many years of this problem and has minimised it by careful analysis of exam results to determine information leakage and by ensuring return of all papers by courier and by careful design of the exam papers.' The team did not see the reports of these studies or the analysis of results in respect of students being assessed at Ramat-Gan. It cannot, therefore, express a view on their validity or reliability. However, the University, as the awarding body for the MBA, will doubtless wish to continue to confirm regularly that these measures are effective and may wish to consider whether they are in themselves sufficient to ensure that its objectives of securing its academic standards are fully achieved.
39 In most cases, there are two components to the examination (open, essay-type on the one hand and multiple choice and true/false sections on the other). Where this applies, officials at the British Council separate the two sections of the examination scripts. Completed scripts are separately despatched to both Heriot-Watt (in the case of multiple choice segments of the answers which are not language-dependent) and to the College, in the case of the remainder of the papers answered in Hebrew or in English. British Council officials select 10 papers in each subject (20 in the case of those subjects where there are no multiple choice or true/false questions), to be copied and sent for independent translation. EBS supervises the translation and they are then forwarded directly to EBS for second marking. This, together with analysis of the marking distribution and the comparison of multiple choice answers, forms the basis for drawing inferences about the comparability of the grades given by EBS staff and Ramat-Gan College staff as examination markers. The audit team was told by officials at the British Council that the sample did not vary according to the size of the cohort sitting the examination. There was some uncertainty amongst EBS staff as to whether 10 or 10 per cent of the scripts were randomly selected for marking at EBS. Clearly this will have a bearing on the representativeness of the sample, since much will depend on the size of the group involved. The University will doubtless want to satisfy itself fully that an appropriate statistical basis for sampling has been established in order to verify the consistency of overall marking standards and may consider it necessary to review the current arrangements.
40 Ramat-Gan College students may answer their examinations in English, and a few choose to do so, but there is no mechanism for identifying these students in advance. Their examination papers go to the College, where they are marked by Ramat-Gan markers along with those who have submitted in Hebrew. The audit team noted that this means that those students who had submitted in English were not treated in a comparable way to other students on the Heriot-Watt MBA programme who were assessed in English. It appeared to the team that senior members of EBS were unaware that there were Ramat-Gan students whose papers in English were being excluded from consideration with the general population of scripts written in English.
41 Marking is undertaken at the College by individuals who are appointed as what the College refers to as 'internal examiners'. These are not always the same people as those who are appointed by the College as tutors. The audit team was told by senior managers at the College that 'internal examiners' would have their position approved by Heriot-Watt University, on the basis of curricula vitae sent to the University. The University, however, does not recognise them as internal examiners, referring to them as 'markers'. None of the 'markers' who met the team were aware of having received any formal appointment for their marking work from Heriot-Watt University, a point confirmed by the University. From information given to the team, it concluded that the examiners at the College considered themselves to have been commissioned by the College and not by the University. Given that the University announces in its Prospectus that 'all scripts are marked by Heriot-Watt faculty', the University will wish to ensure that its arrangements for the appointment of markers and internal examiners are clarified and appropriately approved and may consider it necessary to establish a more formal system.
42 Markers at the College are now normally provided with marking guides which come from EBS, a development which, the audit team was told, was initiated by the University markers at the College. The aim of the guides is to co-ordinate marking standards within the Distance Learning MBA scheme. It was unclear to the team whether or not these guides were used solely by Ramat-Gan markers or also by others within EBS. There is feedback given to the Ramat-Gan based markers which compares their marking profile with that of papers that have been marked by the staff at EBS. Markers to whom the team spoke, considered that this feedback had helped to develop consistency in marking over time. The team concluded that whilst this approach had served as a basis for establishing comparability as time passed, it was not sufficient for the standardising of marks at each presentation.
43 Students told the audit team that it usually took two and a half months for their results to reach them, whereas the time for those assessed in English was given in the Handbook as approximately six weeks. The team noted that this meant that students enrolled at Ramat-Gan for a subsequent module without knowing the outcome of their previous examination. The University may wish to consider the advisability of exploring ways of bringing the experience of Hebrew-speaking students into line with that of those being assessed in English, or at least to correct the claim made in its Student Handbook about the time taken to release marks.
44 The audit team noted that EBS had, in effect, devolved to the College a substantial responsibility for the security, management and processing of the examinations (as allowed for in its memorandum of agreement). The College has accepted this responsibility and its staff mark the work of those students they have taught, placing these 'distance learning' students in practice in the same position as students who have undertaken one of the taught routes offered by the University. The team further learnt that the translators of the EBS examination papers sometimes contacted Ramat-Gan markers for technical advice about the wording of questions. Given this, and the overall centrality of the examination to the standards of the MBA scheme, the University will wish to ensure that it has the means of adequately ensuring the probity of these arrangements and may wish to consider the advisability of introducing a more formal system where it operates in languages other than English.
45 The University's Code of practice for the external examination system for first degrees and postgraduate courses states that 'an external examiner has the right to see any work that contributes to the assessment', and that '(t)hey should see a sample of scripts from the top, middle and bottom of the range. They should normally be sent all scripts of borderline candidates'. In addition, the Student Handbook states 'the External Examiner reads...all marginal cases so that you are assured of as fair a judgement as possible'. Although senior University representatives responsible for quality assurance told the audit team that the work of the students at the College would be seen by the external examiner, it seems that the arrangements within EBS preclude this happening. The team was informed that the external examiner did not see the selected translated material, so none of the work coming through from the College is even theoretically likely to come under the scrutiny of the external examiner. The University has, however, stated that the external examiner 'does examine the [Ramat-Gan] procedures and the statistical distribution of results for each diet and investigates any problems'. Notwithstanding that, taking these arrangements together, the team concluded that the provisions for marking do not accord with what is said in the MBA Prospectus nor what is stated in the Student Handbook, namely that 'all scripts are marked by Heriot-Watt faculty and are subject to rigorous scrutiny by external examiners from other UK universities', nor to the expectations laid down in the University's Quality Manual. In this particular instance the team noted that an opportunity for establishing comparability of standards had been reduced. The University will wish to ensure that none of its students can, because of procedural idiosyncrasies, be exempted from potential scrutiny by the external examiner and it may further wish to consider the necessity of ensuring that the involvement of its external examiners for this award is as extensive as for other awards of the University. It will also recognise the necessity of reviewing the claims made about its MBA to ensure that they accord with the factual position and all the circumstances.
46 The audit team further noted the arrangements for monitoring the marking arrangements of the Ramat-Gan College 'internal examiners'. These procedures were described in the University's briefing document supplied to the team. On the basis of the sampled and translated scripts, EBS 'calculates the average of the marks as a percentage of the total mark for each element' and then compares 'the difference between the Heriot-Watt marker and the Ramat-Gan marker. If this difference is greater than 15 per cent (absolute points) then an adjustment is made in line with that of the Heriot-Watt marker'. The University explains that the 'trigger' of 15 per cent 'is an arbitrary choice but one which is well within accepted bounds'. The University has subsequently stated that the trigger is in fact 7.5 per cent. The team noted that even the variance of up to as much as 7.5 per cent could cross important assessment bands, especially those of failure and distinction. According to the detailed account provided by the University of its examining procedures, the University does not identify for additional scrutiny those examinations papers where an individual candidate is on a borderline. In particular, the team understood, from what it was told, that where the marking of a script in Hebrew by a Ramat-Gan marker placed a candidate on the opposite side of a borderline from what was indicated by the mark extrapolated from the sample reviewed by the second marker at EBS, no additional scrutiny was undertaken to establish the appropriate result. The University has subsequently explained that 'there is no such concept as a 'borderline student' at EBS. A student either passes or fails' and that there is 'no subjective agonising over individual cases at the examination board meeting'. This confirmed to the team that there were no means by which a borderline student (both on the pass:fail and pass:distinction boundaries and including those candidates whose marks and extrapolated marks fall on different sides of the boundary), entered through Ramat-Gan, could come before the Board of Examiners. As already noted, the examination is centrally important to the integrity of the whole MBA scheme, and taking account of the stated principle of equivalence within the MBA assessment regime, the University should review very carefully EBS's arrangements for examining, marking and determining results relating to students being examined on its distance learning programmes to ensure that they fully meet the University's own expectations and requirements.
47 According to its formally stated position, Heriot-Watt University has no responsibilities for students at the College (although it has stated that it has 'concerns for students' at the College). For the distance learning route to the MBA, the University sets its responsibilities as being with the standard of the course material and the robustness of the examination. As already noted, the arrangement with the College is not regarded within the University as a collaborative partnership, but rather as an 'agency' and as such has not fallen within the University arrangements for either approving or monitoring what is on offer to students studying there.
48 In view of the differences in marking protocols between the UK and Israel, where the usual pass mark on a 100 point scale appeared to be very much higher than at Heriot-Watt University, the audit team sought to see if students understood the calibration that prevailed at Heriot-Watt University. It was clear that they did, even though it had taken them some time to accept this scale fully. The team considered that this might account for the number of appeals about grading. Students are able to appeal to Heriot-Watt on matters to do with the assessment of their work. The briefing document provided to the team by Heriot-Watt notes that 'Israeli students, like any others have the right to appeal; all such appeals to date have been concerned with examinations results'. Nevertheless the team noted that neither the MBA Handbook in English, nor the Ramat-Gan course Handbook in Hebrew, makes explicit mention of appeal arrangements. The University's data reveal that in 1996-97 there were 87 appeals on the ground of dispute over grading, with a third of these leading to the adjustment of marks. An appeal can lead to marks being lowered as well as raised. The University will need to consider the necessity of ensuring that the basis for appeals by students studying on the Hebrew version of the distance learning programme of the MBA are consistent with what is permitted by its regulations, and, in particular, those which might refer to the admissibility of appeal against academic judgements.
49 In theory, the experience of those students taking the MBA by distance learning depends almost entirely on the effectiveness of the course materials. EBS explains that 'study materials...are fully self-sufficient and there is no requirement for tutor or student contact'. However, the audit team was concerned to hear how infrequently the material was updated and to learn from the Ramat-Gan staff of their unease at teaching material that they viewed as out-of-date, or that included in some cases incorrect answers to exemplar multi-choice questions. The team understood that a revision of the material was planned, but was unable to identify a formal and regular system for this purpose. Of more unease to the team was the apparent lack of response from EBS when Israeli academic colleagues had formally drawn these matters to their attention. The team was informed that the distance learning material produced in English by Financial Times Management has relatively short print runs and was regularly updated by staff at EBS, in between the normal seven year cycle of complete re-writing. In view of what the team was told by staff at Ramat-Gan, these revisions do not apparently appear in the Hebrew language versions used in Israel. If the quality of the student experience on the distance learning mode is specifically dependent on the quality of the materials provided - as EBS says it is - the University may wish to consider the desirability of ensuring that the Hebrew and any other non-English language materials are also included in its arrangements for the updating and review mechanism which apply for the quality assurance of its other academic provision.
50 The MBA distance learning programme runs a help line for students, which is described in the English language MBA Student Handbook issued to all students. However, students with whom the audit team spoke had not used this, relying instead on what they described as the Ramat-Gan College 'facility' to liaise with EBS should this be necessary. As there is no full and equivalent Hebrew translation of the English language MBA Student Handbook, students studying in Hebrew may not be aware of the help line. If the University wishes a measure of parity in the experience of students studying for the MBA in Hebrew, it may wish to consider the advisability of harmonising the content of student handbooks.
Staffing and staff development
51 Arrangements for the appointment of tutors rests with the College although,
as already noted, those with marking responsibilities are expected by the
College authorities to have their position ratified by EBS. Those who are
markers (and many of those teaching on the programme at the College who
do not have this role) are very well qualified, holding simultaneous appointments
at other Israeli universities. There is contact between the College and
EBS staff over the grading and marking of work, and through the Ramat-Gan
link at least one Israeli academic has been invited to teach at EBS. A
number of the Israeli tutors, spoken to by the audit team, indicated that
they had brought to the attention of their Heriot-Watt University colleagues
various concerns about the course material or the examinations. The rapidity
of responses to these observations had apparently varied. The team concluded
that there was no systematic staff development except, in a limited way,
around the matter of assessment protocols, but this was consistent with
the University's view of its relationship with the College.
Claims made for approval in Israel
52 The audit team understood from the University and the College that attempts had been made in the past to obtain formal approval from the Israeli Council for Higher Education (CHE) but that recognition for distance learning provision had been outside the CHE's remit at that time. The law concerning approval and licensing of higher education undertaken by foreign universities in Israel was changed immediately prior to the audit visit and the University may in consequence wish to clarify the position of students and its relationship to Ramat-Gan. In particular the University will wish to ensure that the information given to students in future editions of the Handbook reflects fully and accurately the basis for any Israeli recognition of their award. In discussions following the audit visit, the University intimated that it had in fact been in discussion with Ramat-Gan for some time to see if arrangements could be made to meet the legal requirements of the new law. If this were to be achieved, then it is likely that the basis of the association with Ramat-Gan would need to be fundamentally changed, as would the relationship of this route with other forms of the MBA distance learning degree.
Publicity and promotional materials
53 As already noted (see above, paragraph 22), the audit team was told that publicity material relating to the Ramat-Gan initiative had to be approved by Heriot-Watt University, with the Director of EBS approving on behalf of the University any publicity that included the name or crest of the University. The team understood that there had been concerns expressed through the University's own quality assurance arrangements about the effectiveness of the control exercised by EBS over the accuracy of information provided to the media. In addition, while the University's internal academic audit did not comment directly on the accuracy of the publicity of the EBS itself, the discussion of accreditation in that report does, by implication, cast doubt on what is suggested in the section on course accreditation in the MBA Student Handbook. The University will wish to establish appropriate means for it to ensure the accuracy of statements in publicity material that is issued under its name and imprimatur.
54 The University provided the audit team with a sample certificate that would be issued to students completing the MBA via the College. This makes no reference to the fact that the qualification had been secured in Hebrew. The HEQC Code of Practice for Overseas Collaborative Provision in Higher Education 1996 (2nd edition) recommends that 'very careful consideration needs to be given to the implications and consequences of not recording, on certificates...the language of instruction, where this is not English'. Since the Hebrew language mode had not been considered formally and fully by the University, this consideration could not have been given, nor was there any evidence that the issue, including the implications and consequences of this change, had been given any serious thought by those with operational responsibility for the programme, or by those with responsibility for quality within the University more widely.
Conclusions
55 The Distance Learning MBA is the best known activity associated
with the Edinburgh Business School of Heriot-Watt University. This provision
has expanded rapidly over recent years, and now has almost world-wide coverage.
The University has contracted out much of the work associated with the
production, marketing, sales and distribution of the course texts to Financial
Times Management Ltd, which in turn appoints local agents around the world,
some of whom provide tuition to back up the course materials. Students
are not required to buy any of this supplementary tuition and the University
claims no responsibility for it. The University has represented that student
satisfaction with the distance learning programme is high - 'we have 9,000
well satisfied students undertaking this postgraduate course'.
56 The introduction of the Hebrew language version of its materials in conjunction with the College in 1996 represented a notable departure from previous arrangements. In developing its association with the College the University has sought to protect the distinctive nature of the Distance Learning MBA, in particular the self-sufficiency of the course materials. The University maintains that its relationship with the College is limited to the commercial arrangements concerning the course materials, and is not a partnership or academic relationship. It further maintains that it has no responsibility for students registered at the College to undertake Ramat-Gan's tuition programmes in connection with the Hebrew version of the Distance Learning MBA.
57 The University's relationship with the College does, however, go further than that. Its formal memorandum of agreement makes provision for the College to conduct the University's MBA examinations, and it supplies markers for the Hebrew (and some English) scripts. There is also some evidence of more extensive interaction between the two organisations, although much of that may be at an informal level. The University does not accept that any additional interaction has been of significance.
58 The implications of this disparity between the University's stated position and the totality of its interactions with the College, as they may be perceived by others, are important. A number of the areas in which the College is acting on behalf of the University, notably in the marking of scripts and grading of students, are central to the maintenance of the University's academic standards, need to be acknowledged as such, and the arrangements reviewed by the University itself, not just by EBS. It is important that all departments for which the University is responsible should come fully within its processes for quality assurance, and that reporting and monitoring arrangements are sufficiently robust to secure compliance with its own requirements.
59 The University is in the process of introducing enhanced arrangements for the assurance of quality, and a continuation of internal auditing procedures, begun in 1991, is built into this. A largely commendable internal academic audit report on the EBS, produced early in 1998, whilst not focusing on the Distance Learning MBA delivered through the College, did contain some specific criticisms of the processes through which EBS, and thereby the University, could assure itself of the quality of its programmes. The report's recommendations for strengthening procedures, if implemented, would go some way to ensuring that a number of the ambiguities apparently inherent in the approval and monitoring of the provision were addressed.
60 EBS has begun to implement plans in other countries for further arrangements similar to those it is operating in Israel. In monitoring the establishment and operation of these arrangements, the University will wish to satisfy itself that they meet its expectations and requirements for the security of academic quality and standards in all relevant respects, using robust, rigorous and independent means to do so.
Top