Download Comparative Analysis of Student Feedback Mechanisms in Higher Education and more Summaries Technology in PDF only on Docsity!
Learning and Teaching Support Network
Collecting and
using student
feedback
A guide to good practice
By John Brennan and Ruth Williams
This Guide is based on a HEFCE-funded project undertaken by the Centre for Higher Education Research and Information, Segal Quince Wicksteed Limited and National Opinion Polls on the collection and use of student feedback on quality and standards of learning and teaching in higher education. (See Brennan J, Williams R, [in collaboration with SQW Ltd and NOP], Collecting and Using Student Feedback on Quality and Standards of Learning and Teaching in Higher Education. A report to HEFCE. The report can be found at http://www.hefce.ac.uk/pubs/rdreports/2003/rd08_03/). It uses information on the policies and practices in the universities and colleges who responded to a consultation and visits to 20 of these. Use has also been made of other material we have come across or have been made aware of during the course of the project. The Guide also draws on the discussions of a seminar held in March 2003 with practitioners and others with an interest and expertise in student feedback (Annex A lists the participants of the seminar). We are grateful to the institutions, the seminar participants and others in helping us to produce this Guide.
About the Authors
John Brennan is professor of higher education research at the Open University and director of the Centre for Higher Education Research and Information. He has directed many projects in the areas of quality assurance in higher education and the experiences of graduates. His books include ‘Managing Quality in Higher Education’ (with Shah) and ‘Courses, Students and Jobs’ (with McGeevor, Lyon and Murray). He has also written many reports and articles on higher education policy topics, within UK and international contexts. Originally a sociologist, Professor Brennan is currently active in many international higher education research and policy networks. From 1 January 2004, he will be directing a major new four year research project entitled ‘What is learned at university. The social and organisational mediation of university learning’, part of the ESRC Teaching and Learning Research Programme.
Ruth Williams is a projects and development officer with the Centre for Higher Education Research and Information of the Open University. She graduated in politics and international relations and has a masters degree in higher and further education from the University of London. Her work in the Centre includes research on issues concerning quality assurance. This publication is based on research she undertook for the Higher Education Funding Council for England looking at the ways in which UK higher education institutions collect and use feedback from students on the quality and standards of their programmes of study. She is undertaking similar research in Scotland.
Section four: Student representation and other methods 43
4.1 Student representation 43
4.2 Staff student liaison committees 45
4.3 Discussion groups 46
Section five: Actions and decision-making 49
5.1 Processes 49
5.2 Actions taken outside committee 50
5.3 Student input to strategy 50
5.4 Data for other users and uses 51
5.5 Other sources of information 51
Section six: Publication and dissemination 53
6.1 The importance of feeding back to students 53
6.2 What should be published? 53
6.3 How should it be published? 54
6.4 The need for improvement 55
Section seven: Conclusions 57
Annexes
Annex A Good practice in student feedback seminar i
Annex B Suggested further reading ii
Annex C University of Essex, Student Assessment of Teaching Questionnaire: Spring 1999-2000 iii
University of Ulster, Assessment of Teaching Student Questionnaire: 2002-2003 iv
University of Hertfordshire, Student Feedback: 2002-2003, Staff and Module Questionnaire viii
Annex D University of Gloucestershire, Student Evaluation Form, Field of English Studies, Field of Creative and Contemporary Writing x
University of Salford, Module Evaluation Questionnaire xi
Birmingham College of Food, Tourism and Creative Studies, Unit Evaluation Questionnaire xii
Annex E Bolton Institute, Student Satisfaction Questionnaire: 2002-2003 xiii
Coventry University, Course Feedback Questionnaire xvii
Annex FUniversity of Central England in Birmingham, Centre for Research into Quality, Extract from the 2002 Report on the Student Experience at UCE xviii
Annex G Cost drivers for the centralised system run by one institution xxvi
Annex H Cost drivers for the online system run by one institution, including the transition from the paper-based system xxvii
Annex I University of Wolverhampton, School of Art and Design, Extract from the Module Guide (DS2019: Developing Personal and Professional Skills Through Student Representation) xxviii
Annex J University of Gloucestershire, Student representative job description xxx
Annex K University of Central Lancashire, Student liaison officer job description xxxi
Annex L University of Reading, Director’s Handbook for the Course Representatives Scheme xxxii
Annex M Nottingham Trent University, Mid-year Review Questionnaire: Full-time xliv
Annex N Chelsea College of Art and Design, School of Design, Extract from the Yearly Course Review Process xlv
Annex O University of Ulster, Module Evaluation – Agenda for a Module Forum xlvi
Annex P University of Hertfordshire, What Students Think of UH, Questionnaire Results 2001-2002 xlvii
Annex Q Sheffield Hallam University, ‘You Talk We Listen!’, Action from the Student Experience Survey xlviii
Annex R Loughborough University, School of Sport and Exercise Sciences, Extract from the Module Handbook (O2PEC102: Applied Sports Science – Year 3 Cricket) li
Structure of the Guide
The Guide is organised in terms of six main sections which follow the ‘student feedback cycle’ set out in the diagram below:
- Purposes
- Choice of mechanisms
- Student feedback questionnaires
- Student representation and other methods
- Actions and decision-making
- Publication and dissemination.
Section three and Section four focus on the most commonly used feedback mechanisms: questionnaires, student representation on committees and staff student liaison committees (or their equivalent). Section four also looks at the use of discussion groups.
We have used examples throughout the Guide to help illustrate some of the points made. These examples are drawn from the information supplied to us by the
universities and colleges consulted and visited in the HEFCE-funded project, and elsewhere. As noted above, it is important to bear in mind that the examples used are specific to a particular institutional or departmental context and purpose, and as such may not always translate well to other contexts and purposes. In drawing on these particular examples, we recognise that many other institutions will have established similar procedures and initiatives.
But before we move into the sections, we first include a discussion about the changing context within which student feedback is now collected and used.
A changing context
New methods of quality assurance
The new methods of quality assurance in higher education in England and Northern Ireland, as described in the consultation document, Quality assurance in higher education (HEFCE 01/45) promise a ‘lightness of touch’, a greater recognition of the responsibilities of individual higher education institutions for setting, maintaining and reviewing quality and standards, and meeting public information needs.
The new arrangements replace processes of external subject review with a greater reliance on and utilisation of the processes and outcomes of institutions’ own quality assurance procedures. In most institutions, these latter have developed substantially over the last ten years, responding in part to the requirements of the external quality assurance procedures operated variously by the higher education funding councils, the Higher Education Quality Council (HEQC) and the Quality Assurance Agency for Higher Education (QAA).
Among the pieces of information arising out of the operation of institutional quality assurance arrangements, the Final Report of the Task Group on Information on Quality and Standards in Higher Education^2 recommended that the following should be available in all HEIs :
- information on institutional context
- information on student admission, progression and completion
- information on the HEI’s internal procedures for assuring academic quality and standards.
Among the latter would be ‘information on student satisfaction with their higher education experience, covering the views of students on:
- arrangements for academic and tutorial guidance, support and supervision
- library services and IT support
- suitability of accommodation, equipment and facilities for teaching and learning
- perceptions of the quality of teaching and the range of teaching and learning methods
- assessment arrangements
- quality of pastoral care’ (p 7-8).
Moreover, the Task Group went on to recommend that some aspects of information on the quality and standards of teaching and learning should be published. The two elements of such information relevant to the project and this Guide are:
- feedback from recent graduates, disaggregated by institution, collected through a national survey
- feedback from current students collected through HEI’s own surveys, undertaken on a more consistent basis than now.
However, whilst the former has been confirmed by the White Paper 3 , the project has recommended that institutions should not be required to publish the results of internal feedback, although such publication may well occur where institutions deem it appropriate.
The Task Group’s recommendations are also detailed in the annexes of the QAA’s Handbook for Institutional Audit^4 ( Annex E ) as ‘information sets’. These are referred to as Part A (information which should be available in HEIs) and Part B (information which should be published). The Handbook states the following:
The audit team will routinely require access to all items in Part B of the information sets. It will also need access to some of the material in Part A, according to the precise nature of its enquiries.
Furthermore, and of relevance to this Guide, the Handbook advises institutions to consult with students about the contents of the institution’s self-evaluation document, which is submitted prior to the audit team’s visit. In addition, the Agency invites students, through their representative body, to make their own written submission to the audit team. Within the guidance offered in the Handbook regarding the content
(^2) A Task Group was set up to produce recommendations on ‘the categories of data, information and judgements about quality and standards of teaching and learning that should be available within higher education institutions (HEIs) and those which should be published’ (Information on quality and standards in higher education. Final Report of the Task Group, HEFCE 02/15). (^3) Cm 5735 (2003), The future of higher education. London: HMSO.
The new national survey of graduates may primarily be about providing information to inform student choice but its existence will also need to be taken into account by institutions in developing their internal systems of feedback. Depending on the final decisions about the timing of the national survey, institutions will need to avoid duplication in the collection of feedback at the end of the final year. More positively, the national survey will provide institutions with comparative data and the means of benchmarking. The national survey may throw up questions for institutions that will need to be explored further through internal feedback.
The twin processes of institutional audit and subject review have clearly done much to build up quite extensive formal procedures of quality assurance in most higher education institutions. Yet, as much of the literature on the subject emphasises, quality assurance has to achieve a balance between accountability and improvement and it has been suggested that reactions to some aspects of quality assurance have been marked by compliance rather than commitment, at least in some departments and institutions. In particular, that which is imposed from outside the unit or institution in question may be viewed with some suspicion by those within. Sensitive to these features of academic culture, many institutions have allowed considerable variation in the implementation of quality assurance procedures, partly to reflect subject differences and partly to maximise the commitment of staff. Devolved procedures to strengthen staff commitment and to achieve the improvement functions of quality assurance may limit the consistency of approach between and within institutions and the comparability of the information that is generated by such procedures.
In summary then, the context for the use of student feedback includes the following:
- a considerable growth in formal institutional arrangements for quality assurance in recent years, in which student feedback surveys play an important part
- variation between institutions and between departments in the details of these arrangements, to reflect the growing diversity of higher education and its students and to achieve ownership and commitment by staff
- increasing pressures on staff as a result of worsening staff student ratios, RAE etc
- the existence of a lot of institutional information about quality and standards but a lack of consistency and comparability in much of it
- a growing emphasis in public policy on consumer choice and competition between institutions – creating new needs for information on quality and standards
- a new national survey of final year students
- a possible tension between quality assurance and quality enhancement.
With new demands arising from changes in context, feedback arrangements have tended to expand to accommodate them. Which brings us to a final contextual point: the need to ensure ‘value for money’. In student feedback as in everything else, questions of what is needed must be balanced by questions of what can be afforded. Thus, in considering making changes to student feedback arrangements, institutions will have very real cost considerations to take into account.
Purposes
This section focuses on the purpose of collecting and using student feedback. The issues discussed include purposes as identified by institutions, the range of different users and purposes, the value of discussion about purposes, and the implications for level, timing and standardisation of feedback processes.
It is perhaps a rather obvious point to stress the importance of clarity about the purposes of an activity. Yet we found in our project some ambiguity within institutions about the purposes of student feedback and this seemed often to be associated with a lack of commitment on the parts of both staff and students to both the collection and use of feedback data.
1.1 A range of purposes
In responding to the consultation exercise, institutions referred to the main purposes of student feedback as:
- enhancing the students’ experience of learning and teaching
- contributing to monitoring and review of quality and standards.
Other purposes cited included:
- ensuring the effectiveness of course design and delivery
- enabling a dialogue with students
- helping students reflect upon their experiences
- as part of the teaching and learning process
- identifying good practice
- measuring student satisfaction
- contributing to staff development.
Although overwhelmingly concerned with a notion of quality enhancement, purposes nevertheless differed in emphasis: for example according to whether student feedback was considered to be a part of the student learning process or rather a commentary on that process; according to whether it was seen as being primarily about whether programme objectives were being achieved or providing an opportunity to critique those objectives.
Underlying these differences of emphasis could be different conceptions of student feedback. While some people equated student feedback with student satisfaction, this view was by no means universal. Student feedback could be accounts by students of their learning styles and study methods – clearly potentially valuable to their teachers but not calling for evaluation or opinion from the students. Student feedback could be student views about whether their objectives had been met. While such information is clearly related to ‘satisfaction’ it is not necessarily the same as feedback on satisfaction with the teaching and learning processes of specific programmes, modules 7 or services.
Alongside the majority view of the purposes of student feedback as being concerned with quality enhancement, some people noted the role of student feedback in staff appraisals and promotions; this is discussed further in Section three. We might also note possible new purposes – emphasised by Government and part of the new external quality regime – of feedback to inform intending students about the quality and standards of higher education programmes. Relatedly, student feedback on particular modules might well be used to inform module choice by students within an institution. Whatever its stated purpose – or combination of purposes - the collection of student feedback must take account of its intended uses and the nature of the institutional quality assurance and enhancement procedures.
Section one
(^7) The term ‘module’ is used to refer to the components or units of curricula structures that are unitised or modularised.
standardisation of data – both in regard to what is collected, how it is analysed and how it is presented (see below). Users closer to the ‘chalk-face’ have little need of standardisation and it may be in conflict with their needs. This might imply different data collections to meet the purposes of users at different levels, or perhaps reliance on some mechanisms rather than others to achieve certain purposes (for example, student representatives rather than feedback questionnaires).
1.5 Use at other levels?
Recognition that purposes differ, especially at different levels within an institution, could of course result in a proliferation of feedback activity as every user emphasised the unique nature of his or her needs for feedback. In such situations, the question inevitably arises as to whether data collected at one level for one purpose can also be used at other levels for other purposes. Can data be aggregated or, for that matter disaggregated, to serve different purposes? It is certainly the case that aggregating module feedback does not convert it into feedback on the student experience of the programme as a whole. It does, however, provide some information on the module ‘set’ that constitutes the programme and this is likely to be of interest to those with responsibilities at this level. Module level feedback can be of use at the programme level in conjunction with other relevant data, for example student profiles, progression data, external examiner reports. Nevertheless, if feedback is to be obtained about the student experience of a whole programme of study, this is likely to require feedback obtained at that level. This might be achievable through a one-off exercise during a 3 or 4 year programme and so does not represent a large additional load upon staff and students, who may also be obtaining feedback on each separate module. It is also likely that data obtained by the new national survey of graduates will provide institutions with some valuable information on the student experience at the broad subject level.
1.6 Timing
Another discussion point should be that of timing. When should feedback data be collected? When will it be needed? When can it be used to best effect? End of module questionnaires are common but they cannot provide information to improve the learning experiences of the particular group of students taking the module. Maybe that is not important, as long as there are sufficient opportunities for informal exchange between students and teachers to render the use of formal mechanisms redundant. But this may not always be the case. A mid-point feedback questionnaire might be useful on large enrolment programmes but, if this is followed by an end of module questionnaire, it may be seen as overkill by students and staff alike, especially if the module is semester length.
A further point about timing of feedback concerns its links to action and change. At higher levels within institutions, ensuring that feedback is available to meet the timetable of committees may be essential if it is going to affect decision-making and action. Thus, it may be necessary to work back from the committee timetable to determine when feedback should be obtained.
1.7 Standardisation
A further discussion point is whether the purposes require feedback arrangements to be standardised. As indicated above, for some purposes this will undoubtedly be helpful. For other purposes, not only will standardised arrangements be unhelpful but a bespoke, one-off exercise might be called for. Thus, if the effects of a particular programme innovation are to be assessed, specific feedback on that innovation will need to be obtained. It is by no means clear that standardised arrangements would achieve the purpose. Or to take another example, the purposes of collecting feedback from students on an entirely new degree programme may differ from the purposes of collecting it on
an established, long-running degree. Standardisation, along with centralisation, is discussed further in Section three.
1.8 Clarity of purpose
The main point we would emphasise, however, is that the purposes of student feedback should be clear to all involved – especially students - if their commitment to the process is to be maximised. Ten years ago, Pat Partington and colleagues produced a CVCP document which advocated that students should be made aware that (i) they will not suffer for their feedback, (ii) they are being listened to, (iii) the process is being taken seriously, (iv) something will be done because of it,
and (v) they are actively involved as partners in the process^8. We would endorse all those points today.
We found a number of examples where purpose was stated explicitly – especially in the use of questionnaires. It is also important to state how feedback will be used and how results/actions will be disseminated to students. Such information is often stated in guidelines to staff and students. However, we believe it is especially important to state purpose and use, and how results and actions will be reported at the point when feedback is being requested. A number of institutions indicate how this will be done on the questionnaires they use, as illustrated by the following examples.
Box 1: Some examples of purpose statements and how results/actions will subsequently be reported on questionnaires
“You are asked to complete this form to assist the university in its procedures for the maintenance of academic standards and to enable staff teaching this course to assess and make appropriate modifications as needed. The results of this evaluation exercise will be displayed on appropriate departmental notice boards.” ( Manchester Metropolitan University )
“The University continually seeks to review and improve the quality of its teaching and learning. The purpose of this questionnaire is to enable staff to use student opinion in their review of this module. Your responses are highly valued and the questionnaire should only take a few minutes to complete. Notes of guidance are available. No individual responses will be identified and your contribution will be aggregated with all others. Comments should be constructive and truthful. They might cover such aspects as the subject matter, the style and delivery of the lecturer, classes or any other part of the module. The aggregated responses will be considered by your School’s Staff-Student Committee, and other committees where necessary, and feedback will be given to you.” ( University of Salford )
“In reflecting upon your learning, it would be helpful to have your views on the unit of study/module just completed. The information will be considered by your tutor and the module team and a response to issues raised will be posted publicly. You may give your views anonymously. Please complete as much of the form as possible within the time allotted.” ( University of Brighton )
“Your Programme Team and the Institute are committed to continuously improving your experience of the education we offer. The responses you make to the following statements will be taken seriously and acted upon. You will be made aware of any action to be taken, or where it is not possible to respond positively at present, reasons will be provided.” ( Bolton Institute )
(^8) Partington P (ed) (1993), Student feedback – context, issues and practice. Sheffield: CVCP.
Purposes: section summary
In summary, we believe that in many institutions there would be value in reviewing the purposes of student feedback and in ascertaining whether these are shared and understood across the whole institution. This might involve asking the following questions:
- Have the purposes of collecting and using student feedback been discussed sufficiently and do they take account of the needs of different users?
- Is feedback required on student satisfaction, on learning processes (study methods) or on student objectives and their achievement? Is it possible to combine them and if so, has this been done?
- Have the needs of users at different levels of the institution been clarified and has consideration been given to whether these can be met by a single set of feedback arrangements?
- Can different purposes and the needs of different users be accommodated? (For example, between quality enhancement, staff appraisal and informing student choice.)
- Have the implications for the timing of the collection, analysis and presentation of feedback been considered in relation to the different uses of feedback?
- Do the purposes of different users allow the possibility of standardised feedback arrangements? Has discussion taken account of both the advantages and disadvantages of standardisation (see Section three)?
- Has discussion taken account of the extent to which feedback data might be aggregated from module level to serve the needs of users at other institutional levels?
- Are the purposes of collecting and using student feedback clear to all involved in the process? Do they explicitly state how, when and where results and actions taken will be disseminated to students?
- Has discussion taken account of the most effective mechanisms for gathering student feedback?
- Has discussion taken account of QAA’s expectations in seeking the views of students?
Choice of mechanisms
This section considers the types of mechanisms that exist for collecting student feedback and those which are most commonly used by institutions. Each has implications for collecting students’ views, analysing and interpreting the results, and feeding back the results and actions taken to students. Thus, we have also included at the end of this section a basic summary of the advantages and disadvantages of the different mechanisms.
2.1 Types of mechanisms
Institutions use many mechanisms to collect student feedback, both qualitative and quantitative. They include:
- Questionnaires
- Student representation on local and institutional committees
- Staff student liaison committees (or their equivalent)
- The lecture or seminar
- The tutorial
- Discussion groups (for example, focus groups, structured group discussions, nominal group technique and so on)
- Other informal mechanisms.
2.2 A combination of mechanisms
Any single mechanism has its drawbacks. Feedback through questionnaires is generally considered to be relative or indicative rather than absolute. It may for example be affected by factors such as the timing of the distribution of the questionnaire and whether it is distributed ‘in-class’ or by some other means. Student representation on the other hand cannot be relied upon to be truly representative of the student body. In recognition of the limitations of any single mechanism, all institutions use a range of mechanisms, generally ensuring a mix of qualitative and quantitative feedback. Quantitative
feedback (for example, through questionnaires) can be used to provide ‘evidence’ that something is going well or not so well and such evidence will normally be required for quality assurance purposes. Qualitative information (for example, through open-ended response sections of questionnaires and from student representatives) can help explain why something is going well or not so well.
Thus, while questionnaires are by far the most commonly used mechanism, we found few situations where complete reliance was placed upon them, or indeed upon any one mechanism. Most institutions use a combination of formal mechanisms – normally questionnaires, student representation and staff student liaison committees – although informal channels of feedback are also highly valued.
Indeed, a number of institutions require or encourage departments to use a combination of mechanisms. Nottingham Trent University’s Staff Resource Pack^9 on student feedback provides details of a number of techniques, including the following:
- Structured feedback sessions
- Focus groups
- Self research
- Log books
- Tutorials
- Questionnaires and surveys
- Shadowing
- Course representatives
- Informal chats
- One minute methods
- Student diaries
- Suggestion boxes/books.
In addition to outlining the purpose, structure and outcomes of each technique, including the ideal group size, it describes the pros and cons of each mechanism.
Section two
(^9) The NTU Centre for Learning and Teaching (1998), Student feedback: a staff resource pack (part of a HEFCE-funded Sharing Excellence Project).
2.5 Benefits and support
Whatever mechanisms are used, all involved in the process – staff, managers and the students providing the feedback – need to be convinced of the benefits of gathering feedback and the processes involved. This is most effectively achieved if students and staff find that they are making use of the results of feedback, for example, students in choosing their options, staff in revising their modules/programmes, departments in planning new programmes, and marketing departments in promoting their institutions.
Many institutions have policies and codes of practice on collecting and using student feedback, although how often they are consulted is a matter of speculation. We recognise that support to staff in the collection and use of student feedback goes much further than written policy statements. Other types of support are as follows:
- Induction processes to emphasise the importance and benefits of feedback from students
- Appraisal and development processes to review how staff collect and use student feedback and what further support they might require to make it more effective
- Support at departmental and/or institutional levels and the provision of training regarding what mechanisms are most effective for what purposes, types of analyses of questionnaire data, how to handle abusive or negative comments, how to act upon feedback, and how to feed back results/actions to students.
None of these support processes are without resource implications and institutions will need to weigh up the costs alongside the benefits of the different mechanisms for collecting and using student feedback.
The table overleaf is a brief and basic summary of the advantages and disadvantages of each of the mechanisms mentioned above as reported to us in the project. We recognise that these will be very familiar to many readers of this Guide, although not so familiar to others. For a fuller description of the advantages and disadvantages, readers may wish to consult the list of further readings in Annex B.
Table 1: A summary of some of the advantages and disadvantages of mechanisms used to gather student feedback
Mechanism Advantages Disadvantages
Questionnaires Useful for gathering Need for some specialist skills (e.g. in responses on many issues questionnaire design and analysis) at various levels
Inclusive (assuming a census Tend to be ‘ex post’ and a good response rate)
Can provide quantitative and Frequency and number may induce qualitative information questionnaire fatigue
Depending on the degree of Response rates may be low standardisation, can provide comparisons and trends
Does not require a great deal May be costly in terms of time spent of time commitment from on analysis individual students
Students may not get information on results and any actions taken
Actions may not get taken
Can become ritualistic
Student representatives/staff Provides a means of Possible lack of motivation and student liaison committees on-going formal dialogue participation by students between staff and students
Can provide student input Representatives may of a ‘feed forward’ nature be unrepresentative
Good opportunities for Representatives may not be visible dialogue and testing to the student population they out ideas represent
Actions can be taken Lack of opportunity for representatives immediately or relatively to communicate with the student quickly population
Provides personal Time taken to ensure that development opportunities representatives are effectively trained for students in the necessary skills
