What we are doing

Ask a GenAI tool almost any question from a university course and, within seconds, it will give you an answer, one that is usually fluent and often correct. For a student working alone between lectures, the temptation of GenAI is easy to understand: why struggle with a difficult concept when a chatbot will explain it, summarise it and turn it into a tidy set of revision notes? However, anyone who has taught in HE knows that reading a good explanation isn’t the same as being able to give one. The sense of understanding that comes from following someone else's reasoning can be remarkably persuasive, but remarkably fragile. Students might finish a study session feeling confident about a topic, only to discover in an exam, or in a later module that builds on it, that the confidence might have been misplaced.#

Support of student learning through AI-mediated reflective practice is a collaborative project, funded by QAA, across Leeds Beckett University, Nottingham Trent University and Cardiff Metropolitan University. Our project starts from the question,

What if, instead of asking the AI for answers, students were asked to supply the answers themselves, and the AI's job was to question them?’ 

It’s focused on a deliberately straightforward tool: a standardised framework of six prompts that participants from undergraduate and postgraduate modules work through after each weekly topic. The prompts ask them to explain what they have learned, to connect it with what they already know and to apply it to situations they haven’t yet met.

The AI is pre-instructed to withhold answers, so it won’t explain or correct in the usual sense; instead, it acts as a questioning coach, probing for gaps, picking at half-formed ideas and asking students to say more when an explanation sounds plausible but turns out, on closer inspection, to be thin. Explaining an idea in your own words, and then having that explanation tested, exposes the weak points in your understanding in a way that passive reading rarely does. Research on metacognition suggests that deliberately helping students to become aware of, and to regulate, their own learning can support both learning and performance¹. The approach also speaks to work on feedback literacy, which argues that students' capacity to assess their own work is closely bound up with how well they can make sense of, and act on, feedback² .

The reflection transcripts are data in their own right. We score them with a rubric built on Biggs and Collis's structure of the observed learning outcome (SOLO) taxonomy³, which describes the structure of a response rather than its subject matter: does a student offer a single relevant idea, several ideas set side by side, or an explanation in which those ideas are related to one another and to the question? Because SOLO is concerned with structure, the same rubric can work across disciplines, with each module anchoring it to its own intended learning outcomes. Accuracy is scored separately, since a well-reasoned answer can still be wrong, and each closing summary is also read for whether the participant can name the limits of their own understanding.

Because the framework is level and platform-agnostic and needs only free-tier access to a GenAI tool, no participant is disadvantaged by being unable to pay for a premium subscription. Confidence with digital tools is unevenly spread, however, and an approach that assumes everyone is comfortable typing to a chatbot could easily widen gaps rather than narrow them. Support for students with varying levels of digital confidence is therefore built in

Where we are now

The project is now under way. We have recruited participants and they’re working through the framework each week. We don’t yet know what we’ll find: some participants might find a questioning AI more frustrating than helpful, or the benefits might vary between disciplines. What we are sure of though, is that the question of how AI is used in learning, and not just whether it is, deserves careful and honest evidence.