Project Title: Testing ChatGPT in IR classrooms: Potentialities, Limitations, and What’s Next

Project Team Members: Paola Rivetti, David O’Mullane, Rituparna Banerjee paola.rivetti@dcu.ie

Audience: The project targeted first-year students of International Relations enrolled in the module, Introduction to International Relations and Security, which comprised 203 students in 2023/2024. The students volunteered to participate in weekly laboratories (eight weeks, six sessions per week). An average of 10 students attended each session, resulting in roughly 70 students participating each week, excluding the final week of the semester when attendance lowered.

Aims/Objectives: This project aimed to revise the alignment of our understanding of the learning objectives, our teaching, and the most suitable assessment methods in an environment where GenAI is present and utilised regularly. We considered whether to ban it or instead teach students how to use it responsibly and ethically, as well as develop a shared understanding of its capabilities and limitations (if anything at all).

Project Description: The project targeted first-year students of International Relations enrolled in the module, Introduction to International Relations and Security, which comprised 203 students in 2023/2024. The students volunteered to participate in weekly laboratories (eight weeks, six sessions per week). An average of 10 students attended each session, resulting in roughly 70 students participating each week, excluding the final week of the semester when attendance lowered. A team of three educators tutored students: the module coordinator, Paola Rivetti, and two assistants, David O’Mullane and Rituparna Banerjee. During the laboratory sessions, we conducted practical exercises using ChatGPT in a computer-equipped room. Students could work individually on a computer or collaborate in groups of two to three. They autonomously chose whether to work alone or in groups and selected their group members independently. We used the free version of ChatGPT. Pedagogically, the exercises were designed to foster a shared understanding between students and educators of how to use ChatGPT and its potential applications for learning about International Relations, building on the module’s contents discussed in class. Each exercise was followed by the students’ feedback. We conducted an end-of-the-semester survey, receiving responses from 72 students. Ethics approval was granted by DCU Research Ethics Committee. For each session, the team had specific intended learning outcomes, including how to write a good prompt, interact with the machine, and cross-verify information. In the lab, exercises were run with the students under the close supervision of the team members, who could interact with individual students or small groups of two. From the educators’ perspective, the workload involved preparing laboratories, which included identifying in-class exercises relevant to the intended learning outcomes. We also supervised students during the laboratories, led the sessions and interacted with them during the exercises. Additionally, we assessed the students’ performance and the feedback they offered after each laboratory.

Project Impact: At the end of each laboratory, students were asked to provide feedback on the utility of the exercises and what they learnt about ChatGPT during the session. At the end of the semester, students were also asked to participate in a short survey reflecting on the strengths and weaknesses of ChatGPT for studying International Relations, and the likelihood of them using it in future as a study aid. Seventy-two students completed the survey. Generally, we have found that GenAI presents significant limitations, and students should learn how to use it effectively from the beginning of their university careers. Assessment methods should be calibrated not to prevent students from using ChatGPT (although this is a possibility), but to allow them to use ChatGPT only for what it is good for and to teach them how to use it appropriately. As such, involving students in these pedagogical approaches helps them foster a critical outlook towards AI tools rather than rejecting or embracing them unquestioningly. In the end-of-semester survey, respondents reported scepticism about the tools, which evolved into discovering their potential uses in structuring work and gaining broad insights, primarily in providing non-reflective assistance in research which did not need discernment and critical decision-making. Overall, while a shift occurred towards acknowledging its utility when employed effectively for academic purposes, many participants reported inaccuracy of information, unreliability, and limited usefulness for targeted and specific tasks; the more specific the task was, the less the GenAI chatbots’ reliability. Beyond the core benefits of student learning, the laboratory offered several secondary advantages. It partially addressed challenges associated with a shortened semester and a large class size, and effectively bridged a technology gap. Finally, the team learned that they, and educators in general, are indispensable because ChatGPT’s suggestions are based on probabilistic codes and words, and the machine lacks the nuance, contextual knowledge, and pedagogical consideration that a human educator brings. Concerns also remain about the risk of language uniformity and standardisation. However, as many survey answers suggest, students will likely continue using ChatGPT, even when aware of its limitations. We need to adapt our education system to incorporate these technologies. Furthermore, this should encourage institutions to invest money and resources into educating educators and students about GenAI chatbots – something that, unfortunately, is not happening at an appropriate level, raising concerns about the quality of education being provided.

Key Learnings: Beyond the core benefits of student learning, the laboratory offered several secondary advantages. It partially addressed challenges associated with a shortened semester and a large class size, and effectively bridged a technology gap. Finally, the team learned that they, and educators in general, are indispensable because ChatGPT’s suggestions are based on probabilistic codes and words, and the machine lacks the nuance, contextual knowledge, and pedagogical consideration that a human educator brings. Concerns also remain about the risk of language uniformity and standardisation. However, as many survey answers suggest, students will likely continue using ChatGPT, even when aware of its limitations. We need to adapt our education system to incorporate these technologies. Furthermore, this should encourage institutions to invest money and resources into educating educators and students about GenAI chatbots – something that, unfortunately, is not happening at an appropriate level, raising concerns about the quality of education being provided.

Further information: https://ucclibrary.pressbooks.pub/genai/chapter/how-should-we-revise-our-assessment-methods-in-political-science-international-relations-considering-the-impact-of-generative-artificial-intelligence/, https://doi.org/10.1017/S1049096524000817