GLORIA: ferramenta interativa integrada a um Large Language Model para auxiliar na modelagem de dados e na implementação de técnicas de Game Learning Analytics

Abstract

Game Learning Analytics (GLA) encompasses a set of techniques for collecting and analyzing player interaction data (logs), making it possible to track players’ progress throughout a game and identify evidence of learning. GLBoard is a model that enables the implementation of these techniques through a generic JSON data template. However, learning designers, particularly students who perform this role in academic courses, may face difficulties when modeling data for educational games, such as selecting and justifying the data to be collected according to the game’s learning objectives. Data definition guides the creation and organization of GLA variables within a data template and provides the foundation for efficient data collection. In this context, Generative Artificial Intelligence (GAI), particularly Large Language Models (LLMs), may offer an alternative. These models can generate natural-language texts and perform human-like tasks, and researchers have already used them in Learning Analytics and Game Analytics contexts. Accordingly, this dissertation investigates data modeling for educational games and the implications of using LLMs in the GLA field. It operationalizes its findings through the proposal and evaluation of a tool integrated with an open-source LLM, named GLORIA (Game, Learning, Observation, Recommendation, Intelligence, and Analytics). Its purpose is to support data modeling in GLA, and it may later be incorporated into the GLBoard system as an introductory module. The research adopts the methodological and epistemological approach of Design Science Research (DSR), and its results relate to the following cycles: (i) Relevance: literature reviews and investigations into the challenges of data modeling and the use of LLMs for GLA, as well as the implementation of a Participatory Design process; (ii) Design: the creation and evaluation of an intelligent agent for GLA, the development of the tool using an educational game design methodology, and the evaluation of GLORIA; and (iii) Rigor: the development of a knowledge base derived from the literature findings and the investigations. The tool was evaluated through a heuristic evaluation involving UX/UI experts and an empirical study involving learning designers. The results provide evidence that GLORIA can support data modeling for educational games. They also indicate that LLMs for GLA can act as knowledge mediators by encouraging learning designers to reflect on their decisions through scaffolding. However, the tool still requires refinement, particularly regarding its visual design and interaction with the chatbot. The main contribution of this dissertation is to broaden discussions on data modeling for educational games and investigate the use of LLMs to support this process. The research produced valuable findings and a tool that guides learning designers through their first steps in GLA, encourages them to reflect on this stage, and may help them learn how to model data.