Uma arquitetura conceitual integrando diagnóstico cognitivo e modelos de linguagem de grande escala para avaliação de habilidades

Abstract

The measurement of skills in Intelligent Tutoring Systems (ITS) is contingent upon the capacity to symbolise the knowledge under assessment, to deduce the student’s cognitive condition, and to utilise this information to direct the learning process. However, conventional approaches frequently employ aggregated metrics of correct and incorrect answers or address skill representation, cognitive diagnosis, and pedagogical adaptation in a decoupled manner. Cognitive Diagnostic Models (CDM) have been shown to assist in overcoming some of these limitations by providing multidimensional, interpretable profiles of skill mastery. However, the utilisation of these tools necessitates the establishment of relationships between assessment items and Knowledge Components (KC), which are typically represented by a Q-matrix. The construction of this structure is predominantly a manual process that is dependent on expert knowledge, which hinders maintenance and scalability. In this context, the principal contribution of this thesis is CAASA (Cognitive-Adaptive Architecture for Structured Assessment), a conceptual architecture for structured skill assessment in ITS. The architecture integrates item-KC representation via a Q-matrix, Large Language Model (LLM) support for generating and reviewing these relationships, skill mastery inference using CDM, student modelling, pedagogical adaptation, and diagnostic visualisation, within a single assessment and adaptation cycle. The CAASA architecture is the result of an incremental research process developed throughout the doctoral study. A preliminary review of the literature revealed certain limitations with regard to the assessment and modelling of STIs. Subsequently, a multi-agent architecture was explored, with the aim of investigating recommendation and pedagogical intervention mechanisms based on student actions. The research progressed by incorporating a CDM into an ITS, enabling the representation of relationships between items and KC via a Q-matrix, as well as the generation of individual and collective diagnostic profiles within a real-world educational context. The limitations observed during the process, specifically the necessity of manual construction of the Q-matrix, provided the motivation for a third study. This third study investigated the use of LLM as a mechanism to support the generation of item–KC relationships. The resulting Q-matrices were evaluated in terms of both structure by comparing them with expert-defined matrices and function by assessing their impact on cognitive diagnosis outcomes. The consolidation of these findings resulted in the formalisation of CAASA, which organises the components developed and evaluated during the research in a modular fashion and outlines the responsibilities associated with authoring, domain representation, assisted Q-matrix generation, cognitive diagnosis, student modelling, pedagogical adaptation, and result visualisation. The results demonstrated the feasibility of using machine-generated competence profiles to produce interpretable skill-mastery profiles in situated contexts. Furthermore, the results indicated that Q-matrices generated with language model support can achieve functional performance comparable to expert-crafted matrices in controlled scenarios, despite exhibiting structural differences. The findings presented herein provide evidence regarding the key mechanisms that constitute the proposed architecture. The thesis’s principal contribution is the formalisation of CAASA as an architecture that integrates representation, cognitive inference, assisted automation, and pedagogical adaptation into an integrated skill assessment process, offering a modular and scalable framework for the development of more interpretable ITS.