Uso de códigos-solução para prever a dificuldade de exercícios de programação em sistemas juízes online

Abstract

Teaching programming represents a major challenge in the contemporary educational context. The technical nature of the topics, combined with the requirement for structured logical reasoning, makes the acquisition of programming skills especially complex for novice students. In response to these difficulties, numerous technological tools have been developed to support the pedagogical process, notably online judges—platforms that enable the automated, real-time submission, grading, and evaluation of code. These systems rely on repositories that store information across various topics, constituting valuable databases for both instructors and students. However, a significant gap remains: the absence of effective mechanisms to automatically estimate the difficulty level of newly registered exercises. This limitation impacts instructional planning, potentially causing imbalances in learning progression and affecting student motivation. In this scenario, this work proposes a methodology to automatically estimate the difficulty of new exercises in online judges based on their reference solution codes. The approach combines textual and semantic comparison techniques, utilizing Levenshtein distance and the CodeBERT model. The central hypothesis posits that the difficulty of a new exercise can be inferred from the historical record of existing exercises with similar solutions in the database. For experimental validation, a dataset comprising 958 exercises and 42,000 submissions from an introductory programming course at the Federal University of Amazonas (UFAM) was used. The results indicate Pearson correlation coefficients between 0.49 and 0.67 in the presence of sufficiently similar exercises, demonstrating the feasibility of the approach and highlighting the importance of historical database coverage for estimation efficiency.