Benchmark de modelos de linguagem de código aberto para text-to-sql em dados oncológicos brasileiros

Abstract

This work presents an experimental benchmark to evaluate open-source language models in the conversion of questions written in Brazilian Portuguese into executable queries over public oncology care data. The theoretical foundation covers studies on query generation from natural language, the association between terms in questions and elements of relational schemas, and evaluation based on execution results. The Qwen2.5-Coder 7B, DeepSeek-Coder 6.7B, and Llama 3 8B models were compared locally through Ollama in four scenarios defined by the presence or absence of the database schema and controlled linguistic variations in the questions. The evaluation considered execution-based accuracy, error categories, and generation and execution latencies. Qwen2.5-Coder 7B achieved the highest overall performance, with an accuracy of 80.0%, as well as the lowest average generation latency. Providing the database schema was the most influential factor, increasing the average accuracy of the models from 53.9% to 93.3%. The linguistic variations did not reduce average performance, suggesting stability under the superficial writing changes adopted in the experiment. The findings indicate that open-source language models can generate queries in controlled scenarios, although their performance depends significantly on the structural context provided and on the quality of question formulation.