How good lusophones are data science Llm agents?: evaluating agentic approaches for data science in portuguese contexts

Abstract

Data Science (DS), a complex field often involving advanced concepts of statistics, mathematics, and programming, has historically been restricted to deeply knowledgeable human professionals. Recent advancements in artificial reasoning and the expanding toolset of Large Language Models (LLMs) have challenged this paradigm. More specifically, agentic frameworks, which provide LLMs tools like code execution and document searching, have pushed automated DS performance to near-human levels. However, current research on LLM-based methods for automated DS is still heavily biased towards the English language. In this work, we provide an assessment for agentic DS for prompting, data, and metadata in Portuguese in comparison to English. To this end, we expanded on previous advancements to develop DATAEXPLAINER, an agentic framework for generating DS solutions with step-by-step explanations in the target language. We employed the aforementioned framework to evaluate four different LLM models on a set of seven different translated Kaggle competitions. All assessed LLMs produced coherent Portuguese solutions (more than 97% of text generated in the correct language), with the agent powered by GPT-OSS surpassing the average Kaggle user on five out of the seven evaluated competitions. There is still a gap in performance, with 10.1% lower mean scores for Portuguese across models when compared to English; however, the difference did not reach statistical significance. Additionally, the data indicates a larger gap for regression tasks and code-tuned models, and a smaller gap for thinking models.