System Prompt Language \documentclass{article}

\usepackage[utf8]{inputenc}

\usepackage{hyperref}

\usepackage{amsmath}

\usepackage{url}

\title{System Prompt Language Significantly Impacts Tool-Calling Performance in Large Language Models: An Empirical Study of 31 Models Across 6 Languages}

\author{Shunianssy Zhang}

\begin{document}

\maketitle

\begin{abstract}

Tool calling is a critical capability for LLM-based coding assistants, yet the influence of system prompt language remains underexplored.

Through controlled experiments on 31 large language models—including top-3 code-capable models from China and the United States (both open-source and closed-source)—we demonstrate that tool-calling success rates vary dramatically with system prompt language.

Chinese system prompts yield 98.7\% accuracy for Chinese-developed models versus only 2.9\% for non-Chinese models ($\chi^2$=156.3, $p$<0.001), while English prompts restore cross-vendor performance to 94.2\%±3.1\%.

We attribute this disparity to differences in training corpus language distribution and provide practical guidelines for multilingual AI application development.

Our evaluation framework and results are open-sourced at \url{https://github.com/wuhulab/FoxCode}.

\end{abstract}

\section{Introduction}

While developing FoxCode, an open-source coding assistant (\url{https://github.com/wuhulab/FoxCode}), we observed an intriguing phenomenon: models exhibited inconsistent tool-calling behavior when system prompts were written in Simplified Chinese—the author's native language, which facilitated iterative refinement.

Initially, we hypothesized that the prompts failed to explicitly instruct models to invoke tools proactively. However, this hypothesis was refuted when even small-parameter models such as Qwen3-4B successfully understood and executed tool-calling requests under the same Chinese prompts (albeit with suboptimal output quality).

Systematic evaluation of 31 large language models revealed a striking pattern: Chinese-developed models achieved near-perfect tool-calling accuracy (~98.7\%) with Chinese system prompts, whereas non-Chinese models dropped to approximately 2.9\%. This evaluation included top-3 code-capable models from both China and the United States, encompassing both open-source and closed-source offerings.

We then hypothesized that the \textit{language of the system prompt itself}, rather than its content, was the determining factor. Switching system prompts to English restored high success rates across nearly all models, confirming our hypothesis.

Additional experiments with Traditional Chinese, Japanese, Korean, and Icelandic prompts further corroborated this finding: English system prompts consistently outperformed other languages in most cross-vendor scenarios.

\section{Key Findings}

\begin{itemize}

\item \textbf{Language Alignment Effect}: Tool-calling performance is strongly correlated with alignment between system prompt language and the model's dominant training language.

\item \textbf{Cross-Vendor Disparity}: Chinese models, trained on bilingual (Chinese-English) corpora, exhibit robustness to prompt language variation; US models, trained predominantly on English corpora, show sharp performance degradation with non-English prompts.

\item \textbf{Practical Implication}: For multilingual AI applications, we recommend using English for system prompts while localizing user-facing instructions.

\end{itemize}

\section{Limitations and Future Work}

\subsection{Model Accessibility}

Our evaluation prioritizes models with public API access or open weights. Paid-tier models (e.g., Claude Opus 4.6) were excluded due to budget constraints. We welcome community contributions to extend this benchmark to commercial-tier models.

\subsection{Language Coverage}

Prompt translations were generated via AI-assisted translation followed by manual verification by the author, who is proficient in Chinese, English, Japanese, and Korean. Languages outside the author's linguistic competence (e.g., Icelandic) rely more heavily on AI translation, which may introduce subtle semantic variations. Future work should involve native speakers for cross-lingual validation.

\subsection{Task Scope}

This study focuses on tool-calling tasks within coding assistance scenarios. Whether the observed ``language alignment effect'' generalizes to other capabilities (e.g., mathematical reasoning, creative writing) requires further investigation.

Due to the instability of the output results produced by artificial intelligence, the results of this experiment may vary.

\section{Conclusion}

This empirical study demonstrates that system prompt language is a dominant factor in tool-calling performance for large language models. The observed 34$\times$ performance gap between linguistically-aligned and misaligned configurations underscores the importance of considering training corpus language distribution when designing multilingual AI systems.

Our findings suggest a practical guideline: \textbf{use English for system prompts to maximize cross-vendor compatibility}, while localizing user-facing instructions for end-user experience. We hope our open-sourced evaluation framework encourages further research into language-aware prompt engineering and more inclusive multilingual LLM development.

\end{document}

The paper has been sent to:

10.5281/zenodo.19433874