When asking llama3.3:70b about its supported natural and programming languages it lists more than a dozen each. As a user I am usually asking questions in one natural language for one programming language. Wouldn't LLMs be much more efficient/smaller/capable, if they were restricted to certain natural and programming languages? But no such LLMs seem to exist. Why?
2 Answers
$\begingroup$
$\endgroup$
4
The model wants to generalize concepts and for that it needs a lot of training data. There is not enough training data for that in most languages (some Models have more than 300 billion parameters). So if you use multiple languages the concepts learned in one language are transferred to another language. Tough fine tuning on a specific language could improve performance.
A Module trained on only one language would be more efficient if you had unlimited training data in the language but you don't have unlimited traing data.
-
$\begingroup$ Your argument concerning training data is probably valid for most natural languages. But wouldn't an LLM restricted on training data using only English language need far fewer parameters while still being accessible/usable for most programmers? Wouldn't that save a lot of computing power and energy consumption? $\endgroup$coproc– coproc2025-10-30 03:59:56 +00:00Commented Oct 30, 2025 at 3:59
-
$\begingroup$ How does an LLM (that is based on predicting probabilities for the next token) "generalize concepts"? Can you give me hints/links for further reading? $\endgroup$coproc– coproc2025-10-30 04:01:55 +00:00Commented Oct 30, 2025 at 4:01
-
$\begingroup$ @coproc for the compute power yes I also think so. I think the reason why they don't exist is most of the time most people want the best model and don't care about the energy consumption. Most company restrict or remove older models as soon as the new one come out. Because they would have to have delicate GPUs that constantly have the old model (in our case alternative model) loaded and that can cause GPUs not working (because no one is using the old model ) the GPUs Depreciation so the company looses money. $\endgroup$joerg-de– joerg-de2025-10-30 21:52:23 +00:00Commented Oct 30, 2025 at 21:52
-
$\begingroup$ @coproc "BLOOM: A 176B-Parameter Open-Access Multilingual Language Model" arxiv.org/abs/2211.05100 has some infos about it they call info that is not in one language only in another but is still answered in the first language zero-shot. You can also read the references in the Introduction of "Probing the Emergence of Cross-lingual Alignment during LLM Training" arxiv.org/abs/2406.13229 $\endgroup$joerg-de– joerg-de2025-10-30 21:57:36 +00:00Commented Oct 30, 2025 at 21:57
$\begingroup$
$\endgroup$
I think you will find that in reality it is focused
restricted to certain natural and programming languages
There are really only a "handful" (?) of internet (natural) languages; around 50% (probably more) is English, with some other 30% taken by a few of the widely-used languages and before reaching rank 10 the usage drops below 1%.
Google "ranking of languages of internet"
As about "coding" I would argue that "teaching" an LLM a programing language is a task almost completely separated from the general natural languages training.
