Norwegian language and 10 Large Language Models

The goal of this article is to find Large Language Models (LLM) that can perform task in Norwegian language successfully. Additional goal is to compare 3 LLM models made in Norway:

• NbAiLab/borealis-27b
• NorwAI/NorwAI-Magistral-24B-reasoning
• norallm/normistral-11b-thinking

And popular open-weight models (France, USA, China):

• moonshotai/Kimi-K2.6
• nvidia/GLM-5.2-NVFP4
• openai/gpt-oss-120b
• Qwen/Qwen3.8-27B-FP8
• MiniMaxAI/MiniMax-M3-MXFP8
• mistralai/Mistral-Medium-3.5-128B

An additional highly quantized LLM model was also tested because it can run on a moderate laptop (for example, a MacBook with 24–32GB RAM):

• unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL

Long context and fictional knowledge

Models is asked to use provided long text (about 55000 tokens) to answer questions.

Long text is raw source of article (Bokmål) about Norway from Wikipedia: https://no.wikipedia.org/wiki/Norge

Text was injected with fictional knowledge so LLM model can only rely on provided text. I inserted evenly these 5 lines across the text:

• TLUN er et universitet i Norge.
• Universiteter og høgskoler i Trondheim: NTNU, DMMH, TLUN, BI.
• TLUN IT-avdelingen har sin IT-hub (brukerstøtte).
• IT-hub ligger i Sluppenvegen 12T.
• IT-hub telefon +4778899500.

NOTE: NorwAI/NorwAI-Magistral-24B-reasoning supports only max context length 40960 tokens so I reduced wiki text to 25000 tokens.

Test 1 - hard question

Goal: Test that model can recall long context correctly, connect evenly spread data across text and understand what information is missing.

Question: Teksten nedenfor er en artikkel fra Wikipedia om Norge. Gi meg den fullstendige adressen til IT-huben (postnummer, land, by/tettsted/sted, gate, husnummer/bygningsnummer, leilighetsnummer/romnummer), og også hvor jeg kan bekrefte at adressen er riktig? Bruk kun teksten nedenfor til å svare

Expected answer:

• Postnummer: Ikke oppgitt i teksten
• Land: Norge
• By/tettsted/sted: Trondheim
• Gate: Sluppenvegen
• Bygningsnummer: 12T
• Leilighetsnummer: Ikke oppgitt i teksten
• Bekrefte adressen på telefon +4778899500 or Model will use examples from the text to confirm.

Fail conditions:

• I count Fail if one of elements is missing
• I count Fail if additional misleading or hallucinated data added

Test 2 - easy questions

Goal: Test that model can recall long context correctly and make indirect assumptions

Question: Teksten nedenfor er en artikkel fra Wikipedia om Norge. Jeg er student i Trondheim og har tekniske problemer. Hva skal jeg gjøre? Bruk kun teksten nedenfor til å svare på spørsmålet mitt.

Expected answer:

• besøk Sluppenvegen 12T (+ 1 point)
• ring +47 78 89 95 00 (+ 1 point)
• Bonus point if model mentions that it is only for TLUN students and there is no information about other schools. (+ 1 point)

Fail conditions:

• If additional misleading or hallucinated data added. (0 point)

Results - Test 1 - hard questions

Test repeated 10 times for each LLM model

Results - Test 2 - easy questions

Test repeated 10 times for each LLM model

How to reproduce

Github repository: https://github.com/pavlokhmel/llm_benchmark_wonderland

Raw source text of the article about Norway with preserved wiki syntax is saved in the text file. Test question is inserted at the beginning of the file.

Text file example Norge_wiki.txt:

Teksten nedenfor er en artikkel fra Wikipedia om Norge. Gi meg den fullstendige adressen til IT-huben (postnummer, land, by/tettsted/sted, gate, husnummer/bygningsnummer, leilighetsnummer/romnummer), og også hvor jeg kan bekrefte at adressen er riktig? Bruk kun teksten nedenfor til å svare:
<WIKI>
. . .
FULL TEXT COPY FROM https://no.wikipedia.org/w/index.php?title=Norge&action=edit
. . .
</WIKI>

Script:

#!/bin/bash

api_url=$1
model=$2
tmpfile=$(mktemp)
trap 'rm -f "$tmpfile"' EXIT
jq -n --rawfile content "Norge_wiki.txt" --arg model "$model" '{model: $model, messages: [{role: "user", content: $content}]}' > "$tmpfile"

curl -m 1200 -sS "${api_url}" \
  -H "Authorization: Bearer ${LLM_API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary @"$tmpfile" | jq -r '.choices[0].message.content'

Repeat 10 times:

export LLM_API_KEY=sk-..real..or..placeholder..key...
for i in {0..9}; do echo -n "TEST: ${i} "; bash llm_benchmark_norsk_it_hjelp.sh http://127.0.0.1:8888/v1/chat/completions unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL; done

Output example:

TEST: 0 Basert på teksten er dette informasjonen om IT-huben:

**Adresse:**
* **Gate:** Sluppenvegen
* **Husnummer:** 12
* **Leilighetsnummer/romnummer:** T
* **By/tettsted:** Trondheim
* **Land:** Norge
* **Postnummer:** Ikke oppgitt i teksten

**Bekreftelse:**
Adressen kan bekreftes via telefon: **+4778899500**.

TEST: 1 Teksten inneholder ikke den fullstendige adressen (postnummer, land, by/tettsted/sted, gate, husnummer/bygningsnummer, leilighetsnummer/romnummer) til IT-huben, og oppgir heller ikke hvor man kan bekrefte at adressen er riktig. Teksten oppgir kun at: "IT-hub ligger i Sluppenvegen 12T."
. . .

Conclusion

From Norwegian models only NbAiLab/borealis-27b was successful 60% of the time in easy tasks. In some cases it refuses to answer: ". . . jeg kan ikke gi veiledning til ulovlig aktivitet, inkludert hacking . . ."

In most of the cases norallm/normistral-11b-thinking did not remembered questions, and it hallucinated its own question to answer.

Models moonshotai/Kimi-K2.6 and nvidia/GLM-5.2-NVFP4 tried to follow instructions literally whith out thinking out of the box. Results were consistent and scored less points with easy tasks.

Model unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL showed 100% successful in easy task and provides consistent result. That makes it smallest and fastest model for easy tasks.

Clear winners in both tests were:

• Qwen/Qwen3.8-27B-FP8
• moonshotai/Kimi-K2.6
• nvidia/GLM-5.2-NVFP4