Graded by code. No judge to argue with.
Every case here is checked by a program, not by another AI. There is no rubric to lawyer and no judge to bias, so the result is the same whoever runs it — including you.
No published run yet. Results appear here as soon as a run is recorded.
What it measures
Instruction fidelity in Urdu: staying in the script that was asked for, obeying exact counts, not leaking Devanagari or English into an Urdu answer, Pakistan-specific facts, translation both ways, arithmetic through Urdu, and safe handling of an unsafe request asked in Roman Urdu.
It does not measure general intelligence, coding or reasoning depth — the public benchmarks the large labs already lead cover those. Every model here is far stronger in English than in Urdu. That gap is the thing being measured.
How it is kept fair
- Identical prompts for every model, no per-model tuning.
- Temperature 0 where the provider supports it, so a re-run reproduces.
- A wrong answer is never retried. Network errors are, so the result measures the model and not the weather.
- Competitors are called through their public APIs with no system prompt. RafayGen is called through its own production endpoint, as the shipped product — an advantage worth naming rather than hiding.
Raw data, including every model's full answer to every case, is at /api/benchmark/urdu?full=1. The runner is apps/engine/urdubench.py. Built by Abdul Rafay Amir.