Two models score similarly in a public ranking, but one fails on the organisation’s documents and the other exceeds its compute budget. The ranking is useful evidence about its own tests, which may not represent the application being built.
Selection examines downloadable files and usage rights, application-test results, inference resources and operating cost. Kimi, Soofi, Apertus and Mistral illustrate why those questions must remain distinct.
Check availability and the extent of openness
Weights are learned values. Publishing them may allow downloading and execution under the licence. It does not automatically make training data and every preparation stage public.
The team examines a precise version. A base model, a dialogue-tuned release and a quantised variant may differ in behaviour and terms. Announced results or a restricted preview offer different possibilities from an available release for the intended use.
Documentation should identify files and versions, including compatible runtimes. A model-family name is insufficient to describe what the application will run.
Compare candidates against distinct criteria
The examples below illustrate design and release choices. They are not a permanent ranking; the selected release’s model card remains the reference.
| Example | Item to examine | Project consequence |
|---|---|---|
| Kimi K3 | Large architecture and licence terms | Establish the required infrastructure or hosted service |
| Soofi S Base | Base-model status and access conditions | Distinguish experimentation from a ready-to-use assistant |
| Apertus | Published weights, code and training documentation | Examine provenance separately from task quality |
| Devstral Small 2 | Coding-oriented model | Test repository context, tools and runtime |
The Kimi K3 repository describes 2.8 trillion total parameters. Activated parameters per computation do not represent all weights that must be hosted. Mixture-of-experts designs require distinguishing compute from model storage.
The Soofi S Base card describes a base version. Dialogue adaptation and specialised use require their own evaluation rather than an assumption that a conversational application reproduces base-model results.
Apertus documentation supports an examination of openness and provenance. Those properties differ from extraction accuracy or French-language answer quality.
A useful Devstral Small 2 test uses the project’s languages and libraries, with the coding tasks actually required. Evidence from another repository can inform selection without predicting this one.
Evaluate the intended tasks
The test set contains ordinary and difficult cases. Extraction checks missing fields and ambiguous input. Retrieval examines relevant passages, accurate citations and questions the system should decline for lack of evidence.
Errors need categories. An incomplete answer differs from an invented reference or an unauthorised action. Acceptance thresholds and escalation can reflect those consequences.
Comparisons record instructions and tools, including context and model settings. Those conditions can change the result. The useful score describes the complete application with its intended integrations.
Language requires its own test. Strong performance on general English questions may not extend to a business abbreviation or French legal wording. Users should examine representative answers and errors with their sources.
Measure resources and accepted-result cost
Quantisation lowers numerical precision to reduce memory use. It can change quality and compatibility. Evaluation therefore uses the variant planned for deployment, with room for context and concurrent requests.
Measurements include initial delay and behaviour under load. A fast single request can lose that advantage with several users waiting. The application needs limits and queue handling, with a response to failures.
Complete cost includes compute and operations, together with continuity arrangements. API comparisons count input and output plus retries and additional features. Cost per token does not always express the cost of a correctly completed dossier.
Local execution may support control requirements or sustained use. An API may suit variable demand or a model difficult to host. The outcome depends on quality and availability assumptions rather than a universal break-even threshold.
Preserve a reproducible decision
The organisation keeps the model version and licence, along with the configuration and evaluation set. Hosted use also records the operator and processing terms. Open weights do not establish control over data sent to a provider.
Updates go through the same checks. The team compares results and examines regressions, preserving a return path where possible. The chosen model then serves a defined use in an environment the organisation can maintain.
Rankings help discover candidates. Tests and operating conditions support the actual decision.
Continue with internal AI deployment and local hardware profiles.
The model examples were reviewed for the 7 October 2026 revision. Check availability and licence terms for the exact version before deployment.