A Decision Framework for Picking the Right LLM in 2026

As of April 2026, OpenRouter lists 353 language models from 56 providers. A new flagship lands every couple of weeks. Public benchmarks become obsolete before they stabilise. The practical question for an engineer is no longer "which model is best" — it's "which one is right for this specific thing I'm building, today." Most of us answer that badly. This paper is an attempt to answer it better, paired with an open tool at llm.subralabs.com that walks through the framework interactively.

How we actually pick LLMs (and why it's wrong)

From watching other developers — and catching myself — there are three common failure modes:

• Inertia. "I've always used GPT-4." The model you integrated 18 months ago is no longer the best fit; it's just the one you know. The gap between "familiar" and "optimal" compounds quietly.
• Hype-chasing. Every launch week, the timeline declares a new king. A week later it has been quietly dethroned. Picking based on leaderboards burns integration cost on models that may not matter by Q3.
• Single-axis thinking. "I need the smartest one" or "the cheapest one." Smartest at what? Cheapest for which volume profile? Both collapse a multi-dimensional decision into one number.

The alternative is to be explicit about which dimensions matter for this specific integration and pick the model that dominates on those. That list, for almost any real-world use case, has about seven entries.

The seven dimensions

These aren't exotic — most of them are obvious individually. The value is in asking all of them before committing, in the order below.

1. Task type

Coding, reasoning, vision, tool-use, structured output, creative writing, long-context summarisation, multilingual, agentic planning, audio. A model that tops HumanEval (coding) can score mid-pack on GPQA (reasoning). Don't treat "intelligence" as uniform.

2. Budget

Not "is it cheap" but "what's my cost at expected volume." A flagship at $15/1M output tokens is irrelevant if you expect 10 million tokens per user per month. A free tier is irrelevant if rate limits will throttle you at launch. Model the actual spend.

3. Context window

200K, 1M, 4M — but also: what's the usable context? Effective recall drops sharply past a certain point for most models (the so-called "lost in the middle" problem). If your workload really needs long context, test retrieval in your actual documents, not on synthetic haystacks.

4. Capabilities

Three hard gates: does it support tool calling (and how robustly); does it support structured output (JSON schema or response_format); does it accept non-text inputs (images, audio). Missing any one of these is a hard no — not a minor inconvenience.

5. Privacy & deployment

On-device (Apple Foundation Models, Gemini Nano, Phi, llama.cpp) vs. cloud. On-device wins on privacy, cost, and offline availability; cloud wins on quality, multilingual strength, and tool-calling reliability. The framework here is that if privacy or offline operation is a requirement (healthcare, finance, mobile), on-device is a gate, not a preference. Otherwise cloud usually wins on quality for the same cost ceiling.

6. Ecosystem & lock-in

Native SDK maturity, thread/assistant APIs, structured caching, prompt caching, batch discounts, compliance docs. An inferior model with a mature SDK often ships faster than the "best" one behind a half-finished client. Lock-in has a cost; account for it.

7. Recency

Knowledge cutoff matters for anything involving current events, framework versions, package APIs. A 2023-cutoff model giving confident wrong answers about React 19 is a production incident waiting. Check the knowledge_cutoff field explicitly; don't assume recent = fresh.

The tool: llm.subralabs.com

The framework above is just a list. To make it actually usable, there's an accompanying tool: llm.subralabs.com. It turns the seven dimensions into concrete filters, ranks the catalog live, and — optionally — lets an AI model distil a natural-language brief into those same filters.

How it's built (and why that matters for reproducibility)

• Catalog sourced from OpenRouter's public models API at build time — 353 models with pricing, context length, modality, and supported parameters. Zero manual curation: when a new flagship lands, it appears at the next rebuild.
• Three on-device entries (Apple Foundation Models, Gemini Nano, Phi-4) are added manually since they don't appear in OpenRouter's API.
• Ranking is a pure TypeScript function: hard gates on capability requirements, scoring on use-case match, tier, recency. Client-side, zero backend.
• The "quality proxy" is an open heuristic (tier + capability flags + recency bonus) — explicitly not a benchmark score. Benchmark overlays are future work and will be opt-in.

Ranking with explanations

Every result in the shortlist carries the specific reasons it was picked ("matches coding", "matches tool-use", "cheap $0.13/1M avg"). This is deliberate: a ranker that only gives an ordered list is a black box. A ranker that explains each pick lets you sanity-check the heuristic and catch cases where your mental model and the tool's model diverge.

The AI assistant: self-referential tool calling

There's a second way to drive the tool: describe your use case in plain English and let an LLM fill in the filters for you. It's a small joke with a serious point — using an LLM to help you pick an LLM.

Under the hood it's a strict agentic pattern:

1. The client sends the user's message to a small proxy server (to protect the OpenRouter key).
2. The proxy calls OpenRouter with a system prompt that describes the seven-dimension vocabulary and a single function tool, rankModels(filters).
3. The model reads the user intent, decides which dimensions are in play, and emits a tool_call with concrete filter values.
4. The proxy forwards the parsed tool call back to the client as an SSE event. The client applies the filters locally against its copy of the catalog — the server never sees the catalog.
5. The model writes one short sentence summarising the trade-off. No enumeration of model names (the UI already shows them).

This keeps token cost flat regardless of catalog size, avoids leaking the catalog to the LLM, and makes the behaviour inspectable: the tool call arguments are visible in the chat bubble, so you can see exactly what the AI decided on your behalf.

What this framework deliberately doesn't do

A few things are left out on purpose:

• Benchmark scores. MMLU, HumanEval, GPQA, AIME, SWE-bench. These matter but they age fast, they're gameable, and adding them to a tool creates the illusion of precision where there isn't any. I'd rather surface raw capabilities and pricing and let the engineer read the benchmark reports separately.
• Personal recommendations. There's no "best model" column. The ranking is user-intent-dependent by design — different filters produce different winners.
• Quality benchmarks on-device. The three on-device entries are marked but not scored against cloud models. They operate in a different regime (privacy, offline) and comparing them head-to-head on output quality isn't the right frame.

What I'd like you to take away

Two things.

First: if you're picking an LLM for a new integration in 2026, spend ten minutes running through the seven dimensions explicitly before committing. The gap between "I know what I need" and "I've written it down" is the gap between a principled choice and a reflex one.

Second: the tool is live and free to use — try it at llm.subralabs.com. If the heuristics feel wrong for your use case, that's the point: there's no single right way to rank 353 models, only being explicit about the way you're doing it. Adjust the dimensions to match how you actually decide.