On-Device vs Cloud LLMs for Agentic Tool Calling

We built an AI concierge into a resort directory app for iOS. The feature needed to search a curated dataset of luxury properties, apply filters, find nearby airports, and respond conversationally in Italian. We tested two approaches: Apple's on-device Foundation Models (~3B parameters) and GPT-OSS 20B via OpenRouter. Here is what we found.

The problem

The concierge feature required more than simple question-answering. It needed agentic behaviour: the model receives a user query, decides which tools to call (search, filter, nearby-airport lookup), processes the results, and then composes a coherent, persona-consistent response in Italian. This is a three-step loop — reason, act, synthesise — that repeats until the query is fully resolved.

Three tools were available to the model:

• search — free-text search across names, locations, tags
• filter — structured filtering (area, cost, scores, amenities)
• nearbyAirport — haversine distance from five airports

The challenge is not calling a single tool. It is orchestrating multiple calls in sequence, interpreting partial results, and weaving them into a natural response that feels like a knowledgeable travel advisor — not a database query interface.

Two approaches

On-device: Apple Foundation Models

Apple's Foundation Models framework, introduced with iOS 26, runs a ~3B parameter model directly on the device. No network required. The model supports structured generation and tool calling through Swift-native APIs with @Generable macros.

Advantages on paper: zero latency to first token, complete privacy (data never leaves the device), no API costs, works offline.

Cloud: GPT-OSS 20B via OpenRouter

OpenAI's GPT-OSS 20B, accessed through OpenRouter's API, using server-sent events (SSE) for streaming responses. Tool calls are parsed from the stream and executed locally against the same dataset, with results fed back for the next generation cycle.

Advantages on paper: much larger model, proven multilingual capabilities, mature tool-calling support.

Finding #1: tool calling complexity is the bottleneck

The on-device model can perform tool calling. It correctly identifies which tool to invoke and generates valid parameters. In isolation, this works.

The problem emerges with compound tasks. When the user asks something like "show me luxury resorts near Jeddah with good spa scores", the model needs to:

1. Call nearbyAirport for Jeddah
2. Interpret the results
3. Filter or rank by spa score
4. Compose a coherent response presenting the matches

The 3B model consistently struggled with step 4. The failures were concrete and reproducible:

• The tool returns 2 matching resorts. The model responds: "Here are 3 properties that match your search…" and lists the correct 2.
• The tool returns 2 results. The model says: "There is one resort that matches your criteria" — then correctly presents both.
• The model opens with "I couldn't find any resorts matching that" — then proceeds to show valid, correct results.

The tool calls themselves were correct. The data was correct. But the model could not reconcile the tool output with its own natural language generation. It would count wrong, contradict itself, or negate its own findings — all within the same response.

Conversation context was another failure point. Despite the framework supporting multi-turn conversations, the on-device model would lose track of prior questions. A follow-up like "and which of those have a pool?" after a search would trigger a fresh, unrelated search rather than filtering the previous results.

Multiple prompt engineering approaches were tested:

• Chain-of-thought prompting with explicit reasoning steps
• Separating the "decide" and "respond" phases into distinct prompts
• Providing few-shot examples of complete tool-call-to-response flows
• Reducing the system prompt to minimal instructions

None reliably solved the issue. The model has the mechanical ability to call tools, but lacks the capacity to maintain coherent state across the reason-act-synthesise loop. This is not a prompt engineering problem — it is a parameter count problem.

GPT-OSS 20B handled these compound tasks without difficulty. Tool results were correctly interpreted, counted, contextualised, and presented in natural Italian with the expected concierge persona. Conversation context was maintained across turns.

Finding #2: response quality and language

Beyond tool calling, the general response quality diverged significantly.

The cloud model maintained a consistent "knowledgeable travel advisor" persona across conversations. It understood implicit intent ("somewhere relaxing" maps to spa scores and environment scores), used appropriate Italian register, and avoided the hollow filler phrases that plague many AI assistants.

The on-device model produced grammatically correct Italian but lacked nuance. Responses were functional rather than helpful — more database printout than travel recommendation. For a feature meant to feel like asking a well-connected friend, this gap matters.

Finding #3: cost is not the barrier you think

The assumption going in was that cloud inference would be expensive enough to require careful usage management. The reality:

Metric	Value
Cost per 100K tokens (mixed I/O)	$0.005 – $0.007
Average conversation (5 turns)	~8K – 12K tokens
Cost per conversation	< $0.001
1,000 conversations/month	~$0.50 – $0.70

At this price point, the cloud model costs less than a single App Store subscription tier per month for substantial usage. The cost is low enough to absorb entirely, fold into a freemium model, or offset with minimal monetisation. It removes cost as a meaningful variable in the on-device vs cloud decision.

Trade-off matrix

Dimension	On-device (~3B)	Cloud (20B)
Simple tool calling	Works	Works
Compound agentic tasks	Unreliable	Reliable
Response coherence	Functional	Natural
Non-English quality	Acceptable	Strong
Privacy	Complete	Requires trust in provider
Latency	Instant first token	Acceptable (streaming)
Offline capability	Full	None
Device requirement	Modern hardware + Apple Intelligence	Any device with network
Cost per conversation	$0	< $0.001

What this means in practice

For our use case — a conversational concierge that needs to search, filter, and advise — the cloud model was the only viable option for production. The on-device model could not reliably complete the core task.

This does not mean on-device models are without value. For simpler applications — single-tool invocations, text classification, summarisation, form pre-filling — the privacy and latency advantages of Foundation Models are compelling. The key insight is knowing where the capability boundary sits.

The decision framework we arrived at:

• Use on-device when the task is single-step and the model's output is directly consumable (classify, extract, summarise, single tool call with simple response)
• Use cloud when the task requires multi-step reasoning, tool orchestration, persona consistency, or strong non-English generation
• Offer both with a user-facing toggle when the audience includes users who prioritise privacy over capability, or when you want graceful offline degradation

In our app, we shipped a toggle between the two backends. The cloud model is the default. Users who want on-device inference can switch, understanding the trade-off. This turned a technical limitation into a user choice.

Looking ahead

On-device models will get larger. Apple's hardware trajectory suggests that within two generations, devices could comfortably run 7B+ parameter models. At that scale, the compound-task gap may narrow significantly.

Until then, the practical answer for agentic iOS features is: build your tool-calling infrastructure to be backend-agnostic, default to cloud for quality, and keep the on-device path ready for when the models catch up.