Opening

One morning, you paste the same background into a chatbox for the third time that week: what you’re building, what you tried, what “good” looks like. The model replies instantly, but the real cost isn’t tokens—it’s your attention, spent rebuilding context that the system does not reliably keep. You close the tab with a quiet sense that AI is powerful, yet oddly hard to use.

That feeling isn’t a personal failure. It’s a product design problem. We wrapped a new kind of capability in the oldest interface we had: a conversation window, one prompt at a time.

What happens when “conversation” stops being the product and becomes just the front door?

Part I: Chat Was the Interface, Prompts Became the Method

In the beginning, LLMs looked like a simple machine: you provide input, you receive output. A chat interface made that feel natural, because conversation is how humans already negotiate intent. We didn’t adopt chat because it was the final form; we adopted it because it was the fastest bridge from research to daily use.

Then prompts arrived as the steering wheel. People discovered that a few lines of instruction could change tone, structure, and even reasoning style. Very quickly, “how you ask” became a skill—one that determined whether a model sounded brilliant or useless.

The market responded the way it always does: it packaged the skill. Templates turned into workflows, workflows turned into products, and products turned into SaaS that promised repeatable outcomes. We stopped “talking to a model” and started buying “a writing assistant,” “a coding copilot,” or “a meeting note system,” all powered by the same underlying engine.

This is the first quiet shift: LLM capability doesn’t scale only as raw intelligence; it scales as packaging. And packaging pushes you toward standard inputs and standard outputs.

Part II: Context Gets Spent, Not Stored

Here’s the downside we ran into almost immediately: every submission needs a full, explicit context. You must describe what happened before, what constraints matter, and what the next step should be. When the task is simple, this feels fine; when the task is real, it becomes a tax.

The paradox is that better products often make the tax worse. As business logic gets layered into the app—permissions, tools, compliance checks, role definitions—the prompt space reserved for your real intent shrinks. You end up negotiating with your own software: compressing context, pruning nuance, and hoping the system infers what you omitted.

So vendors race to expand context windows. But even when we can fit more, we can’t assume models use it reliably. Liu and colleagues showed in TACL 2023 that long-context performance can degrade when the relevant information sits in the middle rather than the edges, a pattern that makes “just add more context” a fragile strategy.

For the rest of this essay, we take a practical stance: assume context size is not infinitely elastic. If that’s true, the next evolution must happen above the model—at the software and ecosystem layers.

Part III: From Intent-Driven Tools to Self-Running Services

Today’s AI apps still depend on explicit intent. You open a tool for a defined need—manage tasks, write a chapter, generate code, order food—and you drive it with clear instructions. The user is the controller, and the model is the engine.

But users are already reaching for a different bargain. We want systems that learn our preferences through interaction, reduce the amount of explaining we do, and eventually run with minimal intervention. In other words: less “ask and receive,” more “delegate and verify.”

Once you step into delegation, the model must do more than generate text—it must decide, act, and recover. Research on tool use is a clear signal here: Yao and colleagues’ ReAct framework at ICLR 2023 showed that combining reasoning traces with explicit actions can improve task solving across interactive settings. Schick and colleagues argued in an arXiv 2023 paper that models can be trained to decide when to call external tools rather than hallucinate.

The twist is that agentic behavior turns product design into governance design. If a system can act, it must also be constrained: by policy, by permissions, by audit logs, and by accountability when things go wrong.

Part IV: Complexity Hits, Then Selection Becomes the Real Bottleneck

Under the hood, most AI applications today follow the same recipe: LLM + business control + internal tools + external APIs. It works, and it ships. Yet each new tool adds complexity, and each new integration adds failure modes.

There’s also a structural limit: no company can be best at everything. A single team can’t deeply own shopping, writing, legal advice, customer support, payments, and every future niche that appears. As soon as you accept specialization, you also accept an ecosystem.

That leads to a deceptively simple question: when multiple providers can satisfy the same need, who do we pick?

Quality varies. Prices move. Latency spikes. Compliance differs by region. Users want refunds, explanations, and recourse when outputs cause real harm. The current world mostly handles this with brand trust and vague terms of service.

In a world of autonomous execution, that won’t be enough. We need a way to discover services, compare them, route requests intelligently, and attach responsibility to outcomes.

Part V: A Layered Paradigm for AI Software

The path forward is not one mega-app. It’s a layered ecosystem that separates interaction, governance, specialization, and compute—so each layer can evolve without breaking the others.

The next leap in AI software is not a larger context window, but a coordination layer that can discover, compare, route, and hold services accountable.

Start with the client layer: the operating system, the app, the interface you actually live in. It should be plural, because humans are plural. Some people want a chat window, others want voice, others want a calendar-first workflow, others want a command palette embedded in their IDE.

Then introduce a service hub: a middle layer that manages providers and exposes a unified interface to the client. Think of it less as “a platform” and more as the place where selection becomes systematic. It registers services, measures performance, enforces privacy boundaries, routes requests based on policy, and handles billing and disputes.

Next comes the provider layer: the specialists. A shopping provider can optimize catalogs and fulfillment. A writing provider can tune editorial workflows. A support provider can handle escalations and compliance. Each provider can innovate in its domain without needing to rebuild the entire world.

Finally, the model layer sits underneath, as replaceable infrastructure. Providers choose which models to call, how to prompt them, and how to combine them with tools. The client and hub should not be hostage to a single model family, because the “best model” changes faster than product cycles.

In this architecture, the user stops being a full-time prompt engineer. The client expresses intent. The hub selects and routes. The provider executes. And the model layer supplies the raw capability.

Counter-Argument: Won’t the Service Hub Become the Next Monopoly?

The strongest objection is obvious: whoever controls the hub controls discovery, rankings, and the rules of competition. That can become a new gatekeeper, and gatekeepers tend to extract rents.

So the question is not whether the hub is risky—it is. The question is whether we can design it to be constrained. That means portability (you can move identity, preferences, and history), transparency (ranking and routing are inspectable), competition (multiple hubs can exist), and separation (the hub measures and routes, but doesn’t secretly privilege its own providers).

If we refuse to build a governance layer, we don’t avoid power. We merely push power into hidden heuristics inside apps and models, where users have even less visibility. The honest move is to surface selection as a first-class system problem, then architect it so it can be audited and contested.

Closing

Chat was the fastest on-ramp for LLMs, and it will remain a useful doorway. But as soon as we expect systems to remember, to act, and to coordinate across domains, the chatbox becomes too small for the job.

The future feels less like “one assistant” and more like “a service ecosystem you can trust.” Not because trust is a slogan, but because trust is enforced: measured quality, clear costs, privacy boundaries, and responsibility when something breaks.

We’ve spent the first era teaching ourselves how to talk to models. The next era is teaching software how to deliver outcomes—without making users rebuild context for the thousandth time.

Visual ideas

• A four-layer diagram (Client → Service Hub → Providers → Model Layer) with example flows for one user request.
• A “selection funnel” graphic showing how routing decisions balance quality, cost, latency, and compliance.