LLM Gateway, Proxy, Router, and Control Plane: Where Each Responsibility Begins and Ends

The shortest way to evaluate an “LLM gateway” is not to ask whether it has the right label. Ask what it does on every request, what it controls between requests, and which authority remains somewhere else.

That change in starting point matters because some terminology is standardized and some is not. HTTP gives proxy and gateway precise intermediary meanings. “LLM router,” “AI gateway,” and “AI control plane” are product and architecture terms whose scopes must be declared. A single system can perform several of these roles, and splitting a role across systems does not remove the need for a clear owner.

Treat the labels as a map legend, not as proof that a responsibility exists.

This article defines a bounded working vocabulary, maps the responsibilities, and gives you questions to use in design and procurement reviews. It does not rank vendors or propose a universal market taxonomy.

Start with the one standardized boundary: the HTTP intermediary

RFC 9110 section 3.7 describes three common HTTP intermediary forms: proxy, gateway, and tunnel. In that specification:

• a proxy is a client-chosen message-forwarding agent that receives requests for absolute URIs and attempts to satisfy them through the HTTP interface; and
• a gateway, also called a reverse proxy in the RFC, acts as an origin server on its outward-facing connection, then translates and forwards the request to one or more inbound servers.

Those definitions tell you how the component participates in HTTP. They do not automatically assign caller identity, provider credentials, data classification, model eligibility, budget policy, or audit retention. A proxy may be configured to perform some of that work, but the word proxy alone does not establish it.

The same distinction applies to observability. RFC 9209 defines a Proxy-Status response field through which an intermediary can report handling information. That is a useful protocol mechanism; it is not, by itself, an audit system, a retention policy, or proof that sensitive error bodies are safe to persist.

Working definitions for the AI-specific roles

For the rest of this article, the following are editorial definitions. They are intentionally narrow so responsibilities can be tested.

LLM router. Selects one target from an already allowed set of model endpoints. Its inputs may include requested capabilities, health, latency, cost, region, or a quality signal. The selection mechanism does not become the policy authority merely because it chooses the target.

AI gateway. Owns the request-facing boundary for AI protocol traffic. In this framework it can authenticate the caller, validate and normalize the request, obtain a policy decision, invoke a router, adapt to an upstream provider, and return a bounded response. Which of those functions a real product includes is product-specific.

AI control plane. Owns management-path state: provider and endpoint records, credentials and scopes, logical profiles, policy versions, routing inputs, budgets, retention controls, and reporting configuration. It publishes state consumed by request-path components. It may be deployed with the gateway, but it should still be possible to name the management authority separately from per-request execution.

Workspace. A human-facing chat or task interface. It can call a gateway, but a gateway or control plane does not imply that an employee workspace exists.

These definitions match no single vendor contract. That is deliberate. For example, current Cloudflare AI Gateway dynamic-routing documentation describes named, versioned request flows with conditions, limits, model calls, fallbacks, and deployment/rollback. Current Amazon Bedrock prompt-routing documentation uses router for choosing between specified models according to its routing criteria. The scopes differ. The correct conclusion is not that one label is right; it is that a reviewer must inspect the actual boundary behind the label.

Responsibility and boundary matrix

The matrix below is an evaluation framework, not a standards-mandated product classification.

• Protocol core means the role follows directly from the RFC 9110 intermediary definition.
• Framework core means the responsibility is part of this article’s explicit working definition.
• Common bundle means current products may package the responsibility with that role, but the label does not guarantee it.
• Optional means it can be implemented there if authority and failure behavior are explicit.
• Not implied means the label is insufficient evidence.
• Management path means the responsibility should be owned outside per-request target execution, even if the code is co-deployed.

Responsibility	HTTP forwarding proxy	LLM router	AI gateway	AI control plane
Receive and forward an HTTP request	Protocol core for the client-chosen proxy role	Optional; may be a library behind another service	Framework core at the request boundary	Not implied; usually outside the request path
Translate between protocols or provider request/response shapes	Permitted by the HTTP intermediary definition; implementation-specific	Not implied	Common bundle; framework core when a stable gateway contract is promised	Defines or publishes adapter configuration, if owned here
Select one model endpoint	Not implied	Framework core	Common bundle, often by invoking or embedding a router	Defines the eligible endpoints and selection inputs; should not silently execute a different policy
Authenticate the user or application	Optional	Not implied	Framework core in this article	Owns identity integrations, credentials, scopes, and lifecycle if assigned here
Authorize data, trust, region, capability, or tenant constraints	Not implied by network position	Must consume the allowed set; is not inherently the authority	Framework core to obtain and enforce a policy decision before selection	Management path for policy definition, versioning, review, and publication
Call the selected provider and handle the response stream	Optional forwarding behavior	Optional when embedded in the request executor	Framework core	Usually outside the management path
Define fallback candidates and stop conditions	Not implied	Selects only inside the supplied bounds	Enforces the request’s approved bounds during retry or fallback	Management path for publishing the bounds and their versions
Store provider credentials	Optional, with a separate security design	Not implied	Common bundle for runtime use; storage authority must be explicit	Common management concern: creation, scope, rotation, revocation, and distribution
Manage provider catalog, logical profiles, and policy versions	Not implied	Not implied	May expose management APIs, but the gateway label is insufficient	Framework core
Collect health, quota, or budget inputs	Optional	Consumes relevant selection inputs	Common bundle for per-request enforcement	Framework core for definitions, aggregation, and publication cadence
Record request outcome, usage provenance, and bounded error metadata	Optional	Optional selection evidence	Common bundle at the request boundary	Defines schema, access, retention, reporting, and deletion controls
Provide an employee chat workspace	Not implied	Not implied	Not implied	Not implied; separate product surface

Two boundaries in this table deserve special attention.

First, policy must produce the allowed candidate set before optimization ranks it. A router can be excellent at choosing a cheap or fast target and still be the wrong place to decide whether that target is permitted. Second, configuration ownership is not the same as request execution. A component that reads a policy snapshot is not necessarily the component authorized to create or publish it.

Architecture: separate the request path from the management path

The request path performs work whose latency or failure directly affects the caller. It authenticates request context, obtains a policy result, selects inside that result, translates the provider call, and returns a response or a defined error.

The management path changes what future requests are allowed to do. It creates and versions provider records, credentials, profiles, policy, routing inputs, budgets, and retention rules. A change should have an owner, an effective version, a publication event, and a rollback or replacement path.

The evidence path reports what happened without assuming that every request or response body should be retained. Outcome, selected logical route, usage provenance, latency boundaries, configuration version, and normalized error category can be useful. The schema, access, retention, and deletion behavior remain separate decisions.

The boxes may run in one process or many services. The diagram is about authority and failure propagation, not deployment count.

Why identity cannot be inferred from a hostname

It is tempting to treat a “private” URL, a provider key, a subnet, or a model alias as authorization. That collapses addressing into authority.

NIST SP 800-207A states the zero-trust concern narrowly: implicit trust should not be granted to users, services, or devices based only on network location, affiliation, or ownership. The publication discusses identity-aware policy enforcement using components including API gateways, sidecar proxies, and application identity infrastructure. Applied here, the lesson is scoped: network placement can be a policy input, but it should not be the only evidence that a caller may use a model endpoint.

This does not make an AI gateway “zero-trust compliant,” and this article makes no certification claim. It gives a review question: Which authenticated subject, resource, action, and policy version produced this allowed candidate set? If the answer is only “the request reached this hostname,” the authority boundary is incomplete.

What breaks when roles are blurred

1. The router becomes an accidental policy engine

A team gives the router every reachable provider and tells it to optimize cost. Later, someone assumes the router also knows which data can leave a private environment. No explicit component produces an allowed set, so a ranking decision silently becomes an authorization decision.

The fix is structural: policy evaluates trusted context first; routing ranks only the returned candidates. When the candidate set is empty, the system follows an explicit denial or unavailability contract rather than broadening the set.

2. A proxy credential becomes caller identity

A shared upstream key proves that the intermediary can call a provider. It does not identify the application, user, organization, or workload that caused the call. If every request arrives under one shared credential, per-caller scopes, quotas, revocation, and audit attribution may be impossible.

Name both credentials: the credential presented to the gateway and the provider credential used by the gateway. State who creates, stores, rotates, and revokes each one.

3. The management API and request API share unexamined authority

Co-deployment is not inherently unsafe. The failure comes from treating every code path in the process as equally authorized. Request handlers should not acquire policy-publication or credential-export authority merely because those operations share a service.

Review administrative identity, authorization, change history, version publication, and rollback separately from request authentication.

4. Observability quietly becomes content retention

“The gateway logs requests” is too vague for a technical or privacy review. It could mean status and timing metadata, full prompt and response bodies, raw provider errors, or all of the above.

Define the event fields, provenance, access, retention, deletion, and content exclusions. Do not use the availability of a protocol field such as Proxy-Status as evidence that an entire logging design is safe or sufficient.

5. A workspace is bundled into the infrastructure definition

An employee interface can be valuable, but it changes the identity, history, browser-storage, sharing, and support boundaries. A gateway does not provide those controls merely because a chat UI sends traffic through it. Evaluate the workspace as a separate caller and product surface.

A boundary-first build or buy review

Ask for a diagram and responsibility table that answer these questions with nouns, owners, inputs, outputs, and failure behavior—not feature names alone.

1. Protocol boundary: Which public request and response contract does the caller use? Which transformations are guaranteed, optional, or unsupported?
2. Caller identity: What authenticates a human, application, or service account? What is trusted request context, and who is allowed to set it?
3. Policy authority: Which component evaluates data, tenant, region, capability, and trust constraints? Which configuration version did it use?
4. Candidate-set boundary: Does policy return the candidates that routing may consider, or can the router introduce new candidates?
5. Selection contract: What does the router optimize, with which inputs and freshness? What happens when inputs are missing or contradictory?
6. Fallback contract: Can retry or fallback occur only inside the original allowed set? What changes before the first response event and after output begins?
7. Credential ownership: Where are caller and provider credentials accepted, stored, scoped, rotated, revoked, and prevented from entering logs?
8. Configuration lifecycle: Who can draft, review, publish, roll back, and audit provider, profile, policy, budget, and retention changes?
9. Evidence boundary: Which fields are recorded, where does each value come from, how long is it retained, and what is intentionally absent?
10. Operations ownership: Who responds when identity, policy, router, provider adapter, or management state is unavailable?
11. Workspace boundary: If a human UI exists, which history, sharing, browser, and deletion responsibilities are additional to the gateway?
12. Proof: Which behaviors are documented, which are tested, and which are only planned? A hostname, diagram, or product tier is not runtime evidence.

An answer may legitimately place several responsibilities in one component. The test is whether the boundaries remain explicit enough to reason about authority, change, and failure.

The practical dividing line

Use this compact rule during architecture reviews:

• A proxy explains how a message is forwarded in HTTP.
• An LLM router explains how one allowed target is selected.
• An AI gateway explains the request-facing AI protocol and enforcement boundary.
• An AI control plane explains who defines, versions, and publishes the state used by that boundary.

None of those labels proves the presence of the next responsibility. If a product performs all four roles, it should be able to show all four boundaries.

Primary-source packet

Sources were reviewed on 2026-07-17. Vendor sources are used to establish the scope of their own documented terms, not to rank products or generalize a market standard.

1. RFC 9110, HTTP Semantics, section 3.7: Intermediaries — normative definitions for HTTP proxy, gateway, and tunnel roles.
2. RFC 9209, The Proxy-Status HTTP Response Header Field — standardized intermediary status reporting, cited only for that protocol mechanism.
3. NIST SP 800-207A, A Zero Trust Architecture Model for Access Control in Cloud-Native Applications in Multi-Cloud Environments — identity-aware policy and the limits of network-location-based trust; no product certification implication.
4. Cloudflare AI Gateway: Dynamic routing — current first-party example of gateway-labeled routing, limits, versions, deployment, and fallback configuration.
5. Amazon Bedrock: Intelligent prompt routing — current first-party example of router-labeled target selection under defined criteria.
6. Kubernetes components — a current primary example in which a control plane manages overall system state; used as an analogy, not as an AI definition.

Scoped claim ledger and closure

Claim used in this article	Evidence and observation	Closure
HTTP defines proxy and gateway intermediary roles	RFC 9110 §3.7; reviewed 2026-07-17	Closed — standard. Paraphrased within its HTTP scope; no AI duty inferred.
An intermediary can report handling information with Proxy-Status	RFC 9209; reviewed 2026-07-17	Closed — standard. Limited to the response-field mechanism; not presented as a complete audit design.
“LLM router” can denote target selection under explicit criteria	AWS Bedrock prompt-routing documentation; reviewed 2026-07-17	Closed — vendor-scoped example. Used to support the local definition, not a universal contract.
“AI gateway” can bundle routing, conditions, limits, versions, and fallbacks	Cloudflare AI Gateway dynamic-routing documentation; reviewed 2026-07-17	Closed — vendor-scoped example. Does not imply every AI gateway owns those functions.
AI-market labels require local responsibility definitions	Direct comparison of the two current first-party examples above	Closed — bounded editorial inference. The article explicitly avoids claiming a universal taxonomy or absence of all standards.
Network location alone is insufficient as an identity/authorization basis in zero-trust architecture	NIST SP 800-207A abstract and scope; reviewed 2026-07-17	Closed — scoped architecture guidance. No certification, compliance, or security-outcome claim.
A control plane can be modeled as the owner of system state distinct from workload execution	Kubernetes current component documentation plus this article’s declared definition; reviewed 2026-07-17	Closed — analogy and editorial definition. Not presented as a standardized AI control-plane meaning.
The matrix assigns typical responsibility boundaries	Original Tensor Cortex Editorial framework created 2026-07-16	Closed — original artifact. Legend distinguishes standard, framework core, common bundle, optional, and not implied.
Tensor Cortex publicly names Gateway as the request-facing API surface and Control as the policy/management surface	TensorCortex.com product overview and shared public product copy inspected 2026-07-17	Closed — narrow product naming claim. Availability, readiness, security, compatibility breadth, and performance claims are excluded.

No customer, benchmark, incident, certification, partnership, pricing, market-share, or first-hand deployment claim is made.