Building the Modern AI Infrastructure Stack with Cortex AI Gateway

Building the Modern AI Infrastructure Stack with Cortex AI Gateway

Skip to content

Blog / Product and Technology / Building the Modern AI Infrastructure Stack with Cortex AI Gateway

Oct 8, 2026 / 9 min read Product and Technology Copy post link Open in Claude Open in ChatGPT

Building the Modern AI Infrastructure Stack with Cortex AI Gateway

Ali Taha +2

Over the last few years, the enterprise AI stack has evolved into a sophisticated but fragmented and unwieldy ecosystem. It now encompasses everything including vector databases, monitoring systems and pipeline tuning, connected through a sprawling mix of coding agents, embedded agents and Model Context Protocol (MCP) servers interfacing with enterprise systems — all supported by a model portfolio that shifts every few weeks. Different teams adopt pieces of the stack on different timelines, potentially creating a management and governance gap that can impact security.

In this fractured environment, few enterprises have the full context they need to accurately answer basic questions across these surfaces: Which models are we calling, who is calling them, what actions are they taking, and what is the associated cost?

The instinct in response is to restrict, but this isn’t a viable long-term solution. Hindering employees’ ability to use AI will only lead them to resort to unsanctioned AI tools, which amplifies the risk even further.

That gap isn’t a tooling problem; it’s a missing layer. The modern AI stack needs a control point that sits between every AI client and the enterprise systems they reach. Cortex AI Gateway (public preview 3 ) addresses this by acting as that centralized control plane, providing the unified visibility, governance and oversight that this modern, multifaceted infrastructure requires. In our view, six elements are nonnegotiable in the modern enterprise’s AI infrastructure stack:

Unified, governed inference

Dynamic model routing

MCP and tool governance

Central observability

Security policies and guardrails

Cost governance and controls

Now in public preview, Cortex AI Gateway gives platform teams one place to grant access to models and MCP servers, attribute and control spend, and see what AI has done on behalf of their users. Application teams get a single endpoint to build against. Let’s take a look at how it maps onto each layer of the stack.

Cortex AI Gateway is Snowflake's centralized control plane for enterprise AI infrastructure. It provides a single endpoint for governed model inference, dynamic model routing, MCP tool governance, observability, security guardrails and cost controls — giving platform teams unified visibility across every AI client and agent in the organization.

Unified, governed inference: Use one endpoint for your agents

The first thing that breaks in a growing AI stack is the number of paths to a model: An engineer configures a coding agent against one provider. A platform team wires an application to a second. Someone else scripts against a third. Each path has its own credential, its own log and its own bill.

Cortex AI Gateway collapses those into one endpoint. It implements the Chat Completions API for OpenAI, Grok, GLM, Gemini, Llama, Mistral and DeepSeek models and the Messages API for Claude, which are the two protocols model serving has converged on. Streaming, tool calling, structured output, prompt caching, image input and reasoning all work the same way they do natively, so pointing an existing client at Cortex AI Gateway requires just another base URL and a token. Support for the OpenAI Responses API is also coming very soon. The following example shows all the setup you need to configure OpenCode with Cortex AI Gateway:

export PAT="" export BASE_URL="" export OPENCODE_CONFIG_CONTENT='{ "$schema": "https://opencode.ai/config.json", "model": "snowflake-cortex/openai-gpt-5.4", "small_model": "snowflake-cortex/openai-gpt-5.4", "provider": { "snowflake-cortex": { "options": { "baseURL": "${BASE_URL}", "account": ".snowflakecomputing.com", "apiKey": "", "headers": { "snow-agent-name": "opencode" } } } } }'

Access control uses Snowflake’s existing role-based access control (RBAC) framework. An admin can configure what models are available through the gateway, making sure only those approved by an organization are accessible.

To make it even easier to use coding agents with Cortex AI Gateway, the Snowflake CLI will be able to take care of setting up your traffic to go through the gateway. One command is all it takes to read credentials from your active Snowflake connection, inject them into the coding agent’s config and launch the agent with its traffic routed through Cortex AI Gateway. For example, launching OpenCode using the Snowflake CLI is now as simple as running snow ai opencode .

Dynamic model routing in Cortex AI Gateway: Pick the right model for each task

Most teams have settled on a single frontier model for everything, which is a reasonable way to start and an expensive way to scale. A large share of agent steps are file edits, formatting and lookups that a cheaper model handles with the same quality and accuracy.

Figure 1: Cortex AI Gateway offers dynamic model routing (private preview soon).

Dynamic model routing , coming to private preview soon through Cortex AI Gateway, selects the most affordable model (from the available options) that can confidently complete the task at each step of agent execution. It’s been fine-tuned by the Snowflake research team, so lower-complexity and repetitive work goes to efficient models, and work that needs deeper reasoning goes to frontier models. In an internal evaluation, it completed a dbt pipeline workload with up to 3x greater token efficiency than a frontier-model-only approach at comparable quality, and in a separate coding test engineering teams held pull-request throughput steady while using roughly 25% fewer tokens. 1 Those results can translate into real-world cost savings for enterprises.

Routing is governed just like manual model selection. Only approved models are used, existing data residency settings are respected, and every routing decision is logged, so a compliance team can see which model handled which request. Because routing lives in Cortex AI Gateway rather than in each application, Snowflake can update decisions as model pricing and performance change without agent configurations ever having to change.

The open source model side matters here too. Evaluated on...