siliconflow/sglang_agent

siliconflow/sglang_agent

Description: OpenAI-compatible chat inference proxy sidecar for SGLang

Language: Python

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-08-06T12:52:40Z

Pushed: 2026-08-06T12:52:49Z

Default branch: main

Fork: no

Archived: no

README:

SGLang Agent

OpenAI-compatible chat inference proxy sidecar for SGLang.

This project is the sidecar paired with the sglang-chat GPU Function template. It proxies OpenAI-compatible requests to the sglang HTTP server, handling request/response transformation, streaming/non-streaming responses, cancellation, and message validation.

Components

| File | Purpose | |------|---------| | src/proxy_api.py | FastAPI sidecar (port 8001 by default, overridable via PROXY_PORT) that proxies requests to sglang (port 30000). Handles request/response transformation, streaming/non-streaming responses, cancellation, and message validation. | | tests/test_proxy_api.py | Unit tests for the proxy sidecar. |

Quick Start

Install dependencies:

pip install -r requirements.txt

Run the proxy sidecar:

python src/proxy_api.py

Run tests:

pytest

Docker

Build and run the image:

docker build -t sglang-agent .
docker run -p 8001:8001 sglang-agent

Template Integration

The sglang-chat template uses the prebuilt sglang-agent image as its sidecar instead of injecting Python code at runtime:

hub.6scloud.com/d1r7umcsfi9c73b4drdg/sglang-agent:

Update this tag when a new release is published.

Environment Variables

| Variable | Default | Description | |----------|---------|-------------| | BACKEND_BASE_URL | http://localhost:30000 | sglang backend URL | | PROXY_HOST | 0.0.0.0 | Host the proxy binds to | | PROXY_PORT | 8001 | Port the proxy listens on. Override when the template pins the proxy to a different port to isolate it from sglang internal ports. A preflight check fails fast with a clear message if the port is already taken. | | ENABLE_REASONING_MAPPING | false | Map reasoning reasoning_content | | MODEL_ID | "" | Model ID passed from the template; used for model rewriting | | MODEL_PROCESSING_MODE | rewrite_both | Model name handling (erase/rewrite/rewrite_both) | | SYSTEM_ROLE_POLICY | ignore | Misplaced system role handling (ignore/reject/fix) | | MISSING_USER_POLICY | ignore | Missing user role handling (ignore/reject/fix) | | FORCE_BACKEND_STREAM | true | Force backend streaming for cancellation support | | FORCE_STREAM_USAGE | false | Inject stream_options.include_usage=true on streaming requests | | WARMUP_REQUESTS | [] | JSON/YAML list of warmup request payloads | | MAX_PROXY_TIMEOUT | 3600 | Proxy upstream timeout in seconds |

Cancellation Behavior

sglang natively polls for client disconnect on its non-stream path (4-second interval) and arms a background=create_abort_task hook on its streaming path (2-second sleep before abort). This sidecar still wins the race for two reasons:

1. FBS=true (default): the sidecar forces stream:true on the upstream sglang request and collects SSE itself. The upstream response object is a plain fastapi.Response-like object inside sglang (not a StreamingResponse), so sglang's streaming background abort task is NOT armed. The sidecar's _race_client_disconnect is the only thing that will close the upstream connection promptly when the client cancels. 2. Even on sglang versions with the streaming-path abort task, that task sleeps 2s before issuing the abort; the sidecar race closes the connection immediately, bounding cancel latency to ~0.

Continuous Integration

A GitHub Actions workflow automatically builds and pushes the Docker image to the SiliconFlow container registry when changes are pushed or merged into main.

Required repository secrets (Settings → Secrets and variables → Actions):

| Secret | Description | |--------|-------------| | SILICONFLOW_REGISTRY_USERNAME | Username for hub.6scloud.com | | SILICONFLOW_REGISTRY_PASSWORD | Password or token for hub.6scloud.com |

Pushed images (no latest tag; tags are always the git short SHA or the pushed git tag). Each tag is a multi-arch manifest for linux/amd64 and linux/arm64:

hub.6scloud.com/d1r7umcsfi9c73b4drdg/sglang-agent:
hub.6scloud.com/d1r7umcsfi9c73b4drdg/sglang-agent:

License

MIT