microsoft/python-lsp-compare

microsoft/python-lsp-compare

Language: Python

License: MIT

Stars: 7

Forks: 2

Open issues: 3

Created: 2026-03-19T20:27:23Z

Pushed: 2026-06-10T07:29:04Z

Default branch: main

Fork: no

Archived: no

README:

python-lsp-compare

python-lsp-compare is a small benchmark and regression harness for Python Language Server Protocol implementations.

This repository provides a vendor-neutral comparison harness and test corpus for Python language servers. It is intended to surface behavioral and performance differences across implementations and does not define a specification or normative behavior.

It focuses on four things:

1. Running LSP servers over stdio with raw JSON-RPC messages. 2. Executing repeatable scenarios against those servers. 3. Capturing request/notification timings, payload sizes, and results. 4. Producing machine-readable reports that are easy to diff across servers.

Benchmark suites are package-oriented, not just API-oriented. That means testing LSP behavior against realistic dependency surfaces like SQLAlchemy-heavy code, web frameworks, and data-science imports.

Benchmark runs are intentionally deterministic: each suite creates or reuses its own .venv, installs the suite requirements there, writes temporary workspace configuration for language servers, and then runs every selected server against that same suite-local environment.

Features

• Pure Python implementation of LSP framing over stdio.
• Built-in Python scenarios for hover, completion, and document symbols.
• Config-driven benchmark suites under benchmarks/ with package-specific fixtures and requirements.txt files.
• Per-call metrics including latency, bytes sent, bytes received, success, and errors.
• Aggregate stats for benchmark points including mean, median, min, max, and p95.
• JSON report output for later aggregation.
• MIT licensed from the start.

Quick Start

Create a virtual environment, install the package in editable mode, and point it at an LSP server command.

python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -e .
python -m python_lsp_compare list-scenarios
python -m python_lsp_compare list-benchmarks
python -m python_lsp_compare list-servers
python -m python_lsp_compare run --server-command pylsp --scenario hover --scenario completion
python -m python_lsp_compare run --server-command pyright-langserver --server-arg=--stdio
python -m python_lsp_compare run-benchmark --server-command pyright-langserver --server-arg=--stdio
python -m python_lsp_compare run-servers --scenario hover --scenario completion
python -m python_lsp_compare bench-servers
python -m python_lsp_compare render-report --summary results/bench-servers/summary-20260319T000000Z.json --baseline-server pyright

The default report path is created under results/.

CLI

List the bundled scenarios:

python -m python_lsp_compare list-scenarios

List the benchmark suites:

python -m python_lsp_compare list-benchmarks
python -m python_lsp_compare list-benchmarks --protocol tsp

List the locally configured servers:

python -m python_lsp_compare list-servers
python -m python_lsp_compare list-servers --config path/to/lsp_servers.json

Run one or more scenarios:

python -m python_lsp_compare run \
--server-command pyright-langserver \
--server-arg=--stdio \
--scenario hover \
--scenario completion \
--output results/pyright.json

Run one or more package-oriented benchmark suites:

python -m python_lsp_compare run-benchmark \
--server-command pyright-langserver \
--server-arg=--stdio \
--output results/pyright-benchmarks.json

python -m python_lsp_compare run-benchmark \
--server-command pyrefly \
--server-arg=lsp \
--protocol tsp \
--output results/pyrefly-tsp-benchmarks.json

Run the same scenarios across all servers (using cached GitHub-release or PyPI installs by default):

python -m python_lsp_compare run-servers \
--scenario hover \
--scenario completion \
--output-dir results/servers

Run against servers from a local config file:

python -m python_lsp_compare run-servers \
--config .python-lsp-compare/lsp_servers.json \
--scenario hover \
--scenario completion \
--output-dir results/servers

Run benchmark suites across all servers (using cached GitHub-release or PyPI installs by default):

python -m python_lsp_compare bench-servers \
--baseline-server pyright \
--output-dir results/bench-servers

Render or re-render a markdown comparison report from an existing multi-server summary JSON file:

python -m python_lsp_compare render-report \
--summary results/bench-servers/summary-20260319T000000Z.json \
--baseline-server pyright \
--output results/bench-servers/comparison.md

Arguments:

• --server-command: executable to launch.
• --server-arg: additional argument, repeatable.
• --scenario: scenario name, repeatable. If omitted, all scenarios run.
• --timeout-seconds: per-request timeout.
• --output: JSON report path.

Configured server arguments:

• --config: path to a local server config JSON file. When omitted, servers are automatically acquired from cached GitHub releases or PyPI installs.
• --server: server id to run, repeatable. If omitted, all servers run.
• --output-dir: directory for per-server JSON reports and the summary JSON file.
• --summary-output: optional path for the combined multi-server summary file.
• --markdown-output: optional path for the combined markdown comparison report. If omitted, a markdown report is written next to the summary JSON.
• --csv-output: optional path for the combined CSV comparison report. If omitted, a CSV report is written next to the summary JSON.
• --baseline-server: configured server id or display name to use as the comparison baseline in markdown and CSV reports.

Configured benchmark runner arguments:

• --protocol: choose lsp, tsp, or all to filter which benchmark family runs.
• --timeout-seconds: override the per-request timeout for all benchmark calls.
• --output-dir: directory for per-server JSON reports and the combined summary/report outputs.
• --summary-output, --markdown-output, --csv-output: override report destinations.
• --baseline-server: choose the comparison baseline for the rendered reports.

Benchmark arguments:

• run-benchmark runs all bundled suites under benchmarks/.
• bench-servers runs that same...