databricks/databricks-sql-python

databricks/databricks-sql-python

Description: Databricks SQL Connector for Python

Language: Python

License: Apache-2.0

Stars: 231

Forks: 146

Open issues: 155

Created: 2022-05-18T14:20:03Z

Pushed: 2026-06-11T02:47:43Z

Default branch: main

Fork: no

Archived: no

README:

Databricks SQL Connector for Python


The Databricks SQL Connector for Python allows you to develop Python applications that connect to Databricks clusters and SQL warehouses. It is a Thrift-based client with no dependencies on ODBC or JDBC. It conforms to the Python DB API 2.0 specification.

This connector uses Arrow as the data-exchange format, and supports APIs (e.g. fetchmany_arrow) to directly fetch Arrow tables. Arrow tables are wrapped in the ArrowQueue class to provide a natural API to get several rows at a time. PyArrow is required to enable this and use these APIs, you can install it via pip install pyarrow or pip install databricks-sql-connector[pyarrow].

The connector includes built-in support for HTTP/HTTPS proxy servers with multiple authentication methods including basic authentication and Kerberos/Negotiate authentication. See docs/proxy.md and examples/proxy_authentication.py for details.

You are welcome to file an issue here for general use cases. You can also contact Databricks Support [here](help.databricks.com).

Requirements

Python 3.9 or above is required.

Documentation

For the latest documentation, see

• Databricks
• Azure Databricks

Quickstart

Installing the core library

Install using pip install databricks-sql-connector

Installing the core library with PyArrow

Install using pip install databricks-sql-connector[pyarrow]

Installing with the Rust kernel backend (use_kernel=True)

Install using pip install databricks-sql-connector[kernel]

This adds the optional `databricks-sql-kernel` extension (a native Rust client core, exposed via PyO3). Pass use_kernel=True to sql.connect(...) to route the connection through it instead of the default Thrift backend:

connection = sql.connect(
server_hostname=host,
http_path=http_path,
access_token=token,
use_kernel=True,
)

Notes:

• Requires Python >= 3.10 (the kernel wheel is published as

cp310-abi3). On older interpreters the [kernel] extra installs nothing and use_kernel=True raises an ImportError.

• The extra also pulls in PyArrow, which the kernel result path requires.
• Authentication supports PAT (access_token), OAuth M2M, and OAuth U2M.

export DATABRICKS_HOST=********.databricks.com
export DATABRICKS_HTTP_PATH=/sql/1.0/endpoints/****************

Example usage:

import os
from databricks import sql

host = os.getenv("DATABRICKS_HOST")
http_path = os.getenv("DATABRICKS_HTTP_PATH")

connection = sql.connect(
server_hostname=host,
http_path=http_path)

cursor = connection.cursor()
cursor.execute('SELECT :param `p`, * FROM RANGE(10)', {"param": "foo"})
result = cursor.fetchall()
for row in result:
print(row)

cursor.close()
connection.close()

In the above example:

• server-hostname is the Databricks instance host name.
• http-path is the HTTP Path either to a Databricks SQL endpoint (e.g. /sql/1.0/endpoints/1234567890abcdef),

or to a Databricks Runtime interactive cluster (e.g. /sql/protocolv1/o/1234567890123456/1234-123456-slid123)

> Note: This example uses Databricks OAuth U2M > to authenticate the target Databricks user account and needs to open the browser for authentication. So it > can only run on the user's machine.

Transaction Support

The connector supports multi-statement transactions with manual commit/rollback control. Set connection.autocommit = False to disable autocommit mode, then use connection.commit() and connection.rollback() to control transactions.

For detailed documentation, examples, and best practices, see [TRANSACTIONS.md](TRANSACTIONS.md).

SQLAlchemy

Starting from databricks-sql-connector version 4.0.0 SQLAlchemy support has been extracted to a new library databricks-sqlalchemy.

• Github repository databricks-sqlalchemy github
• PyPI databricks-sqlalchemy pypi

Quick SQLAlchemy guide

Users can now choose between using the SQLAlchemy v1 or SQLAlchemy v2 dialects with the connector core

• Install the latest SQLAlchemy v1 using pip install databricks-sqlalchemy~=1.0
• Install SQLAlchemy v2 using pip install databricks-sqlalchemy

Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md)

License

[Apache License 2.0](LICENSE)