RepoNVIDIANVIDIApublished Sep 23, 2021seen 2d

NVIDIA/cuml-spark

Jupyter Notebook

Open original ↗

Captured source

source ↗
GH
GitHub/github.com/NVIDIA/cuml-spark
NVIDIA/cuml-spark repository metadata
Source ↗
published Sep 23, 2021seen 2dcaptured 9hhttp 200method plain

NVIDIA/cuml-spark

Description: Spark RAPIDS MLlib – accelerate Apache Spark MLlib with GPUs

Language: Jupyter Notebook

License: Apache-2.0

Stars: 90

Forks: 33

Open issues: 35

Created: 2021-09-23T15:27:07Z

Pushed: 2026-06-10T04:07:46Z

Default branch: main

Fork: no

Archived: no

README:

Spark Rapids ML

Spark Rapids ML enables GPU accelerated distributed machine learning on Apache Spark. It provides several PySpark ML compatible algorithms powered by the RAPIDS cuML library.

These APIs seek to minimize any code changes to end user Spark code. After your environment is configured to support GPUs (with drivers, CUDA toolkit, and RAPIDS dependencies), you should be able to just change an import statement or class name to take advantage of GPU acceleration. See [here](./python/README.md#clis-enabling-no-package-import-change) for experimental CLIs that enable GPU acceleration without the need for changing the pyspark.ml package names in an existing pyspark ml application.

![Ask DeepWiki](https://deepwiki.com/NVIDIA/spark-rapids-ml)

Python

# from pyspark.ml.feature import PCA
from spark_rapids_ml.feature import PCA

pca = (
PCA()
.setK(3)
.setInputCol("features")
.setOutputCol("pca_features")
)
pca.fit(df)

Supported Algorithms

The following table shows the currently supported algorithms. The goal is to expand this over time with support from the underlying RAPIDS cuML libraries. If you would like support for a specific algorithm, please file a git issue to help us prioritize.

| Supported Algorithms | Python | | :--------------------- | :----: | | CrossValidator | √ | | DBSCAN (*) | √ | | KMeans | √ | | approx/exact k-NN (*) | √ | | LinearRegression | √ | | LogisticRegression | √ | | PCA | √ | | RandomForestClassifier | √ | | RandomForestRegressor | √ | | UMAP (*) | √ |

(*) Notes:

  • As an alternative to KMeans, we also provide a Spark API for GPU accelerated Density-Based Spatial Clustering of Applications with Noise (DBSCAN), a density based clustering algorithm in the RAPIDS cuML library.
  • Spark does not provide a k-Nearest Neighbors (k-NN) implementation, but it does have an LSH-based Approximate Nearest Neighbor implementation.
  • As an alternative to PCA, we also provide a Spark API for GPU accelerated Uniform Manifold Approximation and Projection (UMAP), a non-linear dimensionality reduction algorithm in the RAPIDS cuML library.

Getting started

For PySpark (Python) users, see [this guide](python/README.md).

Performance

GPU acceleration can provide significant performance and cost benefits. Benchmarking instructions and results can be found [here](python/benchmark/README.md).

Contributing

We welcome community contributions! Please refer to [CONTRIBUTING.md](CONTRIBUTING.md) to get started.