databricks/notebook-best-practices

databricks/notebook-best-practices

Description: An example showing how to apply software engineering best practices to Databricks notebooks.

Language: Python

License: Apache-2.0

Stars: 155

Forks: 75

Open issues: 2

Created: 2022-05-16T17:04:42Z

Pushed: 2024-07-24T18:09:51Z

Default branch: main

Fork: no

Archived: no

README:

Software engineering best practices for Databricks notebooks

This repository is a companion for the example article "Software engineering best practices for Databricks notebooks" (AWS | Azure | GCP).

Going through the example, you will:

• Add notebooks to Databricks Repos for version control.
• Extracts portions of code from one of the notebooks into a shareable component.
• Test the shared code.
• Automatically run notebooks in git on a schedule using a Databricks job.
• Optionally, apply CI/CD to the notebooks and the shared code.

The example is hands-on. We recommend working it step-by-step to learn how to apply these techniques to your own Databricks notebooks.