scaleway/dagster-scaleway

scaleway/dagster-scaleway

Language: Python

Stars: 1

Forks: 2

Open issues: 2

Created: 2023-11-27T07:58:21Z

Pushed: 2024-04-24T08:29:35Z

Default branch: main

Fork: no

Archived: no

README:

Dagster Scaleway (WIP)

This repository contains a Dagster integration for Scaleway.

It allows you to run Dagster pipelines on Scaleway Serverless Jobs.

Basic usage

Build a docker image containing your Dagster code and push it to the Scaleway Registry (or any other registry of your choice):

dagster project scaffold --name my-dagster-project
cd my-dagster-project

cat Dockerfile
FROM python:3.12-slim-bookworm
WORKDIR /app
COPY . .
RUN pip install pendulum==2.0.3
RUN pip install .
# Install the Dagster Scaleway module. You can also specify it in your "setup.py" file
RUN pip install dagster_scaleway
EOF

Build and push the image:

docker build -t rg.fr-par.scw.cloud//dagster-scaleway-example:latest .
docker push rg.fr-par.scw.cloud//dagster-scaleway-example:latest

Then, configure the dagster.yaml file to use this image:

run_launcher:
module: dagster_scaleway
class: ScalewayServerlessJobRunLauncher
config:
docker_image: rg.fr-par.scw.cloud//dagster-scaleway-example:latest

Run Dagster locally:

pip install -e ".[dev]" "dagster-scaleway"
dagster dev

Your Dagster ops will be run as Scaleway Serverless Jobs! :tada:

See the Dagster documentation for more information on how to get started with Dagster.

Examples

See the [examples](./examples) folder for examples of how to use this integration.