amazon-science/DeclarativeDataPipeline

amazon-science/DeclarativeDataPipeline

Language: Scala

License: NOASSERTION

Stars: 0

Forks: 0

Open issues: 0

Created: 2025-11-04T22:03:44Z

Pushed: 2025-11-05T21:53:42Z

Default branch: main

Fork: no

Archived: no

README:

Declarative Data Pipeline

A multi-module Spark-based data pipeline application built with Scala and Java.

Modules

• core - Core declarative data pipeline framework
• examples - Example implementations and sample jobs

Build

# Build all modules
sbt compile

# Build specific module
sbt core/compile
sbt examples/compile

Test

# Test all modules
sbt test

# Test specific module
sbt core/test

Run Examples

# Run the simple example
sbt examples/run

Dependencies

• Apache Spark 3.5.4
• Scala 2.12.19
• Java 17+

Package Structure

• core/ - Core framework with com.amazonaws.declarative.data.pipeline package
• examples/ - Sample jobs demonstrating the declarative data pipeline pattern
• Supports both Scala and Java components
• Includes unit tests for both languages