databricks/learning-spark

databricks/learning-spark

Description: Example code from Learning Spark book

Language: Java

License: MIT

Stars: 3892

Forks: 2397

Open issues: 31

Created: 2014-06-16T04:47:54Z

Pushed: 2025-07-12T00:50:48Z

Default branch: master

Fork: no

Archived: no

README:  Examples for Learning Spark =============== Examples for the Learning Spark book. These examples require a number of libraries and as such have long build files. We have also added a stand alone example with minimal dependencies and a small build file in the mini-complete-example directory.

These examples have been updated to run against Spark 1.3 so they may be slightly different than the versions in your copy of "Learning Spark".

Requirements ==

• JDK 1.7 or higher
• Scala 2.10.3
• scala-lang.org
• Spark 1.3
• Protobuf compiler
• On debian you can install with sudo apt-get install protobuf-compiler
• R & the CRAN package Imap are required for the ChapterSixExample
• The Python examples require urllib3

Python examples ===

From spark just run ./bin/pyspark ./src/python/[example]

Spark Submit ===

You can also create an assembly jar with all of the dependencies for running either the java or scala versions of the code and run the job with the spark-submit script

./sbt/sbt assembly OR mvn package cd $SPARK_HOME; ./bin/spark-submit --class com.oreilly.learningsparkexamples.[lang].[example] ../learning-spark-examples/target/scala-2.10/learning-spark-examples-assembly-0.0.1.jar