databricks/benchbase

databricks/benchbase

Description: Multi-DBMS SQL Benchmarking Framework via JDBC

Language: Java

License: NOASSERTION

Stars: 0

Forks: 0

Open issues: 0

Created: 2022-12-04T22:00:47Z

Pushed: 2023-10-17T08:17:38Z

Default branch: main

Fork: yes

Parent repository: cmu-db/benchbase

Archived: no

README:

BenchBase


BenchBase (formerly OLTPBench) is a Multi-DBMS SQL Benchmarking Framework via JDBC.

Table of Contents

• [Quickstart](#quickstart)
• [Description](#description)
• [Usage Guide](#usage-guide)
• [Contributing](#contributing)
• [Known Issues](#known-issues)
• [Credits](#credits)
• [Citing This Repository](#citing-this-repository)

---

Quickstart

To clone and build BenchBase using the postgres profile,

git clone --depth 1 https://github.com/cmu-db/benchbase.git
cd benchbase
./mvnw clean package -P postgres

This produces artifacts in the target folder, which can be extracted,

cd target
tar xvzf benchbase-postgres.tgz
cd benchbase-postgres

Inside this folder, you can run BenchBase. For example, to execute the tpcc benchmark,

java -jar benchbase.jar -b tpcc -c config/postgres/sample_tpcc_config.xml --create=true --load=true --execute=true

A full list of options can be displayed,

java -jar benchbase.jar -h

---

Description

Benchmarking is incredibly useful, yet endlessly painful. This benchmark suite is the result of a group of PhDs/post-docs/professors getting together and combining their workloads/frameworks/experiences/efforts. We hope this will save other people's time, and will provide an extensible platform, that can be grown in an open-source fashion.

BenchBase is a multi-threaded load generator. The framework is designed to be able to produce variable rate, variable mixture load against any JDBC-enabled relational database. The framework also provides data collection features, e.g., per-transaction-type latency and throughput logs.

The BenchBase framework has the following benchmarks:

• AuctionMark
• CH-benCHmark
• Epinions.com
• hyadapt -- pending configuration files
• NoOp
• OT-Metrics
• Resource Stresser
• SEATS
• SIBench
• SmallBank
• TATP
• TPC-C
• TPC-H
• TPC-DS -- pending configuration files
• Twitter
• Voter
• Wikipedia
• YCSB

This framework is design to allow for easy extension. We provide stub code that a contributor can use to include a new benchmark, leveraging all the system features (logging, controlled speed, controlled mixture, etc.)

---

Usage Guide

How to Build

Run the following command to build the distribution for a given database specified as the profile name (-P). The following profiles are currently supported: postgres, mysql, mariadb, sqlite, cockroachdb, phoenix, and spanner.

./mvnw clean package -P

The following files will be placed in the ./target folder:

• benchbase-.tgz
• benchbase-.zip

How to Run

Once you build and unpack the distribution, you can run benchbase just like any other executable jar. The following examples assume you are running from the root of the expanded .zip or .tgz distribution. If you attempt to run benchbase outside of the distribution structure you may encounter a variety of errors including java.lang.NoClassDefFoundError.

To bring up help contents:

java -jar benchbase.jar -h

To execute the tpcc benchmark:

java -jar benchbase.jar -b tpcc -c config/postgres/sample_tpcc_config.xml --create=true --load=true --execute=true

For composite benchmarks like chbenchmark, which require multiple schemas to be created and loaded, you can provide a comma separated list:

java -jar benchbase.jar -b tpcc,chbenchmark -c config/postgres/sample_chbenchmark_config.xml --create=true --load=true --execute=true

The following options are provided:

usage: benchbase
-b,--bench [required] Benchmark class. Currently
supported: [tpcc, tpch, tatp, wikipedia,
resourcestresser, twitter, epinions, ycsb,
seats, auctionmark, chbenchmark, voter,
sibench, noop, smallbank, hyadapt, otmetrics]
-c,--config [required] Workload configuration file
--clear Clear all records in the database for this
benchmark
--create Initialize the database for this benchmark
-d,--directory Base directory for the result files,
default is current directory
--dialects-export Export benchmark SQL to a dialects file
--execute Execute the benchmark workload
-h,--help Print this help
-im,--interval-monitor Throughput Monitoring Interval in
milliseconds
--load Load data using the benchmark's data
loader
-s,--sample Sampling window

How to Run with Maven

Instead of first building, packaging and extracting before running benchbase, it is possible to execute benchmarks directly against the source code using Maven. Once you have the project cloned you can run any benchmark from the root project directory using the Maven exec:java goal. For example, the following command executes the tpcc benchmark against postgres:

mvn clean compile exec:java -P postgres -Dexec.args="-b tpcc -c config/postgres/sample_tpcc_config.xml --create=true --load=true --execute=true"

this is equivalent to the steps above but eliminates the need to first package and then extract the distribution.

How to Enable Logging

To enable logging, e.g., for the PostgreSQL JDBC driver, add the following JVM property when starting...