UpstageAI/evalverse

UpstageAI/evalverse

Description: The Universe of Evaluation. All about the evaluation for LLMs.

Language: Python

License: Apache-2.0

Stars: 235

Forks: 25

Open issues: 10

Created: 2024-03-22T00:11:47Z

Pushed: 2024-07-09T19:17:14Z

Default branch: main

Fork: no

Archived: yes

README:

🚀 Newly updated

• [2024.05.10] LLM-Evaluation Report of Evalverse is now available on HuggingFace Space.

👋 Welcome to Evalverse!

Evalverse is a freely accessible, open-source project designed to support your LLM (Large Language Model) evaluation needs. We provide a simple, standardized, and user-friendly solution for the processing and management of LLM evaluations, catering to the needs of AI research engineers and scientists. We also support no-code evaluation processes for people who may have less experience working with LLMs. Moreover, you will receive a well-organized report with figures summarizing the evaluation results.

With Evalverse, you are empowered to

• access various evaluation methods without juggling multiple libraries.
• receive insightful report about the evaluation results that helps you to compare the varied scores across different models.
• initiate evaluation and generate reports without any code via Slack bot.

Architecture of Evalverse

Key Features of Evalverse

• Unified evaluation with Submodules: Evalverse extends its evaluation capabilities through Git submodules, effortlessly incorporating frameworks like lm-evaluation-harness and FastChat. Swiftly add new tools and keep pace with the latest in LLM evaluation.
• No-code evaluation request: With Evalverse, request LLM evaluations without any code, simply by sending Request! in a direct message or Slack channel with an activate Evalverse Slack bot. Enter the model name in the Huggingface hub or local model directory path in Slack, and let the bot handle the rest.
• LLM evaluation report: Obtain comprehensive, no-code reports from Evalverse. Request with a simple command -Report!-, select the model and evaluation criteria, and receive detailed reports with scores, rankings, and visuals, all generated from the stored score database.

If you want to know more about Evalverse, please checkout our docs. By clicking below image, it'll take you to a short intro video! [](https://www.youtube.com/watch?v=-VviAutjpgM)

🌌 Installation

🌠 Option 1: Git clone

Before cloning, please make sure you've registered proper SSH keys linked to your GitHub account.

1. Clone the Evalverse repository

• Notes: add --recursive option to also clone submodules

git clone --recursive https://github.com/UpstageAI/evalverse.git

2. Install requirement packages

cd evalverse
pip install -e .

🌠 Option 2: Install via Pypi *(WIP)*

> Currently, installation via Pypi is not supported. Please install Evalverse with option 1.

🌌 Configuration

You have to set an API key and/or Token in the .env file (rename .env_sample to .env) to use all features of Evalverse.

• OpenAI API Key (required for mt_bench)
• Slack BOT/APP Token (required for slack reporter)

OPENAI_API_KEY=sk-...

SLACK_BOT_TOKEN=xoxb-...
SLACK_APP_TOKEN=xapp-...

🌌 Quickstart

More detailed tutorials are here.

• basic_usage.ipynb: Very basic usage, like how to use Evaluator for evaluation and Reporter for generating report.
• advanced_usage.ipynb: Introduces methods for evaluating each benchmark and all benchmarks collectively.

🌠 Evaluation

💫 Evaluation with Library

The following code is a simple example to evaluate the SOLAR-10.7B-Instruct-v1.0 model on the h6_en (Open LLM Leaderboard) benchmark.

import evalverse as ev

evaluator = ev.Evaluator()

model = "upstage/SOLAR-10.7B-Instruct-v1.0"
benchmark = "h6_en"

evaluator.run(model=model, benchmark=benchmark)

💫 Evaluation with CLI

Here is a CLI script that produces the same result as the above code:

cd evalverse

python3 evaluator.py \
--h6_en \
--ckpt_path upstage/SOLAR-10.7B-Instruct-v1.0

🌠 Report

Currently, generating a report is only available through the library. We will work on a Command Line Interface (CLI) version as soon as possible.

import evalverse as ev

db_path = "./db"
output_path = "./results"
reporter = ev.Reporter(db_path=db_path, output_path=output_path)

reporter.update_db(save=True)

model_list = ["SOLAR-10.7B-Instruct-v1.0", "Llama-2-7b-chat-hf"]
benchmark_list = ["h6_en"]
reporter.run(model_list=model_list, benchmark_list=benchmark_list)

| Model | Ranking | total_avg | H6-ARC | H6-Hellaswag | H6-MMLU | H6-TruthfulQA | H6-Winogrande | H6-GSM8k | |--------------------------:|--------:|----------:|-------:|-------------:|--------:|--------------:|--------------:|---------:| | SOLAR-10.7B-Instruct-v1.0 | 1 | 74.62 | 71.33 | 88.19 | 65.52 | 71.72 | 83.19 | 67.78 | | Llama-2-7b-chat-hf | 2 | 53.51 | 53.16 | 78.59 | 47.38 | 45.31 | 72.69 | 23.96 |

🌌 Supported Evaluations

We currently support four evaluation methods. If you have suggestions for new methods, we welcome your input!

| Evaluation | Original Repository | |---------------------------|--------------------------------------------| | H6 (Open LLM Leaderboard) | EleutherAI/lm-evaluation-harness| | MT-bench | lm-sys/FastChat| | IFEval | google-research/instruction_following_eval| | EQ-Bench | EQ-bench/EQ-Bench|

🌌 Evalverse use-case

> If you have any use-cases of your own, please feel free to let us know.…