togethercomputer/lm-evaluation-harness

togethercomputer/lm-evaluation-harness

Description: A framework for few-shot evaluation of autoregressive language models.

License: MIT

Stars: 0

Forks: 0

Open issues: 0

Created: 2023-10-30T16:26:08Z

Pushed: 2023-10-30T16:43:37Z

Default branch: master

Fork: yes

Parent repository: EleutherAI/lm-evaluation-harness

Archived: no

README:

Language Model Evaluation Harness

We're Refactoring LM-Eval!

(as of 6/15/23) We have a revamp of the Evaluation Harness library internals staged on the big-refactor branch! It is far along in progress, but before we start to move the master branch of the repository over to this new design with a new version release, we'd like to ensure that it's been tested by outside users and there are no glaring bugs.

We’d like your help to test it out! you can help by: 1. Trying out your current workloads on the big-refactor branch, and seeing if anything breaks or is counterintuitive, 2. Porting tasks supported in the previous version of the harness to the new YAML configuration format. Please check out our task implementation guide for more information.

If you choose to port a task not yet completed according to our checklist, then you can contribute it by opening a PR containing [Refactor] in the name with:

• A shell command to run the task in the master branch, and what the score is
• A shell command to run the task in your PR branch to big-refactor, and what the resulting score is, to show that we achieve equality between the two implementations.

Lastly, we'll no longer be accepting new feature requests beyond those that are already open to the master branch as we carry out this switch to the new version over the next week, though we will be accepting bugfixes to master branch and PRs to big-refactor. Feel free to reach out in the #lm-thunderdome channel of the EAI discord for more information.

Overview

This project provides a unified framework to test generative language models on a large number of different evaluation tasks.

Features:

• 200+ tasks implemented. See the [task-table](./docs/task_table.md) for a complete list.
• Support for models loaded via transformers (including quantization via AutoGPTQ), GPT-NeoX, and Megatron-DeepSpeed, with a flexible tokenization-agnostic interface.
• Support for commercial APIs including OpenAI, goose.ai, and TextSynth.
• Support for evaluation on adapters (e.g. LoRa) supported in HuggingFace's PEFT library.
• Evaluating with publicly available prompts ensures reproducibility and comparability between papers.
• Task versioning to ensure reproducibility when tasks are updated.

Install

To install lm-eval from the github repository main branch, run:

git clone https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .

To install additional multilingual tokenization and text segmentation packages, you must install the package with the multilingual extra:

pip install -e ".[multilingual]"

To support loading GPTQ quantized models, install the package with the auto-gptq extra:

pip install -e ".[auto-gptq]"

Basic Usage

> Note: When reporting results from eval harness, please include the task versions (shown in results["versions"]) for reproducibility. This allows bug fixes to tasks while also ensuring that previously reported scores are reproducible. See the [Task Versioning](#task-versioning) section for more info.

Hugging Face transformers

To evaluate a model hosted on the HuggingFace Hub (e.g. GPT-J-6B) on hellaswag you can use the following command:

python main.py \
--model hf-causal \
--model_args pretrained=EleutherAI/gpt-j-6B \
--tasks hellaswag \
--device cuda:0

Additional arguments can be provided to the model constructor using the --model_args flag. Most notably, this supports the common practice of using the revisions feature on the Hub to store partially trained checkpoints, or to specify the datatype for running a model:

python main.py \
--model hf-causal \
--model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float" \
--tasks lambada_openai,hellaswag \
--device cuda:0

To evaluate models that are loaded via AutoSeq2SeqLM in Huggingface, you instead use hf-seq2seq. *To evaluate (causal) models across multiple GPUs, use --model hf-causal-experimental*

> Warning: Choosing the wrong model may result in erroneous outputs despite not erroring.

Commercial APIs

Our library also supports language models served via the OpenAI API:

export OPENAI_API_SECRET_KEY=YOUR_KEY_HERE
python main.py \
--model gpt3 \
--model_args engine=davinci \
--tasks lambada_openai,hellaswag

While this functionality is only officially maintained for the official OpenAI API, it tends to also work for other hosting services that use the same API such as [goose.ai](goose.ai) with minor modification. We also have an implementation for the TextSynth API, using --model textsynth.

To verify the data integrity of the tasks you're performing in addition to running the tasks themselves, you can use the --check_integrity flag:

python main.py \
--model gpt3 \
--model_args engine=davinci \
--tasks lambada_openai,hellaswag \
--check_integrity

Other Frameworks

A number of other libraries contain scripts for calling the eval harness through their library. These include GPT-NeoX, Megatron-DeepSpeed, and mesh-transformer-jax.

💡 Tip: You can inspect what the LM inputs look like by running the following command:

python write_out.py \…