arcee-ai/open-instruct

arcee-ai/open-instruct

License: Apache-2.0

Stars: 1

Forks: 0

Open issues: 1

Created: 2024-12-11T23:54:41Z

Pushed: 2025-02-11T16:26:40Z

Default branch: main

Fork: yes

Parent repository: allenai/open-instruct

Archived: no

README:

Training Open Instruction-Following Language Models

This repo serves as an open effort on instruction-tuning and post-training popular pretrained language models on publicly available datasets. We release this repo and will keep updating it with:

1. Code for finetuning language models with latest techniques and instruction datasets in a unified format. 2. Code for DPO, preference finetuning and reinforcement learning with verifiable rewards (RLVR). 3. Code for running standard evaluation on a range of benchmarks, targeting for differnt capabilities of these language models (now in conjunction with OLMES). 4. Checkpoints or other useful artifacts that we build in our exploration.

The lastest details on open post-training are found in TÜLU 3: Pushing Frontiers in Open Language Model Post-Training.

Please see our first paper How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources for more thoughts behind this project and our initial findings. Please see our second paper Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2 for results using Llama-2 models and direct preference optimization. We are still working on more models. For more recent results involving PPO and DPO please see our third paper Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback.

Try some of the models we train with Open Instruct. There is a free demo or download them from HuggingFace:

| Stage | Llama 3.1 8B | Llama 3.1 70B | OLMo-2 7B | OLMo-2 13B | |----------------------|----------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------| | Base Model | meta-llama/Llama-3.1-8B | meta-llama/Llama-3.1-70B | allenai/OLMo2-7B-1124 | allenai/OLMo-2-13B-1124 | | SFT | allenai/Llama-3.1-Tulu-3-8B-SFT | allenai/Llama-3.1-Tulu-3-70B-SFT | allenai/OLMo-2-1124-7B-SFT | allenai/OLMo-2-1124-13B-SFT | | DPO | allenai/Llama-3.1-Tulu-3-8B-DPO | allenai/Llama-3.1-Tulu-3-70B-DPO | allenai/OLMo-2-1124-7B-DPO | allenai/OLMo-2-1124-13B-DPO | | Final Models (RLVR) | allenai/Llama-3.1-Tulu-3-8B | allenai/Llama-3.1-Tulu-3-70B | allenai/OLMo-2-1124-7B-Instruct | allenai/OLMo-2-1124-13B-Instruct | | Reward Model (RM)| allenai/Llama-3.1-Tulu-3-8B-RM | (Same as 8B) | allenai/OLMo-2-1124-7B-RM | (Same as 7B) |

News

• [2024-11-22] We released TÜLU 3: Pushing Frontiers in Open Language Model Post-Training and updated our entire stack of open post-training recipes with both Llama 3.1 and OLMo 2.
• [2024-07-01] We released Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback and have majorly updated our codebase to support new models and package versions.
• [2023-11-27] We released Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2. Check out our models here. We have added a DPO finetuning script for replicating our results.
• [2023-09-26] We switched to use the official alpaca-eval library to run AlpacaFarm evaluation but use regenerated longer reference outputs. This will change our numbers reported in the paper. We will update the paper soon.
• [2023-09-25] Supported using vLLM for our evaluations, which speeds up the evaluation by 10x.
• [2023-09-17] Supported LoRA and QLoRA finetuning. See [here](#parameter-efficient-finetuning) for more details.
• [2023-08-18] Added support for ToxiGen/TruthfulQA evaluation. Check our scripts/eval/ for examples of running them.
• [2023-08-08] Supported several new instruction dataset, including LIMA / WizardLM / Open-Orca. See the [preparation script](./scripts/data/prepare_train_data.sh) for details. Performance hasn't been evaluated yet.
• [2023-08-06] Supported LLaMa 2 finetuning and FlashAttention-2 by bumping the version of transformers and many other dependencies.
• [2023-06-29] Added [licensing info](#licensing) for our released models.
• [2023-06-09] Released Tülu (a suite of LLaMa models fully-finetuned on a strong mix of datasets) and many other checkpoints on HuggingFace [[Links]](#released-checkpoints).
• [2023-06-09] Initial release of the codebase containing the training and…