Reinforcement Learning With Verifiable Reward

Beyond Supervised Fine Tuning: How Reinforcement Learning Empowers AI with Minimal Labels

GLM 5.2 is live! Opus-level intelligence at open-source rates. Pay per token on serverless. Try it today.

Blog

Reinforcement Learning With Verifiable Reward Beyond Supervised Fine Tuning: How Reinforcement Learning Empowers AI with Minimal Labels

PUBLISHED 1/27/2025

Table of Contents

TL;DR Introduction What is GRPO? Reinforcement Learning with Verifiable Reward How Effective is RLVR?

Experiment 1: digit multiplication

Experiment 2: function picking Looking Forward

Table of Contents

TL;DR

DeepSeek R1 employs a streamlined variant of reinforcement learning (RL), significantly reducing training complexity and data collection costs Fireworks AI explored a comparable RL approach, demonstrating its effectiveness on a fully synthetic dataset This emerging class of algorithms makes RL more accessible, establishing it as a valuable complement to supervised fine-tuning in the post-training toolkit

Introduction

DeepSeek R1 and DeepSeek R1-Zero are all the rage right now. While DeepSeek R1 is likely a more suitable choice for production, DeepSeek R1-Zero as an exploratory model has also sparked significant interest in the community. For those of you who haven’t read the DeepSeek R1 technical report, the DeepSeek R1-Zero is a model trained without any supervised training data using an algorithm called GRPO (Group Relative Policy Optimization), and it was able to self-evolve to solve complex problems through complex chain of thought. What is GRPO?

GRPO is a reinforcement learning algorithm that shares many similarities with the PPO (Proximal Policy Optimization) algorithm that OpenAI famously adopted in their very original GPT3 training. While PPO is effective, there are several downsides that make it harder to adopt in practice. To name a few: PPO requires co-training of a Value Model that is used to estimate the rollout baseline in GAE (Generalized Advantage Estimation) . Since the Value Model is typically around similar size as the Policy Model, it introduces significant compute and memory burden to the training pipeline PPO typically requires a token level baseline (the output from Value Model). So you will need a value for each token. This further increases the amount of computations and intermediate memory required during training The need for co-training a Value Model also means that there are more parameters to tune, making parameter searching harder There are more implementation details in PPO algorithm to be taken care of because of the complex GAE calculations and Value Model updates

The GRPO algorithm originally introduced in the DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models paper aims to tackle the above downsides of PPO. So they got rid of the Value Model, and used the normalized reward for different generations on the same prompt as the baseline to estimate an advantage. Token level reward is also no longer required with the removal of the Value Model. Instead, all the advantage estimations are performed on a sequence level (i.e. on the whole completion). Below is a diagram comparing PPO with GRPO taken from the DeepSeekMath paper mentioned above. The other aspect of the GRPO algorithm is similar to the PPO algorithm, (check out this article from our researcher Webster to learn more about how they work in general). To provide some basic intuition to help you read this blogpost along, I typically explain GRPO/PPO or related RL algorithms as this: You ask the model to generate multiple responses on the same prompt, and assign a score to each response via the reward model, or a reward function. Then you nudge the model slightly to make it more likely to generate the responses with higher score, and conversely less likely to generate the responses with lower score. The “nudge” is done by gradient descent.

Reinforcement Learning with Verifiable Reward

So what’s interesting here? We notice that the reinforcement learning algorithm itself is only asking for a reward model to assign a score to each generation, and there is absolutely no requirement on what this reward model should be. Some choices you have are: A deep learning model : you can use a full deep learning model to assign scores to the generations. In LLM training, this reward model could very well be a similar sized LLM A hardcoded function : you could also hardcode a function, encoding a set of rules that checks the model generation and assigns a score to it A combination of the above

What DeepSeek team did for DeepSeek R1-Zero training is essentially option 2) above, that is, assigning scores to generations purely based on a set of rules. For referential consistency, let’s call this function the Verifiable Reward Function . The Verifiable Reward Function could be as simple as: taking a reference answer (if it is given), and the response from the model, and returning a positive score if the response matches with the reference answer, and 0 otherwise. 1 2 3 def verfiable_reward_function ( reference_answer : str , model_response : return 10 if reference_answer == model_response else 0 str ) :

The team utilized set rules to measure how good a response is on verifiable tasks , i.e. the set of questions where the accuracy/correctness of the responses can be easily verified. In particular, they rewarded the responses based on: Whether the response is correct Whether the response is formatted correctly (i.e. putting thinking processes in between predefined tags and then generate the final response)

It was discovered that as training proceeds, the model learned to solve more and more complex tasks with longer and longer reasoning chains. How Effective is RLVR?

Prior to the release of the DeepSeek R1 models and technical report, the Fireworks AI research team also conducted experiments on the effectiveness of the RLVR approach. While we are less interested in tasks such as mathematical problem solving, it would be great to understand how good RL based approaches can be adapted to fine tune models with simple supervision signals to achieve top quality results on constrained task settings, and even beating top of the notch closed source models. We conducted two experiments on two datasets. Experiment 1: digit multiplication

We started off with a simple setup: train a model to perform four digit by 2 digit number multiplications with a fixed prompt: _Please answer the following...