AgileRL AI Agent

<p align="center"> <img src=https://user-images.githubusercontent.com/47857277/222710068-e09a4e3c-368c-458a-9e01-b68674806887.png height="120"> </p> <p align="center"><b>The complete post-training stack for LLMs + RL for any problem</b><br>Visit our <a href="https://agilerl.com">website</a>. View <a href="https://docs.agilerl.com">documentation</a>.<br>Join the <a href="https://discord.gg/eB8HyTA2ux">Discord Server</a> for questions, help and collaboration.</p> <div align="center">

<br>

<h3><i>🚀 <b>Train and deploy at scale with <a href="https://arena.agilerl.com">Arena</a> from AgileRL 🚀</b></i></h3> </div> <br>

AgileRL takes an open-weights LLM from supervised fine-tuning through preference tuning to multi-turn agentic RL, in one library. The same library covers classic deep RL (single-agent, multi-agent, offline and bandits) with evolutionary hyperparameter optimization built in.

Table of Contents

• LLM Post-Training
• Get Started
• Train at Scale on Arena
• Train an LLM Locally
• Classic RL
• Tutorials
• Algorithms
• Citing AgileRL

LLM Post-Training

• The whole pipeline. SFT and DPO, then RL with GRPO, CISPO, GSPO, REINFORCE or PPO.
• Multi-turn agentic RL. Train on any OpenEnv environment: a Python class, a library entrypoint such as GEM, or a remote env server. Or use a dataset plus a reward function. See Environments.
• LoRA. Train small adapters on a frozen base. For RL post-training, LoRA can match full fine-tuning at a fraction of the memory. QLoRA squeezes in bigger models for colocated local runs.
• Torch-native parallelism. Data parallel or FSDP2 sharding, launched with torchrun. One FSDPConfig works for every LLM algorithm. See Multi-GPU LLM training.
• Memory optimizations. Chunked fused log-probs, activation checkpointing and offload, CPU offload for optimizer state or parameters, and trainer offload during rollout.
• Model-specific optimizations. Per-family kernels and settings for high throughput at the lowest memory, on dense, MoE and hybrid Mamba models. MoE LoRA runs per expert without materializing full tensors, and hybrid models like Nemotron-H get fused kernels and fixes for Mamba2.
• Fast rollouts. vLLM generation, with the option to colocate the trainer and vLLM on a single GPU.
• Evolutionary HPO for LLMs. Train a population and let learning rate, KL penalty and more tune themselves during the run.
• Agent-friendly CLI. A training run is one YAML manifest and one command. Your coding agent can read the manifest schema, validate its own config and launch runs on Arena without a human in the loop. See Built for coding agents.
• Async training at scale. Move to Arena for async training on managed GPU clusters, with models up to 120B such as Nemotron 3.5 Super VL. We keep adding models.
• Deploy and chat. Deploy the best checkpoint on Arena and chat with it from the CLI with arena agent generate.

Benchmark

AgileRL trains on over 4x more tokens/s than ART and TRL, and reaches higher reward on half the GPU memory.

AgileRL's CISPO against ART and TRL on the GEM Sudoku Hard task: 32k-token context, up to 50 turns per rollout. The async and HPO runs were on Arena. AgileRL ran on A100 40GB nodes. ART and TRL needed A100 80GB. All runs used the same starting hyperparameters.

<p align="center"> <img src="https://raw.githubusercontent.com/AgileRL/AgileRL/main/docs/_static/multi_turn_llm_benchmarks.png" min-width="100%" width="700"> </p>

Get Started

pip install "agilerl[llm]"        # LLM post-training
pip install agilerl               # classic RL + the Arena CLI

Installation	Description
agilerl[llm]	Hugging Face transformers, PEFT, datasets, Liger, bitsandbytes, and vLLM (Linux).
agilerl[cpu-llm]	Same Hugging Face stack as [llm], without vLLM. Use this when you do not need the vLLM engine (for example with [cpu]).
agilerl[box2d]	Box2D physics engine for Gymnasium environments.
agilerl[cpu]	CPU-only PyTorch wheels (no NVIDIA stack).
agilerl[all]	Box2D and LLM extras.
agilerl-arena	Arena SDK and CLI only, no torch. Included with agilerl.

LLM training on CUDA needs FlashAttention 2 for most models. Install it after torch: pip install flash-attn --no-build-isolation.

PyPI is the released line. main can be ahead.

For the development tip of main: pip install git+https://github.com/AgileRL/AgileRL.git@main. For development mode: clone the repo and run pip install -e ..

Train at Scale on Arena

Every training run is a YAML manifest. Arena runs it on managed GPU clusters, with async rollouts and population-based HPO across nodes. Supported base models range from Qwen, Granite and Gemma up to 100B+ models such as Nemotron 3.5 Super VL 120B-A12B, and the list keeps growing. The arena CLI ships with agilerl:

arena login
arena models list                                   # supported base models
arena experiments submit my_manifest.yaml --project my-project
arena agent deploy my-experiment                    # deploy the best checkpoint
arena agent run my-deployment
arena agent generate --prompt "What is 17 * 23?"

Built for coding agents

Point Cursor, Claude Code or Codex at the arena CLI and it can run the whole loop itself: write a manifest, check it, launch it, pull metrics and deploy.

export ARENA_API_KEY="arena_pat_..."                # no interactive login
arena manifest schema                               # JSON Schema the agent writes against
arena manifest validate my_manifest.yaml --json     # one JSON verdict, stable exit codes
arena experiments submit my_manifest.yaml --project my-project
arena experiments metrics my-experiment             # download metrics to judge the run

Upload your own data with arena datasets create, or your own environment with arena env validate. The same operations are on ArenaClient in Python. See the Arena docs and the agilerl-arena README.

Train an LLM Locally

The same manifests run on your own GPU. This one trains Qwen2.5-0.5B with GRPO on GEM's multi-turn Guess the Number game:

pip install gem-llm
python -m agilerl.train configs/training/llm_finetuning/grpo_env.yaml

Or from Python:

from agilerl import LocalTrainer

trainer = LocalTrainer.from_manifest("configs/training/llm_finetuning/grpo_env.yaml")
population, fitnesses = trainer.train()

More manifests for SFT, DPO, CISPO, GSPO, LLM PPO and LLM REINFORCE are in configs/training/llm_finetuning.

Classic RL

<p align="center"> <img src=https://user-images.githubusercontent.com/47857277/236407686-21363eb3-ffcf-419f-b019-0be4ddf1ed4a.gif width="100%" max-width="900"> </p>

On-policy, off-policy, offline, multi-agent and contextual bandit algorithms, all with evolutionary HPO. Train a population of agents and the hyperparameters evolve during a single run, instead of running hundreds of trials first.

from agilerl import LocalTrainer
from agilerl.models import TrainingSpec

trainer = LocalTrainer(
    algorithm="DQN",
    environment="LunarLander-v3",
    training=TrainingSpec(pop_size=4),
    hpo=True,
)
population, fitnesses = trainer.train()

Or from a manifest: python -m agilerl.train configs/training/dqn/dqn.yaml. You can swap in your own Gymnasium or PettingZoo environments, evolvable networks, algorithms or training loops. See Trainers.

A single AgileRL run with evolutionary HPO against the multiple Optuna runs needed to tune other frameworks. Global steps counts every step taken by every agent in the population:

<p align="center"> <img src=https://user-images.githubusercontent.com/47857277/227481592-27a9688f-7c0a-4655-ab32-90d659a71c69.png min-width="100%" width="600"> </p>

Tutorials

Tutorial Type	Description	Tutorials
LLM Fine-tuning	SFT, DPO, single- and multi-turn RL, HPO and remote environments.	GRPO reasoning <br> SFT and DPO <br> GRPO with HPO <br> Multi-turn GRPO and PPO <br> Remote env server
Training on Arena	Upload and validate custom environments, submit training jobs on managed cloud infrastructure, and deploy trained agents for inference.	PPO - Acrobot Custom Environment
Single-agent tasks	Train on- and off-policy agents on Gymnasium environments.	PPO - Acrobot <br> TD3 - Lunar Lander <br> Rainbow DQN - CartPole <br> Recurrent PPO - Masked Pendulum
Multi-agent tasks	PettingZoo environments, including Connect Four with curriculum learning and self-play.	DQN - Connect Four <br> MADDPG - Space Invaders <br> MATD3 - Speaker Listener
Hierarchical curriculum learning	Teach agents skills and combine them to reach an end goal.	PPO - Lunar Lander
Contextual multi-arm bandits	Make the right decision in single-timestep environments.	NeuralUCB - Iris Dataset <br> NeuralTS - PenDigits
Custom Modules & Networks	Build custom evolvable modules and networks.	Dueling Distributional Q Network <br> EvolvableSimBa

Algorithms

LLM Post-Training

Type	Algorithm
RL	Group Relative Policy Optimization (GRPO) <br> Clipped Importance Sampling Policy Optimization (CISPO) <br> Group Sequence Policy Optimization (GSPO) <br> LLM Proximal Policy Optimization (LLM PPO) <br> LLM REINFORCE
Preference	Direct Preference Optimization (DPO)
Supervised	Supervised Fine-Tuning (SFT)

Single-agent

RL	Algorithm
On-Policy	Proximal Policy Optimization (PPO)
Off-Policy	Deep Q Learning (DQN) <br> Rainbow DQN <br> Deep Deterministic Policy Gradient (DDPG) <br> Twin Delayed Deep Deterministic Policy Gradient (TD3)
Offline	Conservative Q-Learning (CQL) <br> Implicit Language Q-Learning (ILQL)

Multi-agent

RL	Algorithm
Multi-agent	Multi-Agent Deep Deterministic Policy Gradient (MADDPG) <br> Multi-Agent Twin-Delayed Deep Deterministic Policy Gradient (MATD3) <br> Independent Proximal Policy Optimization (IPPO)

Contextual multi-armed bandit

RL	Algorithm
Bandits	Neural Contextual Bandits with UCB-based Exploration (NeuralUCB) <br> Neural Contextual Bandits with Thompson Sampling (NeuralTS)

Citing AgileRL

If you use AgileRL in your work, please cite the repository:

@software{Ustaran-Anderegg_AgileRL,
author = {Ustaran-Anderegg, Nicholas and Pratt, Michael and Sabal-Bermudez, Jaime and Doherty, Michael},
license = {Apache-2.0},
title = {{AgileRL}},
url = {https://github.com/AgileRL/AgileRL}
}