What Reinforcement Learning Is and Why It Matters in 2026

Reinforcement learning (RL) is the branch of machine learning concerned with learning to act. Instead of learning from a fixed dataset of labelled examples, an RL agent learns by interacting with an environment: it takes actions, observes the consequences, and gradually improves its behaviour to maximise a numerical reward. It is the framework behind game-playing systems that beat world champions, robots that learn to walk, recommendation systems that optimise long-term engagement, and — most importantly for this course — the alignment of the large language models you use every day.

Educational note: This course is for learning. Any preference data, human feedback, or training corpus you use in practice must respect data-protection law (GDPR in the EU), dataset licences, and copyright. Alignment work carries real ethical weight — you are shaping how a system behaves toward people — so we return to legal and safety obligations throughout the course. Nothing here is legal advice.

Supervised, unsupervised, and reinforcement learning

It helps to place RL against the other two paradigms. In supervised learning you have inputs paired with correct outputs, and the model learns to reproduce the mapping. In unsupervised learning you have inputs with no labels, and the model finds structure. Reinforcement learning is different in three fundamental ways:

1. There is no supervisor, only a reward signal. Nobody tells the agent the correct action. It only receives a scalar reward that says how good the outcome was, and it must figure out which actions led to that reward.
2. Feedback can be delayed. An action taken now may only pay off many steps later. Winning a chess game depends on moves made long before the final position. This is the credit assignment problem, and it haunts every algorithm we will study.
3. Data is not independent and identically distributed. The agent's own choices determine what it sees next. Explore poorly and you never even observe the states where the good rewards live.

A comparison table makes the contrasts concrete:

These properties make RL powerful and also notoriously difficult. Much of this course is about the algorithms and engineering tricks that make it work anyway.

The agent-environment loop

At the heart of RL is a simple loop. At each time step t the agent observes a state s_t, chooses an action a_t, and the environment responds with a reward r_{t+1} and a new state s_{t+1}. The agent's goal is to choose actions that maximise the cumulative reward over time, not just the immediate one.

# The canonical reinforcement learning loop, expressed against the
# Gymnasium API (the maintained successor to OpenAI Gym).
import gymnasium as gym

env = gym.make("CartPole-v1")
observation, info = env.reset(seed=42)

total_reward = 0.0
for _ in range(1000):
    action = env.action_space.sample()          # a random policy, for now
    observation, reward, terminated, truncated, info = env.step(action)
    total_reward += reward
    if terminated or truncated:
        observation, info = env.reset()

env.close()
print("Total reward collected:", total_reward)

This snippet already contains the entire vocabulary of RL: an environment, observations (states), an action space, a reward, and the notions of terminated (the task ended on its own terms — the pole fell) and truncated (an external time limit was hit). The distinction matters more than it looks: when an episode is truncated, the state was not actually terminal, and value-based algorithms must still bootstrap from it; treating truncation as termination is one of the most common silent bugs in RL code. Everything else in this course is about replacing env.action_space.sample() — a random policy — with something that learns.

States versus observations

Formally, the state is a complete description of the environment at a moment in time, while an observation is what the agent actually perceives. In chess the two coincide: the board is fully visible, so the environment is fully observable. In poker, or for a robot with a single camera, the observation is only a partial, noisy window onto the true state — the environment is partially observable, and the agent may need memory (a recurrent network, or a stack of recent frames as DQN uses for Atari) to act well. Most of this course assumes full observability because it keeps the mathematics clean, but you should always ask, when an agent underperforms, whether it can even see what it needs to decide.

Episodic and continuing tasks

Tasks divide into episodic ones, which end — a game of chess, one CartPole balancing attempt — and continuing ones, which run indefinitely, like a server-cooling controller. Episodic tasks let us speak of the return of an episode; continuing tasks force us to discount future rewards so that the infinite sum stays finite. The discount factor gamma, which we formalise in the next lesson, is the knob that trades off immediate against future reward in both settings.

Reward is the only objective

A defining principle of RL is the reward hypothesis: any goal can be framed as the maximisation of expected cumulative reward. This is deceptively powerful. You do not program how to solve the task; you specify what success looks like via the reward, and learning discovers the how. But it is also the source of the field's deepest hazard. If your reward does not capture exactly what you want, the agent will happily maximise the reward you wrote while doing something you never intended — a failure mode called reward hacking (or specification gaming). A boat-racing agent that discovers it can loop forever collecting respawning bonus targets instead of finishing the race is the classic illustration; a language model that learns to sound confident rather than be correct is the version that matters commercially in 2026. We dedicate an entire lesson to this failure mode later, because in RLHF the "environment" includes human judgement, and gaming human judgement is exactly what an over-optimised model learns to do.

A brief map of the algorithm landscape

Before diving into mathematics, it pays to know the lay of the land, because every algorithm we study sits somewhere on three axes:

• Value-based versus policy-based. Value-based methods (Q-learning, DQN) learn how good each action is and act greedily on that estimate. Policy-based methods (REINFORCE, PPO) learn the acting behaviour directly. Actor-critic methods combine both.
• Model-free versus model-based. Model-free methods learn from raw experience without predicting how the environment works. Model-based methods learn or use a model of the environment's dynamics to plan ahead (as AlphaZero does with its known game rules). This course is predominantly model-free, because that is what RLHF uses.
• On-policy versus off-policy. On-policy methods (PPO) can only learn from data generated by the current policy; off-policy methods (Q-learning, DQN) can reuse old or even foreign experience. The distinction drives sample efficiency, stability, and infrastructure design, and we will return to it constantly.

Keep this map in mind: when you meet a new algorithm — including the 2026 alignment methods — locating it on these three axes tells you most of what to expect about its behaviour.

A short history that explains the present

RL is not new. Temporal-difference learning powered TD-Gammon in the early 1990s, a backgammon program that reached top human level by self-play. The field's modern era began when DQN (DeepMind, published in Nature in 2015) learned dozens of Atari games from raw pixels with one architecture, proving deep networks and RL could be combined stably. AlphaGo defeated Lee Sedol at Go in 2016, and its successors AlphaZero and MuZero generalised the recipe. Then in 2022, InstructGPT showed that the same machinery — a policy optimised with PPO against a learned reward model — could turn a raw language model into an assistant that follows instructions. That recipe, RLHF, is the direct ancestor of the systems you use today. Since then the field has moved fast: DPO (2023) removed the need for an explicit RL loop in many cases, GRPO (2024) simplified PPO for language models, and reasoning-focused training with verifiable rewards (prominent since late 2024 and central to models like DeepSeek-R1, released in January 2025) brought classic RL ideas roaring back. Every one of these gets a dedicated lesson.

Why RL matters right now

For a while, RL was seen as a niche of games and robotics. That changed dramatically. The modern conversational assistants — the systems behind ChatGPT, Claude Opus 4.8, Claude Sonnet 5, GPT-5.5 and Gemini 3.1 Pro — are not merely trained to predict the next token. After pretraining, they are aligned using human (and increasingly AI) feedback, and the mathematical engine of that alignment is reinforcement learning. Reinforcement Learning from Human Feedback (RLHF) and its newer relatives are the reason these models are helpful, follow instructions, and refuse harmful requests instead of simply continuing text. Reasoning-oriented models additionally use RL against verifiable rewards — unit tests, mathematical answer checking — to learn multi-step problem solving. Understanding RL is now a prerequisite for understanding how state-of-the-art AI is actually built, and it is a differentiating skill for ML engineers: far fewer practitioners can debug a PPO run or a reward model than can fine-tune a classifier.

When RL is the right tool — and when it is not

RL shines when decisions are sequential, feedback is evaluative rather than instructive, and the data you need can only be produced by interacting with the system. It is usually the wrong tool when you already have labelled examples of correct behaviour (use supervised learning — it is simpler and more stable), when a hand-written controller or classical optimisation solves the problem, or when exploration is unacceptable in the real system and you have no simulator. A useful pre-flight checklist before reaching for RL:

• Can I simulate the environment cheaply, or safely collect real interactions?
• Can I write a reward that genuinely captures success, and audit it for gaming?
• Is the sequential structure essential, or am I dressing up a one-shot prediction problem?
• Do I have the engineering budget for instability, hyperparameter sensitivity, and evaluation?

If any answer is no, reconsider. Many production "RL problems" are solved better by supervised fine-tuning plus a ranking model — a theme that returns when we compare RLHF with DPO.

What this course covers

We proceed in two arcs. The first builds classical and deep RL from the ground up: Markov decision processes, value functions and the Bellman equations, dynamic programming, exploration, Monte Carlo and temporal-difference learning, Q-learning and Deep Q-Networks with their modern refinements, policy gradients, actor-critic methods, continuous control, and PPO — the workhorse algorithm. The second arc applies this machinery to language models: preference data and its legal obligations, reward modelling, PPO-based RLHF as used to train assistants, the simpler modern alternatives (DPO, GRPO, RLAIF, Constitutional AI, verifiable rewards), followed by reward hacking, alignment and safety, and hands-on pipelines with Gymnasium, Stable-Baselines3 and Hugging Face TRL.

By the end you will not only be able to train agents and align models — you will understand why each method works, which is what lets you debug the inevitable failures and adapt as the field keeps moving. Let us begin with the formal object that underlies everything: the Markov decision process.