Mastering Multi-Agent Reinforcement Learning with MADDPG
Mastering Multi-Agent Reinforcement Learning with MADDPG
In the world of Reinforcement Learning (RL), moving from a single agent to a multi-agent system isn't just about adding more actors—it's about solving a fundamental problem: non-stationarity. When multiple agents learn simultaneously, the environment changes from the perspective of any single agent, rendering traditional RL algorithms unstable.
Enter MADDPG (Multi-Agent Deep Deterministic Policy Gradient). This framework introduces a powerful paradigm known as Centralized Training with Decentralized Execution (CTDE) to bring stability to the chaos of multi-agent environments.
The Core Intuition: CTDE
The primary challenge in Multi-Agent RL (MARL) is that as Agent A improves its policy, Agent B's environment changes, making Agent B's previous experience obsolete. This "moving target" problem often leads to divergence.
MADDPG solves this by decoupling what the agent knows during training versus what it knows during execution.
1. Centralized Training (The "God's Eye" View)
During training, we allow each agent to have a Critic that has access to global information. This includes the observations and actions of all other agents. By knowing what everyone else is doing, the Critic can provide a stable value signal, effectively accounting for the changing behaviors of peers.
2. Decentralized Execution (The "Local" View)
Once trained, the Critic is discarded. The Actor (the policy) relies solely on its own local observations to make decisions. This ensures that the agents can operate independently in real-time without needing a high-bandwidth communication link to every other agent in the field.
Architectural Deep Dive
The Mathematical Foundation
MADDPG extends the Deterministic Policy Gradient (DPG) to the multi-agent setting. The goal is to optimize the policy $\mu_{\theta}$ to maximize the expected return.
The Policy Gradient: $$\nabla_{\theta} J(\theta) = \mathbb{E}{s \sim p^{\mu}} [\nabla{a} Q^{\mu}(s, a)|{a=\mu{\theta}(s)} \nabla_{\theta} \mu_{\theta}(s)]$$
The Multi-Agent Critic: Unlike standard DDPG, the Critic $Q_i$ for agent $i$ takes as input the joint observations and joint actions of all $N$ agents: $$Q_i(o_1, a_1, \dots, o_N, a_N)$$
System Workflow
The following diagram illustrates how data flows from the environment into the centralized trainer and how the decentralized actors are deployed.
Implementation Guide
Below is a production-ready PyTorch implementation of the MADDPG architecture.
1. Network Definitions
We define a simple MLP for the Actor (local input) and a larger MLP for the Critic (global input).
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import numpy as np
from collections import deque
import random
class Actor(nn.Module):
def __init__(self, obs_dim, act_dim, hidden_dim=64):
super(Actor, self).__init__()
self.net = nn.Sequential(
nn.Linear(obs_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, act_dim),
nn.Tanh() # Actions scaled to [-1, 1]
)
def forward(self, obs):
return self.net(obs)
class Critic(nn.Module):
def __init__(self, total_obs_dim, total_act_dim, hidden_dim=64):
super(Critic, self).__init__()
# Input: concatenated observations of all agents + actions of all agents
self.net = nn.Sequential(
nn.Linear(total_obs_dim + total_act_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, 1)
)
def forward(self, obs_all, act_all):
x = torch.cat([obs_all, act_all], dim=-1)
return self.net(x)2. The MADDPG Trainer
The trainer manages the interaction between multiple agents and handles the centralized update logic.
class MADDPGAgent:
def __init__(self, agent_id, obs_dim, act_dim, total_obs_dim, total_act_dim):
self.agent_id = agent_id
self.actor = Actor(obs_dim, act_dim)
self.actor_target = Actor(obs_dim, act_dim)
self.actor_target.load_state_dict(self.actor.state_dict())
self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=1e-3)
self.critic = Critic(total_obs_dim, total_act_dim)
self.critic_target = Critic(total_obs_dim, total_act_dim)
self.critic_target.load_state_dict(self.critic.state_dict())
self.critic_optimizer = optim.Adam(self.critic.parameters(), lr=1e-3)
def act(self, obs, noise=0.1):
obs = torch.FloatTensor(obs).unsqueeze(0)
action = self.actor(obs).detach().numpy()[0]
action += np.random.normal(0, noise, size=action.shape)
return np.clip(action, -1, 1)
class MADDPGTrainer:
def __init__(self, n_agents, obs_dim, act_dim, gamma=0.95, tau=0.01):
self.n_agents = n_agents
self.gamma = gamma
self.tau = tau
total_obs_dim = n_agents * obs_dim
total_act_dim = n_agents * act_dim
self.agents = [MADDPGAgent(i, obs_dim, act_dim, total_obs_dim, total_act_dim) for i in range(n_agents)]
self.memory = deque(maxlen=10000)
def store(self, obs, act, rew, next_obs, done):
self.memory.append((obs, act, rew, next_obs, done))
def update(self, batch_size=64):
if len(self.memory) < batch_size: return
batch = random.sample(self.memory, batch_size)
obs = torch.FloatTensor(np.array([x[0] for x in batch]))
act = torch.FloatTensor(np.array([x[1] for x in batch]))
rew = torch.FloatTensor(np.array([x[2] for x in batch]))
next_obs = torch.FloatTensor(np.array([x[3] for x in batch]))
done = torch.FloatTensor(np.array([x[4] for x in batch])).unsqueeze(-1)
for i in range(self.n_agents):
agent = self.agents[i]
# --- Update Critic ---
with torch.no_grad():
next_act_all = torch.cat([self.agents[j].actor_target(next_obs[:, j, :]) for j in range(self.n_agents)], dim=-1)
target_q = rew[:, i].unsqueeze(-1) + self.gamma * (1 - done) * \
agent.critic_target(next_obs.view(batch_size, -1), next_act_all)
current_q = agent.critic(obs.view(batch_size, -1), act.view(batch_size, -1))
critic_loss = F.mse_loss(current_q, target_q)
agent.critic_optimizer.zero_grad(); critic_loss.backward(); agent.critic_optimizer.step()
# --- Update Actor ---
curr_act_all = [self.agents[j].actor(obs[:, j, :]) if j != i else agent.actor(obs[:, i, :]) for j in range(self.n_agents)]
curr_act_all = torch.cat(curr_act_all, dim=-1)
actor_loss = -agent.critic(obs.view(batch_size, -1), curr_act_all).mean()
agent.actor_optimizer.zero_grad(); actor_loss.backward(); agent.actor_optimizer.step()
# --- Soft Update Target Networks ---
for p, tp in zip(agent.actor.parameters(), agent.actor_target.parameters()):
tp.data.copy_(self.tau * p.data + (1 - self.tau) * tp.data)
for p, tp in zip(agent.critic.parameters(), agent.critic_target.parameters()):
tp.data.copy_(self.tau * p.data + (1 - self.tau) * tp.data)Summary & Key Takeaways
| Feature | Standard DDPG | MADDPG |
|---|---|---|
| Observation Space | Local state | Local (Actor) / Global (Critic) |
| Environment | Stationary | Non-stationary (Multi-agent) |
| Training | Independent | Centralized (CTDE) |
| Execution | Independent | Independent |
Final Pro-Tips for Deployment:
- Reward Shaping: In cooperative tasks, ensure rewards are aligned. In competitive tasks, zero-sum rewards work best.
- Exploration: Since MADDPG is deterministic, the
noiseadded during theact()phase is critical to prevent premature convergence. - Scaling: As the number of agents increases, the Critic's input space grows linearly. Consider using attention mechanisms (like MAAC) for very large swarms.