Reinforcement Learning & Decision Making • 11 Aug 2026

Solving Partial Observability: A Deep Dive into Deep Recurrent Q-Networks (DRQN)

#Deep Reinforcement Learning #Deep Q-Network #DRQN #POMDP #LSTM #Recurrent Neural Networks #Atari 2600 #Partial Observability

Solving Partial Observability: A Deep Dive into Deep Recurrent Q-Networks (DRQN)

In the world of Reinforcement Learning (RL), we often assume the agent has a perfect view of the world. This is known as a Markov Decision Process (MDP). But in the real world—and in many complex games—the agent rarely sees everything.

Imagine playing a game where a ball disappears behind a wall. If you only look at the current frame, the ball is gone. To know where it is, you need to remember where it was and how fast it was moving. This is a Partially Observable Markov Decision Process (POMDP).

While the legendary Deep Q-Network (DQN) tried to solve this by stacking a few recent frames, there is a more elegant solution: Deep Recurrent Q-Networks (DRQN).

The Core Intuition: From Snapshots to Memory

The standard DQN uses a "sliding window" of frames (usually 4) to infer motion. However, this approach has two major flaws:

  1. Fixed Memory: It can only "remember" as far back as the window size.
  2. Inefficiency: It increases the input dimensionality significantly.

DRQN solves this by replacing the first fully connected layer after the convolutional layers with a Long Short-Term Memory (LSTM) module. Instead of looking at a fixed window, the network maintains an internal hidden state. This state acts as a working memory, integrating information over an arbitrary number of previous timesteps to estimate the true underlying state of the environment.

High-Level Architecture

flowchart TD subgraph Input_Stage ["Input Stage"] Input["Observation Sequence (batch, seq_len, C, H, W)"] end subgraph Feature_Extraction ["Spatial Feature Extraction (CNN)"] Conv1["Conv2d (8x8, stride 4) + ReLU"] Conv2["Conv2d (4x4, stride 2) + ReLU"] Conv3["Conv2d (3x3, stride 1) + ReLU"] Flatten["Flatten Layer"] Input --> Conv1 Conv1 --> Conv2 Conv2 --> Conv3 Conv3 --> Flatten end subgraph Temporal_Integration ["Temporal Integration (Recurrent)"] Reshape["Reshape to (batch, seq_len, features)"] LSTM["LSTM Module (Hidden State Memory)"] HiddenState["Hidden State (h_t, c_t)"] Flatten --> Reshape Reshape --> LSTM HiddenState <--> LSTM end subgraph Output_Stage ["Action Value Estimation"] FC["Linear Layer (Fully Connected)"] QValues["Q-Values (batch, seq_len, n_actions)"] LSTM --> FC FC --> QValues end QValues --> Action["Action Selection (argmax)"] style LSTM fill:#f9f,stroke:#333,stroke-width:2px style HiddenState fill:#fff4dd,stroke:#d4a017,stroke-dasharray: 5 5

The Mathematics of DRQN

DRQN retains the fundamental goal of Q-Learning: learning a function $Q(s, a)$ that predicts the expected future reward for taking action $a$ in state $s$.

1. The Bellman Equation

The network approximates the optimal Q-value using the Bellman equation: $$Q(s, a | \theta) \approx R_t + \gamma \max_{a'} Q(s', a' | \theta)$$

2. The Loss Function

To train the network, we minimize the Mean Squared Error (MSE) between the predicted Q-value and the target $y_i$: $$L(\theta) = \mathbb{E}_{s, a, r, s' \sim D} \left[ \left( y_i - Q(s, a | \theta) \right)^2 \right]$$

3. The Target Calculation

To maintain stability, a separate target network ($\theta^-$) is used to calculate the target value: $$y_i = r_t + \gamma \max_{a'} \hat{Q}(s_{t+1}, a' ; \theta^-)$$

Implementation Guide

Implementing DRQN requires a shift in how we handle data. Unlike DQN, where we sample random transitions, DRQN requires sequences of transitions to train the LSTM's memory.

PyTorch Implementation

PYTHON 
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import random
from collections import deque

class DRQN(nn.Module):
    def __init__(self, input_shape, n_actions):
        super(DRQN, self).__init__()
        # Spatial Feature Extraction
        self.conv = nn.Sequential(
            nn.Conv2d(input_shape[0], 32, kernel_size=8, stride=4),
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=4, stride=2),
            nn.ReLU(),
            nn.Conv2d(64, 64, kernel_size=3, stride=1),
            nn.ReLU(),
            nn.Flatten()
        )
        
        with torch.no_grad():
            dummy_input = torch.zeros(1, *input_shape)
            conv_out_size = self.conv(dummy_input).shape[1]
        
        # The Recurrent Layer: The "Memory" of the agent
        self.lstm = nn.LSTM(conv_out_size, 512, batch_first=True)
        self.fc = nn.Linear(512, n_actions)

    def forward(self, x, hidden=None):
        # x shape: (batch, seq_len, channels, h, w)
        batch_size, seq_len, c, h, w = x.shape
        
        # Process all frames in the sequence through CNN
        x = x.view(batch_size * seq_len, c, h, w)
        features = self.conv(x)
        
        # Reshape for LSTM: (batch, seq_len, features)
        features = features.view(batch_size, seq_len, -1)
        lstm_out, hidden = self.lstm(features, hidden)
        
        q_values = self.fc(lstm_out)
        return q_values, hidden

The Training Strategy: Bootstrapped Random Updates

Training a recurrent network on RL data is tricky. If you sample a random transition, the LSTM has no history. DRQN uses Bootstrapped Random Updates:

  1. Sample an Episode: Instead of a single transition, sample a full episode from memory.
  2. Random Slice: Pick a random starting point and unroll the network for a fixed number of steps (e.g., 10 steps).
  3. Zero State: Initialize the LSTM hidden state to zero at the start of the slice.
  4. BPTT: Perform Backpropagation Through Time (BPTT) over that sequence.

Summary Checklist: DQN vs. DRQN

Feature DQN DRQN
State Representation Frame Stacking (Fixed Window) Hidden State (LSTM)
Environment Type MDP (Fully Observable) POMDP (Partially Observable)
Memory Short-term / Fixed Long-term / Dynamic
Training Unit Single Transition $(s, a, r, s')$ Sequence of Transitions
Complexity Lower Higher (BPTT required)

Final Thoughts

DRQN represents a pivotal step in making RL agents more "human-like" by giving them a sense of time and history. By integrating LSTMs into the Q-learning framework, we enable agents to navigate environments where the present moment doesn't tell the whole story. Whether it's tracking a hidden object in a game or analyzing time-series data in finance, the ability to remember is the key to intelligent action.