Reinforcement Learning and Sequential Decision-Making

Manual: General · Subject: Artificial Intelligence

Introduces Markov decision processes, dynamic programming, policy optimization, and modern RL algorithms.

Learning by Interaction

Sequential decision-making

Reinforcement learning studies agents that learn through interaction, balancing immediate rewards and long-term return. The canonical formalism is the Markov decision process, defined by states, actions, transitions, rewards, and discounting.

Value functions

The state-value function is $V^\pi(s)=\mathbb{E}_\pi\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right]$ , and the action-value function is $Q^\pi(s,a)$ . These quantities support policy evaluation and control.

RL algorithm families

Family	Example	Strength
Dynamic programming	Value iteration	Exact if model is known
Monte Carlo methods	Policy evaluation from episodes	Simple and model-free
Temporal-difference learning	Q-learning	Bootstraps from partial returns
Policy gradients	REINFORCE, actor-critic	Directly optimizes stochastic policies

⚠️

RL is data hungry

Many RL algorithms are sample-inefficient, sensitive to reward design, and unstable under function approximation; improving these properties is a central research challenge.

Which statement best describes the exploration-exploitation trade-off?

Always choose the current best-known action Balance trying uncertain actions against using known good ones Avoid all randomness Use only supervised labels

Why is off-policy learning useful?

Policy optimization styles

Value-based

Learn action values
Derive policy from values
Can struggle with continuous actions

Policy-based

Directly parameterize the policy
Works naturally in continuous spaces
Often higher gradient variance

Designing an RL experiment

1

Step 1: Define reward, termination, and discounting.
2

Step 2: Choose whether a model is available.
3

Step 3: Select an algorithm matched to action-space structure.
4

Step 4: Measure sample efficiency and stability.
5

Step 5: Evaluate robustness across seeds and environment variations.

Neuroanatomy Explorer

Memory Deck

Match the Pairs

Concept Constellation