Policy Gradient Methods in Reinforcement Learning

 

Policy Gradient Methods in Reinforcement Learning

Policy Gradient Methods are a class of model-free reinforcement learning (RL) algorithms that optimize the policy directly by adjusting its parameters based on the gradients of the expected cumulative reward with respect to the policy parameters. These methods are particularly useful in environments with continuous action spaces or where value-based methods (like Q-learning) struggle or fail to perform well.

In contrast to value-based methods (like Q-learning and SARSA), which learn a value function (such as the Q-function) to estimate the expected reward for state-action pairs, policy gradient methods directly learn the policy itself, which maps states to actions.

---

Key Concepts of Policy Gradient Methods

1. Policy (π):

   • A policy $\pi(a|s, \theta)$ defines the agent's behavior by mapping each state $s$ to a probability distribution over possible actions $a$. The policy can be deterministic or stochastic.
   • The policy is parameterized by $\theta$, which are the parameters (weights) of the model that need to be learned. The goal of policy gradient methods is to find the optimal parameters $\theta^*$ that maximize the expected cumulative reward.

2. Objective Function (Expected Reward):

   • The agent's goal is to maximize the expected cumulative reward over time. This is usually represented as: $$J(\theta) = \mathbb{E}_{\pi_\theta} \left[ \sum_{t=0}^{T} \gamma^t R_t \right]$$ Where:
     • $\mathbb{E}_{\pi_\theta}$ denotes the expected value under the policy $\pi_\theta$.
     • $\gamma$ is the discount factor.
     • $R_t$ is the reward received at time $t$.

3. Gradient Ascent:

   • Since we want to maximize the expected reward, we use gradient ascent to update the policy parameters $\theta$ in the direction that increases the objective function. The policy gradient is the gradient of the expected reward with respect to the policy parameters: $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta} \left[ \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot G_t \right]$$ Where:
     • $\nabla_\theta \log \pi_\theta(a_t|s_t)$ is the score function (or log likelihood), which is the gradient of the log of the policy with respect to its parameters.
     • $G_t$ is the return or discounted cumulative reward starting from time step $t$.

   The return $G_t$ can be computed in different ways, such as using Monte Carlo methods or temporal difference (TD) learning.

4. Stochastic Gradient Ascent:

   • The policy gradient is typically estimated through Monte Carlo sampling or bootstrapping methods. Since computing the exact expectation is often not feasible, we approximate the gradient using samples from the environment.

---

Policy Gradient Theorem

The Policy Gradient Theorem provides the mathematical foundation for policy gradient methods. It states that the gradient of the expected return with respect to the policy parameters can be expressed as:

$$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta} \left[ \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot G_t \right]$$

This theorem allows us to adjust the parameters $\theta$ of the policy in the direction of the gradient, thereby improving the policy.

---

Key Types of Policy Gradient Methods

1. REINFORCE (Monte Carlo Policy Gradient)

   REINFORCE is one of the most basic policy gradient algorithms. It uses Monte Carlo sampling to estimate the gradient of the objective function.

   • In REINFORCE, the agent interacts with the environment and collects episodes of state-action-reward trajectories. After completing an episode, the agent updates its policy based on the return of the episode.
   • The update rule for REINFORCE is: $$\theta_{t+1} = \theta_t + \alpha \cdot \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot G_t$$ Where:
     • $\alpha$ is the learning rate.
     • $G_t$ is the total return from time step $t$.

   Advantages:

   • Simple and easy to implement.
   • Can work well in environments with large state or action spaces.

   Disadvantages:

   • High variance in the gradient estimates due to the use of Monte Carlo methods.
   • It may require many episodes to get an accurate estimate of the gradient, which can be computationally expensive.

2. Actor-Critic Methods

   Actor-Critic is a more sophisticated approach that combines value-based and policy-based methods. The actor is responsible for selecting actions based on the policy, while the critic estimates the value function (often the state-value function $V(s)$ or action-value function $Q(s, a)$) to guide the actor.

   • Actor: Learns and updates the policy directly using the gradient of the expected return.
   • Critic: Estimates the value function, which helps the actor reduce variance in the policy gradient updates.

   The advantage function $A(s_t, a_t)$ is used to estimate how much better or worse the chosen action was compared to the average action in that state. It is defined as:

   $$A(s_t, a_t) = G_t - V(s_t)$$

   Where:

   • $G_t$ is the return (cumulative reward) starting at time step $t$.
   • $V(s_t)$ is the value estimate for state $s_t$.

   The update rule for the actor is:

   $$\theta_{t+1} = \theta_t + \alpha \cdot \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot A(s_t, a_t)$$

   Advantages:

   • Reduces variance in policy gradient estimates by using the critic’s value estimates.
   • Typically more efficient than REINFORCE, as it does not require full episodes to update the policy.

   Disadvantages:

   • Requires both policy and value function estimations, which adds complexity.
   • Convergence may still be slow depending on the choice of value function and learning rate.

3. Proximal Policy Optimization (PPO)

   PPO is an advanced policy gradient method that aims to improve stability and performance by limiting the size of policy updates.

   • PPO uses clipped objective functions to restrict the change in the policy between updates. The idea is to avoid large updates that can destabilize training.
   • The objective function for PPO is designed to prevent the new policy from deviating too far from the old policy, which helps prevent policy collapse.

   The objective for PPO is:

   $$L^{\text{clip}}(\theta) = \mathbb{E}_t \left[ \min \left( \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)} A_t, \text{clip} \left( \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}, 1-\epsilon, 1+\epsilon \right) A_t \right) \right]$$

   Where:

   • $A_t$ is the advantage function.
   • $\epsilon$ is a small constant (e.g., 0.2), controlling the range of policy updates.

   Advantages:

   • More stable than methods like REINFORCE and Actor-Critic.
   • Often leads to better performance and faster convergence.

   Disadvantages:

   • More complex to implement and requires tuning of hyperparameters.

---

Summary of Policy Gradient Methods

• Policy gradient methods learn directly from the gradients of the expected reward, adjusting the policy parameters to maximize cumulative reward.
• REINFORCE is a simple policy gradient method that uses Monte Carlo sampling to estimate the gradients, but suffers from high variance.
• Actor-Critic methods combine policy gradient with value function estimation to reduce variance in updates and improve performance.
• Proximal Policy Optimization (PPO) is a more advanced algorithm that improves stability and efficiency by clipping the policy updates to prevent large, destabilizing changes.

These methods are useful for a variety of RL problems, especially those with large or continuous action spaces where traditional Q-learning or value-based methods struggle to perform well.