Two DDPG Agents Learning to Play Tennis

After the Banana Collector, I moved on to Unity Tennis. It is harder for two reasons:

• The actions are smooth, not a menu. In the banana game the agent picked one of four buttons. Here it has to decide how much to move and jump. You can’t score every possible amount, so DQN no longer fits.
• There are two players. Each one is learning while the other is also learning, so the game keeps changing under their feet.

The code is in the repo. This project took far longer than the other two in the series, and most of that time went into things that didn’t work.

The environment¶

Each agent gets 24 numbers about the ball and its own racket. It controls two things:

• Moving toward the net or away from it.
• Jumping.

The rewards are:

• +0.1 when the agent hits the ball over the net.
• −0.01 when the ball hits the ground or goes out.

The task is solved when the better of the two agents averages +0.5 or more over 100 episodes.

Those rewards are what make this hard. Nobody scores unless the rally keeps going, so one agent can’t get anywhere until both have a little skill. At the start, every episode ends after two or three frames and almost every reward is zero. There is very little to learn from.

Why DDPG¶

DDPG borrows the replay buffer and the target networks from DQN and makes them work with smooth actions. Instead of one network that scores a list of choices, it uses two:

• The actor looks at the situation and says what to do.
• The critic looks at the situation and the actor’s move, and says how good that move was.

Training nudges the actor toward moves the critic rates highly. Think of a player and a coach who improve together.

Each network has six fully-connected layers:

• Actor: 400 → 300 → 200 → 100 → 50 units with ReLU, then batch normalization, then a tanh output with the size of the action space. The tanh keeps the actions inside their allowed range.
• Critic: The state goes through 400 units. The action joins at the second layer. Then the layers get smaller (200 → 100 → 50) and end in a single number.

The settings are:

• Replay buffer: 1e6
• Batch size: 1024
• Gamma: 0.99
• Tau: 1e-3
• Actor LR: 1e-4, critic LR: 1e-3
• Weight decay: 0, optimizer AdamW
• Exploration noise: Gaussian

What did not work¶

My first version had three layers in each network. To make the agents try new things, it added Ornstein-Uhlenbeck (OU) noise to their actions, exactly as the DDPG paper does. It ran for more than 20,000 episodes and never got close.

Deeper networks helped a little. After roughly 20,000 more episodes, the task still wasn’t solved. What finally worked was three changes together:

1. Use Gaussian noise instead of OU noise.
2. Add batch normalization to the actor.
3. Update the networks every 10 episodes instead of every step.

The noise mattered most. OU noise is “sticky”: if it pushed the agent left a moment ago, it will probably push left again. That is great for a robot arm following a smooth path. A tennis player has to react to the ball right now, and the stickiness seems to get in the way. Gaussian noise is a fresh random nudge every time.

On Reacher, I saw the exact opposite: OU noise was what made training work. Same algorithm, different task, different answer.

Results¶

cd scripts
python train_agent.py --unity-app Tennis.app --target-score 0.5

Episode 100 Average Score: 0.00400
Episode 200 Average Score: 0.00780
Episode 300 Average Score: 0.00600
Episode 400 Average Score: 0.02550
Episode 500 Average Score: 0.04760
Episode 600 Average Score: 0.12620
Episode 700 Average Score: 0.31120

Environment solved in 722 episodes! Average Score: 0.50260

Compare this curve with the steady climb in the Banana Collector post. Here the score sits near zero for 400 episodes and then takes off. Nothing happens until both agents can return the ball. Once they can, every extra hit in a rally adds to the score, and progress snowballs.

The trained agents clearly beat the untrained ones:

[>] Try untrained Tennis agents.
[-] Score: 0.2700000088661909
[>] Try a trained DDPG agent to play tennis.
[-] Score: 0.7650000127032399

This video shows both:

Links¶

• Repo: github.com/sina5/tennis
• Report — full architecture and hyperparameters
• Continuous Control with Deep Reinforcement Learning

The next method I want to try is D4PG. The mlagents scripts come from the Udacity deep-reinforcement-learning repo (MIT).