Two DDPG Agents Learning to Play Tennis

After the Banana Collector, the next environment was Unity Tennis. Tennis is more difficult for two reasons:

• The actions are continuous. DQN selects the maximum value from a small set of actions. With continuous actions, this method does not work.
• There are two agents. Each agent is a part of the environment of the other agent.

The code is in the repo. This project took much more time than the other two projects. Most of that time was on methods that did not work.

The environment¶

Each agent gets 24 numbers and controls two continuous values:

• Movement toward the net or away from the net.
• Jump.

The rewards are:

• +0.1 when the agent hits the ball over the net.
• −0.01 when the ball falls to the ground or goes out.

The task is complete when the better agent has an average score of +0.5 or more for 100 episodes.

The rewards make the task difficult. Both agents get a reward only when the rally continues. Thus, an agent gets no score until both agents have some skill. At the start of training, each episode stops after two or three frames, and almost all rewards are zero.

Why DDPG¶

DDPG uses the replay buffer and the target networks from DQN, and it applies them to continuous actions. DQN calculates a value for each possible action and selects the best action. DDPG uses two networks:

• The actor. It gets a state and gives an action directly.
• The critic. It gets the state and the action, and it gives a value for that action.

The training changes the actor to give actions that get a high value from the critic.

Each network has six fully-connected layers:

• Actor: 400 → 300 → 200 → 100 → 50 units with ReLU, then batch normalization, then a tanh output with the size of the action space. The tanh function keeps the actions in their limits.
• Critic: The state goes through 400 units. The action joins at the second layer. Then the layers become smaller (200 → 100 → 50) and give one value.

The settings are:

• Replay buffer: 1e6
• Batch size: 1024
• Gamma: 0.99
• Tau: 1e-3
• Actor LR: 1e-4, critic LR: 1e-3
• Weight decay: 0, optimizer AdamW
• Exploration noise: Gaussian

The methods that did not work¶

The first version had three layers in each network. It used an Ornstein-Uhlenbeck (OU) process for exploration noise, as in the DDPG paper. It ran for more than 20,000 episodes and did not get near the target.

Deeper networks made the result a little better. But after approximately 20,000 more episodes, the task was still not complete. Three changes together gave a solution:

1. Use Gaussian noise, not OU noise.
2. Add batch normalization to the actor.
3. Update the networks every 10 episodes, not at each step.

The noise is the most important change. OU noise has a correlation in time. It pushes the agent to continue in the same direction. This is good for a robot arm that moves along a smooth path. In Tennis, the agent must react to the ball, and the correlation seems to prevent this.

On Reacher, the result was the opposite: OU noise made the training converge. The algorithm is the same, but the correct noise is different. The correct noise depends on the type of task.

Results¶

cd scripts
python train_agent.py --unity-app Tennis.app --target-score 0.5

Episode 100 Average Score: 0.00400
Episode 200 Average Score: 0.00780
Episode 300 Average Score: 0.00600
Episode 400 Average Score: 0.02550
Episode 500 Average Score: 0.04760
Episode 600 Average Score: 0.12620
Episode 700 Average Score: 0.31120

Environment solved in 722 episodes! Average Score: 0.50260

Compare this curve with the Banana Collector curve. The score stays near zero for 400 episodes, then it increases quickly. The agents get no reward until both agents can hit the ball back. After that, each additional hit in a rally increases the score.

The trained agents are much better than the untrained agents:

[>] Try untrained Tennis agents.
[-] Score: 0.2700000088661909
[>] Try a trained DDPG agent to play tennis.
[-] Score: 0.7650000127032399

This video shows the two cases:

Links¶

• Repo: github.com/sina5/tennis
• Report — full architecture and hyperparameters
• Continuous Control with Deep Reinforcement Learning

The next method to try is D4PG. The mlagents scripts come from the Udacity deep-reinforcement-learning repo (MIT).