Sina's Blog

CS and AI

3 min readAIProgramming

Two DDPG Agents Learning to Play Tennis

After the Banana Collector I moved on to Unity’s Tennis environment, which is harder in two ways. The actions are continuous, so DQN’s trick of taking the max over four actions doesn’t work anymore. And there are two agents, each one sitting inside the other’s environment.

Code is in the repo. This took me far longer than the other two, and almost all of that was spent on things that didn’t work.

The environment

Each agent sees 24 numbers and controls two continuous ones: move toward or away from the net, and jump. A hit over the net pays +0.1. Dropping the ball or putting it out costs −0.01. Solved means the better of the two agents averages +0.5 over 100 episodes.

The rewards are what make it awkward. Both agents get paid for keeping a rally going, so neither of them scores anything until both have some idea what they’re doing. For a long stretch at the start, every episode is over in two or three frames and nearly every reward is zero.

Why DDPG

DDPG takes the parts of DQN that work — replay buffer, target networks — and makes them work with continuous actions. Instead of scoring every possible action and picking the best one, you keep two networks. The actor takes a state and outputs an action directly. The critic takes the state and that action and says how good it is. Then you train the actor to produce actions the critic rates highly.

Both of mine are six fully-connected layers. The actor goes 400 → 300 → 200 → 100 → 50 with ReLU, then batch norm, then a tanh output sized to the action space — tanh because the actions are bounded. The critic starts with the state through 400 units, mixes the action in at the second layer, then narrows through 200 → 100 → 50 to a single number.

Settings:

  • Replay buffer: 1e6
  • Batch size: 1024
  • Gamma: 0.99
  • Tau: 1e-3
  • Actor LR: 1e-4, critic LR: 1e-3
  • Weight decay: 0, optimizer AdamW
  • Exploration noise: Gaussian

The part that didn’t work

My first attempt had three layers in each network and used an Ornstein-Uhlenbeck process for exploration noise, which is what the DDPG paper does. It ran for more than 20,000 episodes and never got near the target.

I made the networks deeper. That helped a bit, and still didn’t solve it after another 20,000 or so. What finally worked was three changes at once: swapping the OU noise for plain Gaussian noise, adding batch norm to the actor, and updating the networks every 10 episodes instead of continuously.

If you take one thing from this post, make it the noise. OU noise is correlated over time — it pushes the agent to keep going the way it was already going, which is great for a robot arm sweeping through a smooth path. In Tennis you want the agent reacting to a ball, and that correlation seems to get in the way. The funny part is that I hit the exact opposite on Reacher, where OU noise was the thing that made it converge. Same algorithm, opposite answer, depending on what the task actually looks like.

Results

cd scripts
python train_agent.py --unity-app Tennis.app --target-score 0.5

Episode 100 Average Score: 0.00400
Episode 200 Average Score: 0.00780
Episode 300 Average Score: 0.00600
Episode 400 Average Score: 0.02550
Episode 500 Average Score: 0.04760
Episode 600 Average Score: 0.12620
Episode 700 Average Score: 0.31120

Environment solved in 722 episodes! Average Score: 0.50260

Average score per episode, flat near zero for 400 episodes then rising sharply to cross 0.5 at episode 722.

Put that curve next to the Banana Collector’s and you can see the rally problem in the data. It sits near zero for 400 episodes, then bends up hard. Nothing pays until both agents can return a ball; once they can, every extra exchange in a rally adds up.

Trained versus untrained is a clear gap:

[>] Try untrained Tennis agents.
[-] Score: 0.2700000088661909
[>] Try a trained DDPG agent to play tennis.
[-] Score: 0.7650000127032399

I recorded the two side by side:

Untrained agents, then the trained pair keeping a rally going.

If I go back to it, the next thing to try is D4PG. The mlagents scripts came from Udacity’s deep-reinforcement-learning repo (MIT).

Related posts