Sina's Blog

CS and AI

3 min readAIProgramming

DDPG on Reacher: Twenty Arms Chasing a Balloon

Reacher is the environment I had the easiest time with, and the one that taught me the most — because it contradicted what I’d learned on Tennis a year earlier.

The setup: twenty double-jointed robot arms, each with a target balloon floating around it. The arm gets rewarded for every step its hand is inside the target. That’s the whole task — keep your hand on the balloon while the balloon moves.

Code is in the repo.

The environment

Each arm sees 33 numbers — position, rotation, velocity, angular velocity — and controls 4 continuous ones, the torques on its two joints. Everything is continuous, so this is DDPG territory rather than DQN.

Solved means averaging +30 over 100 consecutive episodes, across all twenty arms.

The twenty arms are the interesting part. They’re not competing and they’re not cooperating; they’re twenty copies of the same problem running at once, all feeding one shared replay buffer. You get twenty times the experience per wall-clock second, and the batches you sample are more varied because they come from twenty arms in twenty different situations. That alone does a lot of work.

The model

Same shape as the Tennis agents: an actor that maps a state straight to an action, and a critic that scores the pair, both six fully-connected layers. The actor runs 400 → 300 → 200 → 100 → 50 with ReLU and a tanh output for the bounded torques. The critic takes the state through 400 units, folds the action in at the second layer, then narrows to a single value.

  • Replay buffer: 1e6
  • Batch size: 1024
  • Gamma: 0.99
  • Tau: 1e-3
  • Actor LR: 1e-4, critic LR: 1e-3
  • Weight decay: 0, optimizer AdamW
  • Exploration noise: Ornstein-Uhlenbeck

The noise, again

My first run used shallower three-layer networks and Gaussian exploration noise, and it didn’t reach the target in 2,000 episodes. Deeper networks helped a little — still nothing after another thousand.

Switching the noise to an Ornstein-Uhlenbeck process solved it in 101 episodes.

That’s the same knob that broke Tennis, turned the other way. On Tennis, OU noise was what kept it from converging and Gaussian noise fixed it; here it’s exactly reversed. Once you see it laid out, it makes sense: OU noise is correlated across timesteps, so it nudges the agent to keep pushing in the direction it was already going. A robot arm sweeping toward a target wants that — the useful behavior is a smooth continuous motion. An agent waiting to react to a tennis ball doesn’t.

I don’t think there’s a rule to extract here beyond “try both.” But it’s a good reminder that the paper’s defaults are defaults for the paper’s environment.

Results

cd scripts
python train_agent.py --unity-app Reacher.app --target-score 30

Episode 95  Average Score: 29.26
Episode 96  Average Score: 29.35
Episode 97  Average Score: 29.43
Episode 98  Average Score: 29.52
Episode 99  Average Score: 29.61
Episode 100 Average Score: 29.69
Episode 101 Average Score: 30.08

Environment solved in 101 episodes! Average Score: 30.08

Average score per episode, rising steeply in the first 30 episodes then flattening just above 30.

101 episodes, and the curve is nothing like the other two. It shoots up in the first thirty episodes and then flattens out near the ceiling, mostly waiting for the 100-episode window to catch up with how good the agent already is. Reacher gives you a reward on nearly every step, so there’s no sparse-reward valley to climb out of like there was in Tennis.

Here’s trained versus untrained — the untrained arms flail, the trained ones lock on and track:

Twenty untrained arms, then twenty trained ones tracking their targets.

D4PG is the obvious next step here too. The ddpg and mlagents scripts started from Udacity’s deep-reinforcement-learning repo (MIT).

Related posts