Football RL
A five-a-side football simulation where each team shares one policy network, trained with multi-agent PPO.
A five-a-side football simulation where all five players on a team share one neural network. The network chooses each player’s movement, whether to sprint, and what to do with the ball. There are no prescribed roles, formations, passing patterns or rewards for spacing; the objective is expected-goal difference.
The current policy is trained with multi-agent PPO and a centralised critic. The matches below were recorded with the Python engine against a scripted passing team, using the same seeds for each checkpoint.
recorded matches vs the passing benchmark
Background
I first started this project after watching the World Cup. As always in football, the extensive post-game analysis felt entirely outcome dependent; any move that eventually led to a goal is amazing attacking work and terrible, sloppy defense. Beyond individual brilliance, as both a player and spectator myself, it's often very difficult to work out quite why an opponent feels so difficult or easy to play against.
It's also interesting to wonder if a Fosbury Flop equivalent exists, i.e a completely different local minima for team shape and basic tactics. So, I decided to try building a reinforcement learning model for each player, with somewhat human constraints on movement, speed, passing, team awareness, etc.
The first version, in July, was trained by neuroevolution: each generation played a round-robin tournament, and the best networks were mutated to form the next. Its reward paid for possession and for moving the ball toward goal, and it learnt to carry the ball forward and shoot from close range.
Most of the work since then has gone into the simulation and into evaluation. Re-evaluated under the current rules, the last evolved champion loses to a simple scripted containment team by 0.75 xG per game, and disabling its passes improves it by 0.25.
Simulation
The pitch is 80 by 50 metres, with 40-second matches in 0.1 s steps. Goalkeepers are implicit: they appear only in the shot model and as a keeper zone in front of goal. Players are discs that collide and lose speed when obstructed, with limited acceleration, turning and stamina.
The ball has a physical trajectory, with drag and rolling resistance. Passes are aimed to lead a running receiver, and can be controlled during flight by anyone close enough to the ball’s path. Long passes are lofted. Offside positions are recorded at every kick, including shots, and enforced when the ball is received. Restarts place the ball and play it to the nearest player without moving anyone else.
where a pass can be contested
A shot’s outcome is drawn when it is taken, from a model based on distance, the angle of the goal mouth and how much of it the defenders screen. The ball then travels physically: goals cross the line, and misses are claimed by the keeper or go wide.
shot model with screening
drag any player- distance
- base
- mouth open
- xG
Policy
Each player runs the shared network on its own observation. The June version clamped the angle to players behind, as a field of view. That hid geometry a memoryless policy needs, so observations are now complete, and also include the clock, score, the player’s own facing and sprint state, and a one-hot identity. The identity allows players to take on different responsibilities without any being assigned.
player policy, 81 → 256 → 256 → 8
Objective
Returns are 12 times the expected-goal difference, using the same probabilities that decide the scoreline. This removes the noise of whether a particular chance went in while keeping the same target in expectation.
Shaping is strictly potential-based: each step adds γΦ(s′) − Φ(s), with Φ = 0 at full time, which in principle leaves the optimal policies unchanged (Ng et al., 1999). Φ includes a small possession and threat term, and a pitch-control term: the share of usable, onside space each team controls, weighted by closeness to the goal it attacks. The pitch-control overlay in fig. 1 shows this term. Without it, the team was about as strong, but much more crowded.
Evaluation
Beating random networks says little, so policies are evaluated against scripted opponents with transparent rules (a ball-pursuit team, a containment team and a passing team), against frozen earlier policies, and against the deployed default. Comparisons use paired games with shared seeds, both sides and both kickoffs, and report a standard error.
Intervention tests check whether coordination contributes to the results: disabling passes, freezing attacking support players, or replacing support or defensive movement with plain ball pursuit. A policy that passes often can still be made better by removing its passes.
Search-based training
Before MAPPO, the main approach was to generate training targets by search. From sampled game states, the simulation was cloned and every alternative (carrying, shooting, each pass, or one of nine movement directions) was played out to the end of the match with shared random numbers. An alternative was only taught if it then beat the current choice in 16 independent paired trials, by two standard errors. Selecting and confirming on the same samples had produced false improvements. Every 10 updates, proposed weights also had to win fresh full matches before they replaced the current policy.
This was careful but slow. The first change it accepted improved defending, and the resulting policy still made no passes. The search did find useful passes, but the network did not generalise from them: it reproduced 86% of the confirmed choices it was trained on and 15% on held-out batches. The deployed policy stayed at update 970, and the last fifteen reviews, up to update 1500, were all rejected.
Why the teams bunched
Every policy up to this point played as a narrow group around the ball. The first MAPPO runs learnt the same swarm, so I tested whether it was actually wrong. Against the learned defence, a scripted team that spaced out and passed scored less than the same team with swarming support, and forcing the learned policy to spread out cost it 0.08 to 0.46 xG per game. Under those rules, the swarm was close to optimal.
So the changes went into the game rather than the reward. Lofted passes, which can’t be intercepted in flight, gave width and depth a use. Watching recorded matches also turned up several engine bugs: missed shots could be collected as rebounds, some misses were labelled as going out, a lost keeper duel could look like a goal, and restarts teleported the nearest player up to 32 m onto the ball. Each was fixed in both the scalar and batched engines.
team shape against the passing benchmark
formation_metrics.py on fresh seeds, counting only the learned team. “Pair within 3 m” is
the share of frames where any two teammates are closer than 3 m; width is the lateral spread of the five players;
goal-side counts defenders between the ball and their own goal. Rows were measured at different times, each under
the ruleset then current.
MAPPO
Each update collects 2,048 parallel games of 64 ticks, about a million player-steps, which takes around 3.5 seconds on an RTX 3090. Both sides of self-play games learn. Half the games are self-play, a quarter are against frozen policies (league snapshots and the deployed default), and the rest are against the scripted teams. The centralised critic sees the whole game state in each team’s attacking frame, which gives a much less noisy signal than a critic limited to one player’s view. The exported policy is still an ordinary MLP.
Without extra exploration on the ball choice, PPO eliminated passing almost immediately. A target entropy for the carrier’s choice keeps it exploring.
pass rate during MAPPO training
Sprinting
Each player has a sprint: a 1.5-second burst at up to 10.5 m/s, followed by a 6-second cooldown. In the first version a sprinting carrier could dribble at 8 m/s, and the policy learnt only to sprint past opponents that never sprint, which made it worse against learned opponents. Carriers are now capped at the normal dribbling speed.
The current policy sprints about twice a match, almost always as a player who has just received the ball. Sprinting off the ball, and passes to sprinting teammates, have not appeared.
Results
current policy, xG margin per game (±2 SE)
Against the passing benchmark, the current policy has two teammates within 3 m in 29% of frames when attacking, compared with 82% for the pre-MAPPO default. It is 15 m wide in attack and 17 m in defence, has on average 3.3 of 5 players goal-side when defending, and makes about three passes a game, of which 1.2 are lofted.
The attack still gathers around the ball, passing is infrequent, and through balls and off-ball runs have not emerged. Both probably need coordinated discovery: a run is only worth making if someone plays the pass, and the pass only exists if someone makes the run.