54.8 %
Win rate (±1.0, 95 % CI)
Reinforcement learning · Playable
Flip7: a PPO agent you can play against
I turned the card game Flip7 into a reinforcement learning environment, trained a PPO agent against a near-optimal dynamic-programming opponent, and shipped a web app where anyone can challenge the trained model.
01 / Overview
Motivation
Flip7 is a fast push-your-luck card game: on every turn, a player decides whether to draw another card or stop and bank the points. The decision is simple, but the right choice depends on the score, the remaining deck, and a handful of special cards — a good setting for reinforcement learning.
I turned Flip7 into a compact PPO benchmark: a small action space, an informative state representation, a strong opponent built with dynamic programming, and a reward that targets winning the whole game rather than single rounds.
02 / Architecture
State and action space
The action space is intentionally minimal. The agent makes two kinds of binary decisions:
- Draw or stay: take another card, or stop and bank the current round score.
- Target of a special card: when an action card is drawn, decide which player receives it.
2 binary decisions
14 state features
normalized inputs
Observation Vector
To make these decisions meaningful, the environment exposes a 14-dimensional observation vector, normalized to the range [0, 1]. The state tracks:
- Round context: agent round score, number of unique cards, opponent round score, opponent unique cards, and a flag for whether either player is out of the round.
- Game context: total game scores for agent and opponent.
- Deck & Specials: remaining deck size, x2 and Second Chance status for both players, and frozen flags.
Normalizing all features keeps neural network inputs bounded, which stabilizes PPO training and accelerates convergence.
03 / Rules
Deck construction and scoring
The number cards are skewed toward higher values: the deck contains one 0 and one 1, and every value from 2 to 12 appears as many times as its value (e.g. three 3s, twelve 12s). High cards are therefore more common, which raises the upside of staying in a round longer.
On top of this, the environment adds special cards: +2, +4, +6, +8, +10, x2, Freeze, Second Chance, and Flip3. Scoring follows four steps:
- Sum all number cards.
- If the player has x2, double this base sum.
- Add bonus points from the +2 / +4 / +6 / +8 / +10 cards.
- If at least seven unique cards are collected, add a fixed Flip7 bonus on top.
04 / Variance Management
Risk, busting, and special cards
Flip7 is about managing risk under uncertainty. A player busts, losing the round score, when a duplicate number card is drawn. A Second Chance card absorbs the first duplicate instead.
- Bonus cards: Add fixed points and make banking the score more attractive.
- x2: Doubles all current and future number cards, which makes drawing more valuable.
- Freeze: Locks in the current round score and removes the player from further draws.
- Flip3: Forces three additional draws and can trigger chain reactions.
Flip3 is intentionally volatile. It makes the environment less predictable and forces the PPO policy to handle rare but impactful events instead of overfitting to average-case trajectories.
05 / Opponent Design
A dynamic-programming opponent
Instead of training against a random or heuristic player, the PPO agent faces a policy derived from dynamic programming. The environment compresses the remaining deck into a compact state and calls a DP solver that chooses near-optimal moves.
Turns alternate between the agent and the opponent; players who are frozen, have stayed, or have busted are skipped.
DP-based policy
state compression
non-trivial baseline
06 / Training Design
Why this works well for PPO
Several design choices make Flip7 a natural fit for PPO and similar policy-gradient methods:
- Small action space: Binary decisions keep the policy head simple and reduce exploration noise.
- Informative, normalized state: All relevant context in a stable numeric range.
- Strong opponent: A DP-based policy forces the agent to develop robust strategies.
- Long-horizon objective: Rewarding game wins pushes PPO to optimize long-term expected value under risk.
Game Objective and Rewards
Long-horizon optimization
Flip7 is played as a multi-round game up to a target score (e.g. 200 points). If the PPO agent wins the game, it receives a large terminal reward, and the episode terminates.
This ties success directly to winning the full game. The agent has to learn when to push for more points, when to protect a lead, and how to use special cards to manage variance across many rounds.
07 / Results
How good is the agent?
Evaluated over 10,000 full games against the dynamic-programming opponent. As a sanity check, the DP policy was also played against itself (mirror match), which should and does come out at roughly 50 %.
PPO vs. DP opponent
49.8 %
DP vs. DP mirror match
172
PPO points per game (avg.)
185
DP points per game (avg.)
Reading the numbers
- A small but real edge. 54.8 % against 50 % over 10,000 games is almost ten standard errors above chance. The mirror match at 49.8 % confirms the evaluation is fair.
- It plays for the win, not for points. The agent scores fewer points per game than the DP policy (172 vs. 185) yet wins more often: it takes more risk when behind and stays early when a round win clinches the game, which the score-blind DP policy cannot do.
- Browser version is the same model. The weights in the playable app are equivalent to the evaluated checkpoint.