Background
A staged rollout on Google Play is easy to describe and tedious to run. You push a release to 5% of users and watch the crash numbers. If nothing looks wrong, you bump it: 10%, 15%, up to 100% over several days. At Branch, this meant a release manager opening dashboards and touching the Play console two or three times a day, for seven to twelve days per release, posting six to eight status updates by hand along the way.
Our team spans several continents, so every bump had to wait for whoever was awake at the time. It was normal to say things like “Too late for me, let’s wait until tomorrow” in the middle of a rollout. Moreover, the calls were subjective. One person would say “stability looks fine, bump it a bit” while another person would say “let’s watch it a bit longer”. The same numbers produced different decisions depending on who was looking at them and what kind of day they were having.
The easy part took two days
RAS (Rollout Automation System) started as a two-day build at an internal hackathon. It was written as a set of Python scripts that read the current rollout state from the Google Play API, read a stability score from our crash reporter (the share of sessions that end without a crash or other unexpected errors) and printed a suggestion of increment, hold or halt. The design doc named it the Rollout Automation System, RAS. Not a lot of thought or time went into naming it during the hackathon rush.
Fast forward to today. Nothing about that core has changed since it was first written. The thresholds have been tuned and the guardrails around it have grown for months, but the decision logic running against production today is the logic written that week. The rest of this post is about the scaffolding, which took far longer than the core.
The architecture is small enough to draw as a single loop:

The loop starts at a configuration sheet/ledger listing the releases the bot is allowed to manage. A scheduled job (a GitHub Actions workflow on a timer) wakes every two hours, reads that sheet, calls the Play API and the crash reporter for each active release, decides what to do. It then posts a message to a chat thread and writes the new state back to the sheet, where the next cycle picks it up.
The trust ladder
We did not flip a switch and walk away. Autonomy came in rungs. We only climbed one when the last one had stopped surprising us.

Shadow mode: watching real releases in dry run
The first rung was a sandbox. For months, the bot ran against real releases in dry-run mode. It could see everything but it was not allowed to change anything. Each cycle, it posted what it would have done to a test channel. We compared those suggestions against what the release managers actually did.
After tuning the configuration for poll duration and increment percentages for a few releases, the bot slowly started to make suggestions that made sense. During the dry run, a release manager would often manually bump a rollout to, say, 30 percent. Minutes earlier, independently, the bot would have suggested exactly 30 percent. A human and a script arriving at the same number, without talking to each other, was the first sign that we could trust the bot.
Suggestion mode: posting in real threads
Months later, the bot moved out of the sandbox and into public release-announcement threads. Under each announcement it posted its read of the rollout: the current percentage, the stability score and what it would do next, visible to the whole team.

We wrote an experiment protocol to make the trust measurable. Release managers would follow the bot’s suggestions exactly and flag anything that looked unreasonable. If it judged stability well across a run of releases, it would earn write access. Along the way we found and fixed some ordinary bugs: a duplicate-row timestamp problem, a handful of missed notifications.
Most of the configuration tuning was around setting a velocity for the incremental rollouts. This is because velocity was where the human instincts mattered the most. We interviewed the release managers and encoded what they said as config. For routine releases, eight to ten hours between bumps felt right to the team, so eight to ten hours became the interval.
The team also proposed variable pacing. One way to do this would be “fast steps while exposure was low and gentler ones near the top”. To avoid overcomplicating the algorithm, we settled on one step size and one interval, both constant. The team worked out a constant schedule that emulated the general instinct, bigger steps over longer gaps. When a suggestion felt too fast, the human said so and the config changed. Every value in the sheet began as something a release manager suggested in an interview. Eventually, the final config became a set of interviews, frozen into numbers.
The proof: a dip nobody had to fix
The turning point was a rollout where stability dipped below the increment threshold. Two old crashes had resurfaced and a backend incident was underway. Cycle after cycle, the bot posted hold-steady messages while the humans dug into root cause. The humans reached the same conclusion the bot’s thresholds had already enforced - wait, no patch needed. When stability recovered, the bot signaled resume on the same day the humans concluded the same thing.

The bot had no write access. It was in dry-run mode. But it had matched the human incident response step for step, from hold to resume. That was the point where it stopped feeling like a toy.
Live mode: two lines and a humble kill switch
After months of shadowing and suggesting, a two-line change turned suggestion mode off. RAS could write. Routine bumps stopped waiting for someone to be awake.
Two choices at go-live are worth calling out. First, we made the rollout slower, not faster. Earlier testing had used eight to ten hour intervals. For autonomous operation we fixed 15-percent steps every 16 hours for every production release, roughly four days from 5 percent to full. We made the switch conditional on everyone agreeing to the number. Handing over the wheel called for more caution, not less. Second, the kill switch stayed simple to the point of being boring. Removing a release from the bot’s care means deleting its row in the config sheet. Registration stayed manual too, so the bot never picks up a release by itself unless a human hands one off to it.
Automate the skepticism before the decision
The feature that made live mode possible left the decision logic untouched. What it added was a check on when that logic is allowed to run at all.
Running the bot in live mode exposed a soft failure mode early. The bot once recommended a large increment on the strength of a perfect 100.0 percent stability number, computed over only about a hundred sessions. A stability percentage without a denominator is noise. At our scale, a hundred sessions is nowhere near enough to trust a figure to one decimal place.
The fix was a session-count minimum that scales with exposure. Each release has to accumulate a minimum number of sessions, proportional to its current rollout percentage, before its stability number counts for anything. The linear scaling is the point: a rollout reaching more users should clear a proportionally higher bar. As an illustration only, if we required N sessions at full rollout, a 5 percent rollout would need about N/20. We derived the actual floor from historical release-adoption data and validated it against a past release, rather than picking a round number that felt safe.
With the check in place, the decision core is small enough to fit on a napkin:

The two thresholds carve out three bands. When stability drops below 99.5 percent, the bot halts the rollout. Above 99.8 percent, once the session count clears the bar, it increments. The space between is a deliberate safe zone where the correct move is to wait and gather more data. Handled errors never count against the score because an error the app caught and recovered from is a session behaving as designed. The minimum session count check guards the increment. Automate the skepticism before you automate the decision.
One more asymmetry is worth stating here. The bot treats the halting action as a one-way door. It will halt on its own the moment stability crosses the line, at any hour, without waiting for permission. However, it will never resume on its own. A halted release means something was wrong enough that a machine chose to stop. Deciding when that is over stays with humans. To date, the bot has had to do this once, during a brief outage. The rollout stopped and stayed stopped until a person examined the situation and chose to resume.

Live mode reality
Production had a few lessons no documentation mentioned.
Three days into live mode, the bot failed to finish its first rollout. The Play API rejects setting a rollout to 100 percent. To complete a release, you flip its status to “completed” instead. Nothing we had read in the documentation indicated this.
Later, a halt attempt failed because the API requires the current user fraction in the payload even when you are halting.
The first week also produced a save. The bot placed a hold on a fresh release because stability was trending down, a few hours before humans confirmed a crash and started a patch. The stability check held a bad release at low exposure while the people caught up to what the numbers were already saying. This was a real lifesaver!
There was an etiquette problem too. The bot started by posting every decision to the public channel, including the many cycles where it decided to do nothing. The team ran an emoji poll and voted to broadcast only increments and halts publicly.
These days an autonomous increment draws no reactions at all. That was the goal. The strongest praise a successful automation gets is eventual silence.
What we would tell other teams
The decision logic was a hackathon’s worth of work. It has not changed since it was first written. Everything else, the shadowing, the suggestions, the pacing negotiations, the session check, the boring kill switch and the etiquette vote, was about trust. Trust builds up slowly. We let the automated system watch, then suggest, then act. We checked its work at every rung before we let it climb to the next.
If you are automating something that scares you a little, that is the trade-off to plan for. You can write the core in a weekend. Budget weeks or months for building trust.
Disclaimer: All personally identifiable information (PII) used in the experiments described in this post was anonymised before analysis. No raw PII is included in the results, examples, or diagrams shared here.
Comments
Loading comments…