How we give every youth-sports team its own native app, and keep it solid in the hands of people who aren't engineers. This blog was previously the Affirmi Engineering Blog; Affirmi is now TheMVP.ai.
How we test live youth-sports streaming end-to-end on real phones, real RTMPS, and real Mux, before a single line of code ships.
There's a Mac Mini named Paws. Plugged into its USB ports is an Android phone. Every 30 minutes, without anyone touching it, that phone opens our app, taps "Go Live," and pushes a real broadcast over RTMPS to live Mux production infrastructure. Then a script decodes the resulting HLS playlist and counts the frames. If the audio and video frames don't arrive, the test fails.
That is the whole philosophy in one sentence: a streaming test that doesn't push real bytes through a real radio to a real CDN is theater.
We have a clean vitest suite. It proves manage-stream accepts a valid request and returns JSON with an rtmps:// URL and a stream key. It is necessary. It is also useless against the failures that take a stream down at a Saturday game.
The real failure modes don't live in a Docker container with a mocked Supabase:
None of those are catchable without a phone, a camera, and a cell radio. So we built the rig. The Android Go Live test (android-go-live-adb.sh) is pure adb, no Maestro driver at all, because the Samsung's dev.mobile.maestro package kept drifting out of sync with the host. The sequence is blunt: sweep orphan events, launch the activity, tap Home, tap Go Live, poll the DB until events.mux_live_stream_id populates, poll Mux until idle → active, then run ffprobe to confirm HLS is delivering decoded video plus audio frames. Only then does it stop the stream and wait for stream_status=ended.
Alongside it, an hourly vitest (web/tests/integration/streaming-confidence.test.ts) spins up an ephemeral paid test org, creates a Mux stream through the real edge function, and uses ffmpeg to push testsrc2 at 1280x720@30fps plus a 1kHz sine tone for 35 seconds. It polls Mux for active, warms the HLS manifest over ~15 seconds, and verifies the audio track is genuinely there (mp4a in the CODECS line). It even stops the stream while ffmpeg is still pushing. That is the realistic "End Game" path where the broadcaster bails mid-stream.
A test that fails is not the dangerous case. The dangerous one fails while nobody is looking at the board. Leave the rule below unmade and an alert can sit red for 25 straight hours, screaming about a real streaming bug, while every eye slides past it because the tiles around it are red too, for reasons already known. When the whole board is red, you learn to skip it. A red light only works when red is rare.
So we made a rule, and wrote it into the codebase: "always red" is a forbidden state. Every alert on the infra dashboard has to be in exactly one of three buckets. Real-fire: actively being fixed, with an open task or PR, resolves in under a day or it escalates to a human's phone. Quarantined-with-expiry: explicitly muted, but only with a written until date and a one-line reason in a committed quarantines.json. The dashboard renders those yellow, not green, so they stay visible while reading distinct from live fires. After the expiry passes, the alert un-mutes itself and the next failure pages through. Dead-remove: the test no longer exists, so the alert source is deleted outright. There is no fourth bucket. A tile that's "just always red" is tech debt, and a Friday cron meta-alert counts how much of it has piled up.
None of this is specific to us. A monitoring system is not measured by how many checks it runs. It is measured by whether a single red light still means "drop what you're doing." The instant red becomes ambient, you're flying blind in a busy-looking cockpit.
A flaky rig that pages you at 2am every time the Samsung's WiFi hiccups trains you to ignore it. So the rig fixes itself first.
On the first pass→fail transition, the wrapper queues a paws_repair_queue row (kind=android_go_live_fail, context carries the serial, app ID, org ID, and the last error tail). A separate worker, paws-repair-worker.sh, runs every 2 minutes, claims one row, and dispatches a recovery action: restart adb (recover_adb_dead does a kill-server/start-server cycle and confirms the serial is back), reinstall the Maestro driver, or wait out a transient Mux blip. Exit 0 means recovered. Exit 2 means "rig is busy mid-test" and re-opens the row without incrementing attempts. That is smart backoff, so a live run never gets trampled. Only after two consecutive failed recoveries does Steven get an iMessage.
A device can lie about being reachable. One of our phones sat on dead WiFi for five weeks while adb devices reported it online. A network preflight now pings the Supabase host from inside the device shell and catches that on the next tick.
Take an Android run that comes back with mux_live_stream_id stuck at null. One SELECT on app_logs WHERE session_id=X puts the cause on screen: a stream:activated event with no stream:created before it. That is the sessionalizer at work. Three headers (x-affirmi-session-id, x-affirmi-flow-id, x-affirmi-flow-kind) stamped on every request let one query reconstruct tap → route → edge fn → Mux.
manage-stream reuses an existing live stream when the org already has one open, and that reuse is org-scoped. An iOS test fixture had left a stream active in the shared org, so the Android run bound to it instead of minting its own. A pre-run step force-completes any inherited active streams before the test begins. Two phones, one org, correlated state, read in minutes because the logs were already stitched.
You cannot mock a radio. You cannot mock Mux's 30-second ingest warmup. And a dashboard goes blind the moment red stops being rare.
If the flow is fragile and load-bearing (a live youth basketball game is exactly that), test it on the real metal, on a real cadence, and make it heal itself before it wakes you up. The phone on the desk is the most honest engineer we have.
We let agents write code. So we built a panel of four frontier models, one from each of four different labs, that has to sign off before anything merges.
A lot of our code is now written by AI agents. We say that on purpose. But it raises the obvious question most "AI writes our code" stories skip right past: who reviews it?
If the same kind of model that wrote the change also rubber-stamps it, you haven't built a review process. You've built a hype machine that approves its own homework. So we did the opposite. Every pull request an agent opens has to survive a panel of four frontier models from four different labs before GitHub will let it merge.
When an agent opens a PR, four judges spin up in parallel and review the diff independently, with no conferring and no shared scratchpad:
Four labs, four training runs, four different sets of blind spots. That diversity is the entire idea. A subtle bug that slips past one model's priors gets caught by another's. Redundancy is three people who think alike checking the same thing; diversity is four reviewers who fail differently. We wanted the second one.
Each judge returns a structured verdict, not a vibe. They are held to the same rules our human engineers are, plus a few written straight from our own production incidents:
catch {} is an automatic flag.The decision rule is deliberately not "majority wins," because not all dissent is equal.
Round one: all four judge independently. If everyone approves, it ships. If one model raises a minor nit while the rest approve, it still ships. We are not going to block a real fix over a style preference. But Claude Opus holds a veto: if it flags a genuine problem, the PR is blocked regardless of what the other three said. We'd rather eat a few false alarms than let a real bug through because it got outvoted.
Round two: if the judges materially disagree (one sees a real defect the others missed), they do not average it out. They go into a deliberation round, where each model now sees the others' reviews and re-decides with that context. Sometimes a judge realizes it was wrong. Sometimes it digs in and convinces the rest. Either way the disagreement gets resolved, not buried under a vote count.
Only when that whole process returns a clean decision, and the code compiles, and the real test suite passes against a live stack, does the merge bot turn green. Four required checks, all of them, every time.
A review gate carries an ordering trap. Let a PR auto-merge the moment the reviewers approve, before the test suite finishes, and "the review passed" and "the code is broken" can both be true at once, because the gate checked the wrong thing at the wrong moment. So nothing here auto-merges until the full panel decision and compile and the live test run are all green, enforced by GitHub branch protection rather than good intentions.
A review that finishes before the verification does is theater. It feels like a gate. It stops nothing.
You could argue this is overkill for a youth-sports app. It isn't, for one reason: the people relying on us aren't engineers. A coach trusts that the app with their name on it works. A parent two states away trusts that the live stream of their kid's game will actually play. When agents move fast, the only thing standing between that trust and a silent regression is the gate. So we made the gate real, made it diverse, and gave it teeth.
We let AI write the code. We just don't let it grade itself.
Story 02 explains what the four-judge gate is. This is the one people keep DMing me about: exactly how it's wired, down to the YAML, and the research that says a panel of four models from four labs beats any single reviewer. Including yours.
Every time I post about agents writing our code, someone shows up with "LLMs only produce slop." That take hit a new peak with the uproar over Paige's app, when the pile-on graduated from dunking on one app to declaring every junior dev who ships with AI "an incredible liability" who needs to learn engineering principles before touching a black box tool. Twenty years in, I think that's exactly backwards. Plenty of senior devs can't explain backpressure or CORS either, LLMs already out-review most of them on security, and new engineers are going to use these tools no matter how loud the gatekeeping gets. The useful move is teaching them how to ship secure, performant code with the tools, not demanding they learn every security flaw by hand first. A race condition checked into a serious codebase was never an AI problem. It's a review problem.
One reply informed me my org must have "dogshit PR/MR policies." Fine. Here are the policies: four frontier models from four different labs review every agent pull request, a deterministic script combines their verdicts, and GitHub will not merge until the panel, the compiler, and the live test suite all say yes. This post is the full recipe. Steal it.
Proof it's alive: OpenAI shipped GPT-5.6 on July 9. The next day an agent opened a PR swapping our GPT judge seat to gpt-5.6-sol-pro. The sitting panel (still running the old model) reviewed the upgrade that would replace one of its own members, approved it, and the merge bot shipped it. My involvement: reading the PR after it merged. The new seat is also 6× cheaper: $5/$30 per million tokens instead of $30/$180.
A model reviewing code carries the priors of the model that wrote it. That used to be intuition. Now it's measured. LLM evaluators recognize their own outputs and score them higher, and the better they self-recognize, the stronger the bias (Panickssery et al., NeurIPS 2024). When humans rate a GPT-4 answer unfavorably, GPT-4-as-judge still rates it favorably most of the time; unfavorable-recall drops to ~42.5% (Wataoka et al. 2024). And it goes past self-bias: judges over-score outputs from their own model family, even after you control for quality (2025). Our coding agents are Claude models. A Claude-only review gate would be a hype machine grading its own homework, with receipts.
The fix is old math wearing new clothes. A majority vote beats any individual member if and only if the members make different errors. That's Hansen & Salamon 1990, thirty-six years ago. Dietterich quantified it: 21 voters with independent 30% error rates vote their way to a 2.6% error, an 11× reduction, but only when the errors are independent (Dietterich 2000). Four copies of one model buys you almost nothing. Four different training runs is what buys error independence. Cohere proved the LLM version: a panel of judges from disjoint model families, combined by simple deterministic pooling, beat a single large GPT-4 judge across six datasets with less intra-model bias, 7× cheaper (Verga et al. 2024, "Replacing Judges with Juries"). That paper is the closest published prior art to what we run.
And the reason review works at all: verifying is easier than generating. Training a verifier to judge candidate answers was worth roughly a 30× model-size increase in one OpenAI measurement (Cobbe et al. 2021); models consistently critique better than they produce (Saunders et al. 2022); and RLHF-trained critics reviewing model-written code caught more bugs than paid human reviewers (CriticGPT, 2024). The gate operates exactly where models are strongest.
One workflow, quad-judge-pr.yml, triggered on pull_request (opened, synchronize, labeled, ready_for_review). Every job is gated on the PR carrying the agent label, so human PRs flow through normal review untouched.
1. The diff job. Checks out the PR head with fetch-depth: 0 and computes the diff from the merge base, never two-dot:
MB="$(git merge-base "origin/$BASE_REF" HEAD)"
git diff --unified=5 "$MB" HEAD > /tmp/pr.diffThen uploads pr.diff as an artifact. The merge-base part is scar tissue: a plain git diff base head on a stale swarm branch shows everything merged to main since the branch point as phantom deletions, and the judges (reading honestly) block it as a critical regression. They were right about the diff and wrong about the PR. Diff from the merge base and they only see what the PR actually changes.
2. Four judge jobs, in parallel. Here's the detail people miss: the judges check out the base branch, not the PR head. The only thing from the PR they ever see is the diff artifact. Two reasons. A malicious or buggy PR must not be able to edit the script that judges it. And a judge-prompt improvement on main instantly applies to every open PR. Each job runs one runner script with a judge name (python3 judge-runner.py gpt5pro 1) with only that provider's key in env.
3. One runner script, structured verdicts. Every judge gets the same charter prompt: the diff, the PR title and body (as claims to verify, never as evidence), and explicit lenses: correctness, security, data-contract consistency, migration additivity, silent-failure hunting, observability hooks. It must end its review with a fixed three-line block:
verdict: APPROVE | REQUEST_CHANGES
severity: none | nit | minor | major | critical
security: yes | noThe parser takes the last match in the response, tolerates markdown bold around values, and fails closed: no parseable verdict means REQUEST_CHANGES at severity major. An unparseable judge is a blocking judge, never an approving one.
4. Provider plumbing that survives reality. Reasoning models spend output budget on thinking. Give them a 32k ceiling or your three-line verdict gets eaten. The heavy OpenRouter calls run with stream: true because the dominant failure mode of big reasoning models is a non-streaming request stalling at zero bytes until your client timeout; streaming keeps keep-alives flowing and surfaces a stall at the first silent gap. Truncated stream (no [DONE], no finish_reason) = retry, then fail. Empty content = same. Rate limits get escalating backoff (we wait up to two minutes across six retries) because 429s arrive both as HTTP status and inside 200 bodies. And there is no model substitution: our code once silently swapped the pro model for its cheaper sibling when the heavy call flaked, which meant a "GPT-5.5 Pro" verdict sometimes wasn't. Now a dead model fails closed to SKIP, and a required judge SKIPping blocks the PR. Model IDs live in repo variables (GPT_JUDGE_MODEL, GEMINI_MODEL, ANTHROPIC_JUDGE_MODEL) so an upgrade or rollback is a settings flip with no code change. Keys live in repo secrets (ANTHROPIC_API_KEY, GEMINI_API_KEY, OPENROUTER_API_KEY).
5. The decision is a script, not a model. Each judge job exports verdict/severity/security through GITHUB_OUTPUT and posts its full reasoning as a PR comment. A decision job (also checked out from base) applies rules in order:
1. Claude != APPROVE -> BLOCK
2. any judge security == yes -> BLOCK
3. required judge SKIPped -> BLOCK (fail closed)
4. all approve -> SHIP
5. one dissenter, severity nit/minor -> SHIP
6. one dissenter, severity major/critical -> DELIBERATE
7. otherwise -> BLOCKThe combiner is ~100 lines of auditable Python. Aggregating with another LLM would reintroduce the exact judge-bias the panel exists to cancel.
6. Deliberation, once. On DELIBERATE, all four judges re-run with every Round-1 verdict and reasoning in context, told explicitly not to anchor on their own first vote. Then the same rules. There is no round three. The research backs both halves: exposure to other agents' reasoning corrects errors a lone model can't fix in itself (Du et al. 2023: GSM8K 77→85%; ChatEval: debating evaluators get more accurate, and diversity among them is essential; Khan et al. 2024: debate lifts judge accuracy 48→76%). But unbounded rounds curdle into conformity, and self-correction without an external signal actively degrades accuracy (Huang et al., ICLR 2024). One round, triggered only by real disagreement.
7. The merge bot. The merge job needs: [r1-decision, r2-decision, web-compile, web-test] with if: always() && ... plus explicit per-job result checks (r2 is legitimately skipped when Round 1 is unanimous; always() re-imposes the AND semantics that skipping breaks). Judges are not compilers: a merge-conflict resolution can produce a duplicate import that reads fine to every LLM and still breaks the build, so tsc, a production build, and the curated vitest suite run the PR's actual head, and a SHIP verdict cannot merge past a red one. An earlier two-judge design could clear a PR before its test run finished. This one cannot: nothing auto-merges until compile and tests come back green. Verdict green AND compile green AND tests green, enforced by branch protection, or nothing moves:
gh pr merge "$PR" --merge --auto --delete-branch8. The gate tests itself. A hermetic self-test suite (34 unit tests, no network, providers mocked) runs in CI on every PR: the decision rules, the fail-closed SSE parsing, the no-substitution guarantee, the model-resolution logic. When the GPT-5.6 swap PR went up, the panel demanded tests for the runner change before approving. The gate made its own upgrade earn its way in. That's when you know the thing is real.
Rubric over vibes, by the way, is also measured: decomposed, criterion-anchored evaluation beats holistic judging across the board (G-Eval, FLASK, CheckEval, TICK; checklists raised judge-human agreement +5.8 points and even raised human-human agreement). Our charter is binary, enumerable failure conditions written by our own production incidents. "Is every returned DB error checked," not "rate this code 1–10."
I'd rather publish the caveats than have someone else discover them for me. Frontier models' errors are converging: judges score similar models more favorably even after controlling for accuracy (Goel et al. 2025, "Great Models Think Alike and this Undermines AI Oversight"), and one 2026 measurement found a nine-judge cross-family panel collapsing to ~2.2 effective independent votes (Kohli 2026). So our 4/4-approve is worth roughly two independent checks, not four. That's exactly why the compilers, the live tests, and the human-attention events (any security block, any degraded panel, any post-deliberation block) stay load-bearing. LLM judges are also promptable-injectable in adversarial settings (~90% attack success in JudgeDeceiver). Ours review diffs from our own agents and run from the trusted base branch, which is the right shape of mitigation, but nobody should pretend this is solved. And security recall on subtle vulnerabilities is genuinely unproven (SecLLMHolmes, IEEE S&P 2024): the ANY-judge security veto deliberately buys recall at the cost of precision, and it still sits on top of RLS, auth reviews, and fail-closed platform guards.
What has it caught, concretely? An authorization bypass where an endpoint minted a billing-portal session for the first family record it found, meaning any visitor could reach another family's billing. Two check-then-act races (a waitlist position computed count-then-insert, and a webhook replay double-inserting membership rows). A privileged database function without a pinned search path. Any one of those, shipped, costs more than every judge invocation we will ever pay for. The whole panel runs low single-digit dollars per PR.
Here's the part I keep coming back to: nothing about this math is code-specific. Self-preference bias, ensemble error cancellation, structured deliberation: it applies to a marketing plan or a contract summary the same way it applies to a diff. Most of the research above was actually measured on open-ended questions, not pull requests.
So we built it: Quartet Desk, a native Mac app where every question you ask fans out to four models (bring your own Anthropic, OpenAI, and OpenRouter keys) and comes back as one synthesized answer with the disagreements surfaced instead of averaged away. Attach an image, get a panel verdict, export the result. The quartet, for your day-to-day. It's open source (MIT) because it contains nothing proprietary - the recipe is literally this post. Grab the DMG from the releases page. One early run asked whether to sell 50/50 raffle tickets nationwide through a club app: all four seats independently said no, and only two of them caught that legal sales key on the buyer's physical location at purchase time, not their billing address. One model alone will happily plan your felony.
The gate works because every piece of it distrusts something specific. Four labs distrust one lab's priors. The rubric distrusts vibes. The deterministic combiner distrusts a fifth model. Fail-closed parsing distrusts the network. The compile and test jobs distrust the judges. And the whole thing distrusts me, which is the point - it ran my model swap through the same teeth as everything else.
And if you're a newer engineer reading this: the discourse wants you to feel like a liability for using these tools. Ignore that. Learn the principles, but learn them the way this gate teaches them, as concrete, checkable failure conditions, with four reviewers who never get tired of explaining exactly why your PR blocked. Every block comment is a lesson with receipts. That beats learning security flaws by hand, one production incident at a time.
We let AI write the code. We let a different AI argue about it. We let a script decide. And we let the compiler overrule all of them.
A director signs up, picks their colors, pays. Hours later they have their own iOS and Android app in the stores. Here's the machine that makes that real.
A youth basketball director signs up on a Tuesday night. By the time their team practices on Wednesday, they have their own iOS app on TestFlight and their own Android APK, both branded in their colors, both reading from their data, both with their name on the Home screen. They never opened Xcode. They never touched Android Studio. They never heard the words "bundle identifier."
That is the whole thesis of TheMVP.ai (then Affirmi AppSpace): Everybody Apps. One repository, one codebase, 80% of it shared across every organization. Auth, builds, chat, push, streaming, payments, all written once. The other 20% is what makes a basketball org's app feel like a basketball org's app. Three verticals (Teams, Trainer, Realty) all ride the same engine; a single organizations.vertical flag swaps the tab set at launch.
The repo itself is not small. Thousands of Swift files, 77 Kotlin files, a Next.js web app, and about 20 Supabase edge functions. iOS uses MVVM with @Observable (iOS 17); Android mirrors it with StateFlow. Both pull from the same events-for-user edge function. When one platform changes a contract, the other has to move with it, which is exactly why the build system is the most paranoid part of this codebase.
It starts the instant a director POSTs to /api/interest. We generate a unique bundle ID right there: tv.affirmi.appspace.broski_ballerz_1508. In the same request, an automated Firebase service account hits the Firebase Management API, creates a per-org Android app under /projects/affirmitv-20fc3/androidApps, and stores the resulting google_services_json directly on the org record (registerAndroidAppInFirebase() in web/lib/firebase-api.ts). That JSON gets injected into every future Android build. No human in the loop.
The organizations table carries 80+ columns of app lifecycle state: bundle_id, asc_app_id, asc_bundle_id_resource_id, android_package_name, firebase_android_app_id, google_services_json, payment_status, app_status. The whole pipeline is a state machine, and that table is its memory.
When payment clears, the Stripe webhook does one atomic thing: it sets payment_status='paid' and queues four builds (TestFlight iOS, OTA preview IPA, Play Store AAB, and APK) in the same transaction (commit fd9e75f0). Payment and builds are inseparable by design. The iOS preview builds run on Paws, a Mac Mini, on a 5-minute cron (scripts/preview-build-watcher.sh). It signs with our Apple Distribution cert, imported into the System keychain with the -A flag so cron and SSH can use it without an ACL prompt.
App Store Connect is its own bridge. Apple's API can read and update app records but cannot create them, so a Playwright bot on a Windows VPS fills the "New App" dialog in headless Chrome every 15 minutes, then reports the asc_app_id back to the DB. Google Play works the same way through gpc-create-app.mjs and an Androidbot@affirmi.tv account with TOTP 2FA.
origin/mainHere is the rule that earns its keep. Iron Rule #16: Paws and CI never build origin/main. They build a pinned git tag, mobile-release-ios and mobile-release-android. Mobile work-in-progress lands on main freely so web can ship continuously and agents can iterate, but store builds only ever come from the tag. The tag advances exactly one way, through scripts/promote-mobile-release.sh, which refuses to tag anything that isn't an ancestor of origin/main, refuses a dirty working tree, and writes a forensic line to a locked, append-only audit log: timestamp, PID, user, platform, from_sha, to_sha.
The library that enforces it, scripts/lib/mobile-release.sh, fails closed. If the tag is missing it aborts with exit code 2 and prints, verbatim:
[mobile-release] ✋ ABORT: tag 'mobile-release-ios' does not exist...
[mobile-release] Refusing to fall back to origin/main (would defeat Iron Rule #16).
It would rather skip a build entirely than build the wrong thing. It even does a path-scoped checkout: only ios/ and mac-simulator/ get pinned to the tag, while scripts/, web/, and supabase/ stay at HEAD so infrastructure can iterate without dragging the binary forward.
Why all this ceremony? Because the failure it forecloses is expensive. During the P2 registration push, mobile work sat unfinished on main for days while the schema underneath it churned. A supabase db push from the main repo directory once came within seconds of shipping 56 migrations into the live production database, stopped by a manual project-ref inspection. The tag pin plus .github/workflows/migration-guard.yml, which scans every migration for 11 destructive patterns (DROP TABLE, ALTER COLUMN ... TYPE, TRUNCATE) and blocks the PR unless Steven signs off, exist so the still-deployed binary on a paying org's phone never wakes up to a backend that shifted underneath it.
Every client request (iOS, Android, web) stamps three headers: x-affirmi-session-id, x-affirmi-flow-id, and x-affirmi-flow-kind (one of go_live, watch, invite, create_org, build). A Vercel Log Drain pipes them into the app_logs table (30-day retention via pg_cron). One SELECT ... WHERE session_id=X reconstructs a director's entire path: tap → Vercel route → edge function → Mux call. When a stream needs attention, a 4-tier self-heal kicks in: watchdog, then a director button, then a Claude CLI agent on a 5-minute, $20 budget, then an iMessage to a human from the Mac Mini.
You don't get to a fleet of shipping apps by being clever once. You get there by deciding, in advance, that the machine is allowed to refuse. Build the wrong tag? Refuse. Ship a destructive migration? Block the PR. Lose a signing prompt to a keychain ACL? Pre-import with -A. Every safeguard here exists for a concrete reason. The discipline is not bureaucracy. It is the reason a director can sign up at night and trust their app to be real by morning.
How TheMVP.ai (then Affirmi) turned App Store distribution into a 4-step wizard a non-technical youth-sports director can finish without ever touching a terminal.
A youth basketball director plugs an iPhone into a Mac. Twenty minutes later, a branded team app with their logo, their colors, and their roster is sitting on the home screen. They never opened a terminal. They never read a UDID. They never asked "why isn't anything happening?" And here's the part that breaks every assumption about white-label distribution: when they did all this, they hadn't paid us yet.
That's the whole trick. Apple review takes 24 to 48 hours. A coach deciding whether to spend real money on an app does not have 48 hours of patience, and they definitely don't have the patience to do it before they're convinced. So we inverted the order of operations. The sales sequence isn't "pay, then we build it." It's see → believe → pay → install.
Before a director pays anything, they get the full Director Dashboard and a native macOS Previewer, a real SwiftUI + Catalyst build running locally, not a slide deck. It renders the branded iPhone interface on the left and the live team dashboard on the right. Change the team color or logo and the Previewer auto-rebuilds. A director can hand the laptop to a coach, say "this is your app," and the coach believes it on sight, because it is the app.
This inverted model shipped on 2026-04-21 (commit fd9e75f0). Per .claude-memory/project_inverted_payment_flow_20260421.md, payment now gates four things: App Store / Play Store submission, the iOS OTA install link, the Android APK install link, and Team Dashboard access. Everything that proves value (dashboard, branding, Previewer, preview builds) is unlocked for free. The Previewer is the demo. The install wizard is the activation moment.
After checkout clears, the director plugs in their iPhone and the Previewer shows one card: four steps, four sentences.
1) Connected? → 2) Tap Trust on iPhone → 3) Building (~15 min) → 4) Check email on iPhone.
The Previewer reads the UDID over USB using libimobiledevice binaries we bundle inside the .dmg (mac-simulator/Resources/libimobiledevice/, with idevice_id and ideviceinfo). No Homebrew, no Xcode command-line tools, no "first install this." It works out of the box on a coach's MacBook.
The most important design decision in mac-simulator/App/USB/USBDevicePanel.swift is what we removed. The old post-pay UI had "Install again," "Send to iPhone," "Generate new build," and "Approve build." Per docs/PRD-install-wizard.md, all of those are gone. Every one was a chance for a non-technical person to freeze. The wizard card is pinned to minHeight: 560pt with .animation(.none, value: step) so it doesn't bounce vertically as the steps advance. Small thing, but layout shift is how you make someone feel lost.
Two more details that matter when your user can't debug anything themselves:
USBDeviceMonitor.swift doesn't start reading the device until the director explicitly taps Start. No mystery background activity.Here's the architectural commitment that makes this survivable: the delivery channel is email, not the UI.
When the UDID is captured, web/app/api/devices/register/route.ts validates the format, registers the device in App Store Connect, and upserts a simulator_adhoc_requests row at status=queued (behind director JWT auth, a role gate, and a payment_status=paid check). A Paws cron, scripts/simulator-adhoc-worker.sh, runs every 2 minutes, picks up the queued row, flips it to building, calls our existing ad-hoc IPA signer rebuild-adhoc.sh, and writes back manifest_url. The GET handler polls for that manifest and, the moment it appears, fires sendOtaEmailIfFirstTime().
That email (web/lib/ota-install-email.ts) is engineered for the dumbest possible failure modes. It links to an https landing page button, not a raw itms-services:// URL, because Gmail strips those. It's light-mode only with a YIQ-computed button text color so the brand color stays legible against any background. And it's idempotent: a NULL-filtered UPDATE on ota_email_sent_at atomically claims the send, so a director who unplugs and replugs their phone doesn't get a second email. The branded landing page at web/app/install/[orgId]/page.tsx then detects platform and does the itms-services:// redirect on Safari, or serves the APK on Android.
If the Previewer crashes mid-build, the email still arrives. The coach opens it on their iPhone, taps Install, and the branded app lands on the home screen in about 30 seconds.
The shape of this flow comes from running it end-to-end with a real design-partner club, against the spec in docs/PRD-install-wizard.md. That is what settled the calls a non-technical coach would otherwise trip on. The OTA email fires server-side from a Paws cron, so it sends whether or not the director is watching the screen. A replug re-sends through an atomic idempotency claim on ota_email_sent_at, never a duplicate and never nothing. The "snag" state waits out 120 consecutive errors before it surfaces, so a transient flake never abandons the user. And the email links an https landing page, because Gmail will not open a raw itms URL.
Every one of those is a paper cut a coach would read as "this is broken, forget it." You do not design this flow at a whiteboard. You watch a real coach reach every sharp edge, then file them down.
The hard part of white-label distribution was never the signing pipeline. It was getting a person who has never seen a terminal to install a real, branded, native app, and to want to before paying. We solved it by inverting the money (prove value with a local native Previewer first) and by treating email as the product and the wizard as decoration. If a step can fail silently in front of a non-technical user, assume it will, and make the resilient channel the one that ships the goods. See, believe, pay, install. In that order.
How three little headers let us reconstruct a tap → API → edge function → Mux video call in a single SELECT, and how that same data heals the platform before it pages a human.
The worst bug report in the world is "the stream didn't work." No error, no stack trace, no repro. Just a coach at a gym, a frozen screen, and a parent two states away who missed the tip-off. By the time you hear about it, the moment is gone.
For a long time, debugging that meant archaeology: grep the phone logs, then go dig in the web server logs, then check the video provider's dashboard, then try to line up timestamps across three systems that don't know each other exist. Hours, for one frozen stream. So we built a way to skip all of it. Now, one query against one table replays the entire path a user took through every layer of the stack, in order.
Every request from every client (the iOS app, the Android app, the web portal) stamps three headers before it goes out:
go_live, watch, invite, create_org, build.Those headers ride along as the request hops inward: phone → our web route on Vercel → a Supabase edge function → the call out to Mux. Every layer forwards them and logs against them. A Vercel log drain plus inline logging funnel all of it into a single app_logs table.
The payoff is almost stupidly simple. To replay a user's whole journey, you don't correlate anything by hand. You write:
SELECT * FROM app_logs WHERE session_id = '…' ORDER BY received_at;
…and the entire story comes back in sequence: the tap that started it, the route it hit, the edge function it triggered, the exact Mux call and what Mux said back. Tap to video, across three systems, in one result set. We call it the sessionalizer.
One of our physical test phones surfaced a contradiction: the broadcast went live, but the database never received the video stream's ID, so the watch link had nothing to play. "It went live but there's no stream" is the kind of thing that used to eat an afternoon.
We ran the query. The log sequence showed the stream being activated with no record of a stream being created before it. The system had flipped the "we're live" switch and skipped the step that provisions the video.
That one gap pointed straight at the cause. Our stream code reuses an existing live stream when one is already open for that team. A sibling test had left a stream open on the shared test account, so the new test grabbed that one instead of minting its own, and the new game inherited a stream that pointed nowhere. Without the stitched logs, that is a multi-hour mystery across three services. With them, it was a five-minute read.
Here's the part that turns logging from a debugging tool into infrastructure: those same logs and heartbeats feed a system that fixes problems before a human ever sees them.
It runs in tiers. Everything is instrumented (tier zero). An hourly watchdog catches whole classes of failure and auto-corrects the ones it knows how to (tier one). When a real-device test fails, it doesn't immediately page anyone. It drops a job on a repair queue, and a worker tries targeted recovery: restart the phone's debug bridge, reconnect a dropped Wi-Fi, wait out a transient video hiccup. Only after recovery fails twice in a row does a human get an iMessage.
The discipline that holds it all together is a rule we hold: a dashboard that's always red is useless. When every tile is red for unrelated reasons, your eyes learn to skip the one tile that's screaming about something real. So every alert has to be either actively-being-fixed, explicitly-muted-with-an-expiry-date (rendered yellow, not green, so it's still visible), or deleted. There is no "just always red." A red light only works if red is rare.
You can't fix what you can't see, and you can't see a distributed system by reading each layer's logs separately. The fix is not more logging. It is correlated logging: a shared thread that ties a single human action together across every machine it touches. Stamp three headers, funnel everything to one table, and "the stream didn't work" stops being a ghost story. It becomes a query.
One founder, a fleet of live club apps, and infrastructure that spans Vercel, Supabase, a Mac Mini, a Windows VPS, GitHub Actions, Mux, Resend, and Sentry. Any of it can break at 2 a.m. Here's how the system fixes itself before a human has to.
When something breaks, most fixes follow the same shape: read the logs, find the one piece that's wedged, and restart it. That is exactly the kind of work you can hand to software, and increasingly to an AI agent. So we built it, bottom-up, in tiers.
Every request (a tap in the app, through the web route, through an edge function, out to Mux) is stamped with a session ID and a flow ID, and they all land in one table. One query replays a user's entire path through the whole stack. When something goes wrong, we're not guessing; we can see the exact sequence that led to it.
An hourly job runs six health checks and quietly auto-fixes three of them on its own: it restarts a stuck app-record pipeline, re-triggers a stalled build, and SSHes into the VPS to re-run a cron that missed its heartbeat. No human, no notification, because nothing was ever wrong for a human to see.
Live game streams are the most fragile thing we run, a parent's phone on stadium Wi-Fi. So stream state is reconciled against Mux, the source of truth, not a flaky network blip. A brief hiccup is recognized and recovered, never mistaken for a dead stream. We even suppress the alert for routine sub-60-second blips right after a stream starts. The system heals it, and nobody gets paged.
When the lower tiers can't resolve something, a Claude agent is handed the relevant session logs and asked to diagnose it. It classifies every fix as SAFE or RISKY. Safe fixes (restart this worker, re-queue that job) it performs automatically and opens a pull request documenting exactly what it did. Risky changes, anything touching the database schema, production environment, or secrets, it is not allowed to make, and those are queued for a human. It runs on a hard budget: five minutes, twenty dollars, five repairs a day. Bounded autonomy, on purpose.
Only when none of that works does it reach me, an instant iMessage, with email as the paper trail. And because the agent already tried, I don't arrive blind: I can see what it attempted and why it stopped, so I'm fixing the real problem instead of rediscovering it. Every alert also resolves cleanly. The next healthy check fires an "all clear," so I always know an issue has closed.
There's a popular take that AI customer support is the worst use of AI, a wall between you and a human. I get the frustration. But there is a real version that helps: on complicated distributed systems, "parse these session logs and restart the specific piece that's wedged" often solves the problem faster than waiting for a person. These systems have existed for decades. It is why the cable company can reset your box before an agent picks up. What's new is how much complexity an agent can resolve before routing you to a human, so you talk to one less often, and when you do, they arrive with the full context of what the agent already attempted.
That's the self-healing we built into a weekend project that gives every youth sports team its own app. It's the reason one person can keep every live app running, and the second half of our tagline: Everybody Apps. Nothing Breaks.
Most computer-use agents drive a browser from the inside. We wanted one that could operate anything on the screen: native apps, and the OS dialogs a browser never sees. So we gave it a $4 microcontroller for hands.
We had a pile of interface work that no API could touch: a native mobile app with no automation hooks, an OS-level login dialog, a store console whose only interface is its own UI. A headless browser handles none of that. So instead of scripting a browser, we built something that operates the machine the way a person does. It looks at the screen, decides what to do, and moves a real mouse to do it.
None of the pieces are new. A microcontroller pretending to be a keyboard is the old USB Rubber Ducky trick, except a Rubber Ducky is a blind payload fired into a machine you don't own, and this is a supervised loop on our own hardware. Screenshot-to-action agents are their own crowded field now, from Anthropic's Computer Use to open projects like OmniParser and Skyvern. What we wanted was the combination: out-of-band hardware input, so the agent runs against native apps and system prompts on a machine we own, driven by a vision model that can point at a button.
Flash a Raspberry Pi Pico with the right firmware and the instant you plug it into USB, the OS enumerates a keyboard and a mouse. No driver install, no accessibility grant. To the operating system it is a generic USB HID device, the same class as the mouse on your desk. It is not hiding while it does this: the USB descriptors carry the stock Raspberry Pi vendor ID, which we left alone, and an absolute-positioning mouse report is itself a tell. The point was reach, not stealth.
Browser automation drives a page from inside the DOM, so it only works where a browser goes. Hardware input is just a cursor moving and keys pressing, so the same rig works in a native iPhone app mirrored to the Mac, in a Save dialog, in a device that only exposes its own screen. We gave the Pico a small serial vocabulary: absolute cursor moves on a 0-to-32767 grid (a custom HID report descriptor, since the stock mouse is relative), plus clicks, scrolls, and typed text. One JSON object per line, one acknowledgement back.
A CircuitPython gotcha worth knowing: the firmware trimmed each command with del buf[:n]. That works in desktop Python. In CircuitPython a bytearray does not support slice deletion, so it raised the moment the first newline arrived. It only looked silent until I attached the serial console.
Every step, the rig screenshots the display and hands it to UI-TARS-1.5-7B, ByteDance's GUI agent. We use it purely as a grounding layer and ignore its planning head: given the image and an intent like "the email field," it returns a pixel coordinate. Frontier models reason about a screen well but miss precise pixel targets, so we let Claude do the thinking and UI-TARS do the pointing. We run it through OpenRouter at roughly five hundredths of a cent per call.
It is reliable on large targets and shaky on the small nav icons that sit packed together. The fix is the one a person uses when squinting: zoom in, crop the region, and ask again. We store the answer as a fraction of the display rather than an absolute pixel, and we re-capture and re-ground before every action, so we never depend on a coordinate surviving over time. The one thing that genuinely bites is a second monitor, where the absolute-pointer grid spans the whole virtual desktop and needs its own transform. Get that wrong and the click lands in the dock.
Claude (Opus 4.8) runs the loop. It reads the frame, decides the next action, catches the dialog that was not supposed to be there, and confirms the screen changed the way it expected before moving on. The hands and eyes hold no state on purpose, so the judgment lives in one place. That is what separates this from a recorded macro, which replays keystrokes and falls apart the first time reality differs from the recording.
One note on scope, because "agent with a keyboard" invites the wrong picture. It runs against our own accounts, on our own machine, at a human's pace, with a person directing the work and signing off on anything that goes public. It does not type passwords, approve security prompts, or make purchases; those stay with a human. Where a service offers a real API, we use the API. The rig is for the interfaces that don't have one.
A concrete run: a human created a brand's accounts, and the rig handled the upload-and-publish flow on them, a batch of vertical video ads across two apps in an evening. The division of labor is the part people get wrong. The graphics and the videos were made by code, not by a mouse dragging clips around an editor: HTML rendered to PNG through headless Chrome for the stills, ffmpeg for the reels. The rig handled the part with no API, opening the app, choosing the file, typing the caption, tapping through publish, and clearing the unrelated system prompt that stole focus halfway through a caption.
The failures were rarely where I expected them:
The rig has driven exactly the flows we pointed it at and not one more; every "it can operate anything" instinct we had turned into one of the failures above. But the shape holds: a cheap out-of-band input device, a grounding model for the eyes, and a frontier model to reason is enough to hand off interface work that neither an API nor a browser script can reach.
Every client thread we run has an agent sitting in it. It reads the conversation every two minutes, answers what it can answer, and the same channel carries our production alerts back out.
The way we find out something is wrong is usually a text message at nine on a Saturday night. A director in one of our group chats types a sentence about a stream that never went live, in the words they would use with a friend, and expects a person on the other end. There is one in the thread. There is also an agent.
The foundation is smaller than people expect. Messages on a Mac keeps every conversation in a SQLite database at ~/Library/Messages/chat.db. A process holding Full Disk Access can read it. A process holding automation permission for Messages can send through osascript. Those are the same two grants a backup utility asks for on first launch. There is no private API in this and nothing to wait for.
An sshd session inherits Full Disk Access. A cron or launchd job does not, because the grant is bound to a specific signed binary inside a GUI session, and a read from one comes back unable to open database file. So the read is issued from a session that carries the grant, including when the machine holding the conversation is the one asking.
The read itself is a single statement joining message to chat_message_join and handle, scoped to one chat, filtered to rows above a stored watermark, ordered by date. associated_message_type = 0 drops tapbacks and edit records, so a thumbs up never wakes the model. Sender identity comes from the joined handle row, which is NULL on the agent's own outbound messages and collapses to a literal me.
Modern macOS keeps the body as a streamtyped NSAttributedString blob in attributedBody and leaves message.text NULL for most things carrying a URL or rich content. So we decode it by hand. Find the literal bytes NSString, skip the object header, read a one byte length, and when that byte is 0x81 read a little endian uint16 from the next two instead. The plain text column stays wired up as a fallback for the rows where the blob yields nothing.
Both body columns come back wrapped in hex(). That way a message containing a pipe, a newline, or raw binary cannot corrupt the delimited dump on its way out of SQLite.
The anti-loop is free. The agent's account is a member of the chat, so its own sends come back on the next tick marked is_from_me = 1 and get filtered before anything reaches the model.
Each thread holds one long-lived Claude session. The first tick opens it with --session-id and the full charter. Every tick after that resumes the same conversation and hands it only the new messages. The conversation is the context, which means the agent remembers what it told a director three weeks ago without us building a memory layer for it. When I want to see what it is thinking, I attach to that same live session from a terminal and talk to it directly.
A charter is a plain text file of persona and facts, committed to the repo alongside chat-responder-engine.sh. The engine appends a shared security contract at runtime, so a new persona can add knowledge and voice while the rules underneath it stay fixed.
The most load-bearing characters in the whole system are two empty quotes. The session runs as claude -p --tools "", so the model reading untrusted customer text has no filesystem, no repo, no network, and no secrets. Its entire capability is to emit a string or to emit NO_REPLY.
Every inbound body arrives wrapped in an explicit pair of untrusted-content delimiters. The sender label sits outside that fence, written by the harness from the database handle, so message content has no way to forge it. Text inside the fence is conversation to answer, and it carries none of my authority even when it claims to be me. The contract also forbids credentials, keys, unlisted URLs, infrastructure detail, invented pricing, and invented dates. When the agent does not know something, it says it will check with me.
The reply never touches a shell command line. It is written to a file, copied to the sending machine, and handed to an AppleScript helper by path. An injected reply is a content problem at worst, and it stays one.
Most ticks produce nothing, and that is the design. NO_REPLY is a first-class answer, used whenever people are talking to each other and no assistant belongs in the exchange. If the model ever returns prose that merely contains the sentinel on its first line, the harness treats the tick as silent anyway and logs the drift for inspection.
One integer per chat decides what has been seen. It never defaults to zero: missing or corrupt state seeds to the chat's current maximum row and processes nothing, which keeps a wiped state file from replaying a client's entire message history. Writes go to a temp file and rename into place, atomic against a crash mid-write. The number advances once a send succeeds or once the model has chosen silence, so a failed send comes back around as a retry on the next tick.
Feedback follows a separate path from conversation. A capture job pulls new inbound messages out of our design review thread and writes them into a dedicated table on production. RLS is on with zero policies, which under Postgres means anonymous and authenticated roles are denied everything and only the service role can read it. Dedup authority sits in the database as a UNIQUE constraint on the source message row id, so a reset watermark or a re-run cannot produce duplicate rows. The local watermark advances only after a confirmed write.
A different agent picks the work up from there. The coding sessions that run against our repo open by selecting the rows with no processed_at; the chat agent never touches that table, because it has no tools to touch it with. From there the work moves onto the same rails as everything else. One ticket becomes one session in an isolated git worktree, which becomes one pull request, which faces the four-model judge panel before it can merge. A blocked pull request is re-queued with the judges' feedback verbatim and lands in front of me in the next morning digest.
Corrections that generalize get promoted into repo law. The rules file at the root of the monorepo carries 32 numbered rules with the reason written beside each one, and the memory directory holds 368 notes, 62 of them standing corrections from a human. One sentence from our design partner, asking us to stop using our internal term for the on-screen score overlay because nobody outside the building says it, is now a standing rule covering push notifications, email, landing pages, and store metadata.
Outbound runs through the identical channel. 45 call sites across 20 scripts page us over iMessage, and the reason is latency: carrier email-to-SMS gateways were taking anywhere from 5 to 45 minutes, while a send from a Mac we own arrives immediately.
The channel stays useful because it stays quiet. Every alert opens a tracked state that the next healthy check closes with an all clear, so a silent thread is a resolved one, and anything we cannot action today sits behind the committed quarantine file described earlier in this collection. A self-heal that simply flips a stream back to active within 60 seconds of it starting never pages at all.
A new chat is a config file and a charter dropped into a directory, and chat-dispatch.sh runs every config it finds in parallel, one engine per chat with its own lock, state file, and log, so a slow chat never delays another one and no schedule needs editing. Before a thread goes live, a test mode routes every reply to me alone and never to the customer chat. Enabling it for real runs an ordered gate: check connectivity, seed the watermark, run one manual tick that has to exit clean, and only then install the schedule.
What makes this work is placement. The report arrives in the channel with the lowest friction on the customer side and lands in the one with the most structure on ours: a fenced string in, then a row in a table, a ticket, a pull request. The thread is the interface. Everything after it is ours.
Plenty of clubs run a locked-down sideline: no parents on the floor, no phones out. For every game those clubs play, phone-as-broadcaster is off the table, so hardware became a first-class streamer.
A director registers a camera once. It gets a name, an optional team, and one ingest address that never changes. The operator configures the device a single time in September. On game day they turn it on and press start.
A camera feed has to land on the right game. Timing gets that approximately. Assignment gets it exactly.
So a game points at a camera the way it already points at a parent. events.camera_id sits beside events.streamer_user_id, and a director sets it while building the schedule. The API refuses to put one camera on two games whose windows overlap. A camera films one game at a time, so a double booking is a scheduling mistake, and the moment someone makes it is the cheapest moment to catch it.
The camera holds a stable key. Mux holds a fresh live stream for each game. A relay sits between them. The camera's side never changes: rtmp://cameraingest.streamon3.com:1935/live/<relay_key>, typed into an XbotGo or any other RTMP encoder once in September and left alone. Mux mints a new stream key for every game and expects RTMPS on its own host. The relay's job is narrower than it sounds: turn one long-lived publish identity into the right short-lived one, every time a camera connects.
Handing a camera a Mux key directly would work for one game and stop working for a season, because the key would change every week and an operator on a ladder at a gym should not be the one changing it. It would also put a Mux credential inside hardware we do not control, on a network we do not control. The relay keeps the durable secret on our side and the disposable one at Mux, and the camera only ever holds the durable one.
Two endpoints do the translating, and they answer different questions. POST /api/relay/on-publish fires the moment a camera starts publishing and checks one thing: does this key belong to an active camera with a game open right now. A refusal answers with a non-2xx status and nginx closes the connection at the handshake, before a single frame moves and before any Mux object exists. POST /api/relay/resolve fires right after, from a separate process, and answers a separate question: which Mux stream does this publish land on. Both calls run the same resolution rules (continuity first, then the single open booking window, and any ambiguity refuses everything), so the gate and the push always agree on which game a camera belongs to.
They stay two calls because the gate can never be the one that mints. Gym wifi drops and reconnects a camera more than once a game, and on_publish fires on every one of those reconnects. A gate that minted a Mux stream would fork a new stream, and a new archive, on every blip. Minting lives entirely in resolve, and it is idempotent: the game's Mux stream is written with a guarded update that only succeeds while the column is still empty, so two connects racing the same game cannot fork two streams. The loser rereads the row, finds the winner's stream key, tears down whatever it minted for itself, and pushes to the same destination.
An RTMP publish holds a session open for the life of a game. There is no single request-and-response for an HTTP redirect to answer once video is already flowing. So the relay keeps the camera's session pinned to nginx and, the moment publishing starts, spawns a small script that asks /api/relay/resolve for this game's Mux destination and execs ffmpeg to remux the feed there with -c copy: no decode, no re-encode, the RTMP the camera already sends repackaged straight into Mux's ingest. If a resolve call fails, or Mux is briefly unreachable, the script exits without pushing anywhere, and nginx's default respawn runs it again a few seconds later while the camera keeps publishing. A transient hiccup clears itself before anyone at the gym notices.
XbotGo / OBS / any RTMP encoder
| rtmp://cameraingest.streamon3.com:1935/live/<relay_key> (stable forever)
v
GKE LoadBalancer -> nginx-rtmp pod us-west1-a, autoscaled 2..12
|
| on_publish (gate): POST /api/relay/on-publish
| continuity, then booking window; no match -> non-2xx, publish refused
v
2xx: publish accepted, session stays open
|
| exec_push spawns relay-push.sh
v
POST /api/relay/resolve
reuse the event's Mux stream if one is live, else mint one
and stamp it with a guarded, race-safe update
|
v
ffmpeg -c copy remux -> rtmps://global-live.mux.com:443/app/<game_key>
|
v
Mux --video.live_stream.active--> supabase/functions/mux-webhook
| stream_status = active
| stream_started_at stamped
| LIVE NOW push to the team
|
|--HLS--> /watch/<event>, iOS, Android
|
'--video.asset.ready--> mux_archive_playback_id (the replay)Each seam is chosen so the piece on the far side can change without touching the piece in front of it.
Camera to relay is RTMP, and it is the only contract the operator ever sees. One hostname, one key, no TLS negotiation on gym wifi. The device speaks plain RTMP on port 1935; Mux wants RTMPS on 443. We terminate whatever the camera sends and upgrade the outbound leg ourselves, so hardware that refuses rtmps never has to know Mux exists.
Relay to Mux is one fresh live stream per game. Everything downstream depends on this. events.mux_live_stream_id stays one-to-one, so archive ownership, duration and replay all resolve without a lease or a session table, and Mux runs the transcoding ladder, the HLS packaging, the signed playback and the reconnect window that we have no reason to build ourselves.
Mux to us is webhooks, and they are what publish a game. A camera going live is not something the hardware announces. video.live_stream.active arrives, mux-webhook writes stream_status and stamps stream_started_at, and the LIVE NOW notification goes to the team. The same handler serves phone broadcasts, so a game filmed by a tripod and a game filmed by a parent take identical paths from Mux onward.
The intelligence lives in a normal Next.js route with database access. None of it lives in the media server, which only pushes bytes. The seam at the relay is the one place we rewrite an address, and rewriting an address is what turns a device that knows nothing about our schedule into a camera that shows up on the right game.
Gym wifi drops. Games run into overtime. Both produce the same event: a camera reconnecting while the clock disagrees with reality.
A camera already filming a game keeps filming that game. Before any window arithmetic runs, the relay checks whether this camera has a game in flight, and if it does the feed rebinds there and the schedule is ignored. A human ends a game. A timer does not.
A second feed cannot take over a game that already has one. Until multi-camera exists as a real feature with a real switcher, someone watching a game sees the camera assigned to it for the whole game.
Web traffic drifts up and down over hours. Relay traffic does not drift. AAU basketball, club soccer, an XbotGo-filmed tournament bracket: games start on the hour, and a Saturday morning slate can put dozens of cameras onto the relay inside the same two or three minutes, hold flat for as long as the games run, then fall away together at the final whistle. The right capacity at 9:45am has little to do with the right capacity at 9:15am or at 11:00am.
CPU is the wrong number to size against. The remux step is -c copy: ffmpeg repackages bytes it already has, no decode and no encode. A pod carrying ten quiet games and a pod carrying ten games at twice the bitrate cost roughly the same CPU and very different bandwidth. The unit of load that matters is active publishers: one camera, one relay session, one ffmpeg child, whatever the camera happens to be filming.
The relay runs on GKE with a floor of 2 replicas and a ceiling of 12, and the two directions move at deliberately different speeds. Scale-up adds up to 4 pods a minute, sized for a Saturday-morning wave of cameras connecting inside the same two-minute window. Scale-down removes at most 1 pod every 15 minutes, behind a 30-minute stabilization window, because a pod that looks briefly quiet may be holding a game about to start its next quarter.
Removing a pod from the pool is not the same as killing it. A pod marked for scale-in drops out of the load balancer's endpoints first, so no new camera lands on it, and it keeps every publish it already has until each one finishes on its own. terminationGracePeriodSeconds is set to 28800, eight hours, the same ceiling Mux puts on a single continuous session, so the cluster can never force a pod down out from under a live game. A rolling deploy replaces one pod at a time for the same reason. Nothing about scaling this fleet is allowed to end a live youth game as a side effect.
The bandwidth math is small. One stream costs the relay 7 Mbps, 3.5 inbound and 3.5 back out to Mux. Fifty concurrent games is 350 Mbps sustained through the pool, which packs onto two or three e2-medium nodes. Egress runs roughly 157 GB per two-hour window at that concurrency.
Tested end to end with a real RTMP encoder pushing straight into the relay, a camera's first frame reaches active in about 15 seconds, and it reaches active on Mux's word: the video.live_stream.active webhook flips events.stream_status, never anything the camera itself claims. That is the identical trigger a phone streamer goes through, and the relay's only job is getting the camera into that pipeline.
A camera is a feed that belongs to one game at a time. The rest follows from that sentence. The director assigns, the relay routes, continuity holds a feed through a reconnect, and the fleet scales to the number of cameras actually publishing.
The rig could already move, click, and type like a person. Scroll was the last motion that still read as a machine, so we taught it to glide.
A mouse wheel speaks in notches. The simple way to scroll a page is to send a few big notches and let the operating system sort out the rest. It works, and it reads as a machine every time: the page lurches, overshoots, and settles. A person does not scroll like that. A thumb rolls the wheel in a smooth arc, quick through the middle and slow at the ends, and the page tracks the motion.
So we send it the way a thumb rolls it. Each notch becomes a run of single-unit wheel reports on an ease-in and ease-out cadence, with a little jitter, starting slow, quickening, and easing to a stop. The report the wheel sends is still an ordinary wheel report; there are just more of them, smaller, and paced like a hand. The page glides instead of jumping.
macOS then layers its own scroll acceleration on top, keyed to how fast the reports arrive. The same run of reports paced slower lands softer, because the operating system multiplies a fast flick and leaves a slow one closer to its raw size. That handed us a single dial for feel: change the spacing between reports and the whole scroll gets gentler or sharper.
The humanization lives in the firmware, not the host. The brain sends one high-level intent, scroll this far, and the Pico turns it into the run of reports. That keeps the judgment in one place and means a change to the feel never touches the agent. We exposed the spacing as a per-command knob:
{"scroll": {"amount": 5, "pace": 1.5}}
// pace above 1 is slower and gentler, below 1 is quickerTravel per notch is a second dial, and both sit at the top of the firmware, one edit away for anyone who wants a different feel. Speed is now something you set per scroll, without opening the board back up.
A timing detail worth knowing: the firmware acknowledges a command only after it finishes. A long smooth scroll takes more than a second, and the host's read window was a fixed fraction of that, so the acknowledgement for a big scroll arrived late and got read as the reply to the next command. We scaled the read window to the length of the scroll, so the reply always belongs to the command that earned it.
We tuned it against the Instagram feed, a custom scroll container that shrugs off a lot of synthetic input. It scrolls a post at a time, at whatever speed we ask for, and stops where we tell it.
There is a limit worth naming. A standard HID wheel speaks in whole units, so one report is the smallest step there is; we can make each step gentler by pacing it, but we cannot send half a notch. True sub-notch scrolling, the pixel-smooth kind a trackpad does, needs high-resolution HID: a resolution-multiplier field in the report descriptor that lets the wheel carry a fraction of a line. That is a descriptor change, and it is the next step if we ever want scrolling finer than a thumb. For driving real interfaces, a thumb is the bar.
The rig is meant to be indistinguishable from a person at the keyboard. Scroll was the last motion that still gave the game away. It doesn't anymore.
2026-09-04
"Four Judges, One Diff" (June) explains the gate. "Why the Quartet" (July) shows the wiring. This is what changed between July and September: the panel now lives in one file, the Anthropic seat is chosen per run based on who wrote the code, silence from a judge is a block, every blocker has to name a customer, and the quartet now spends most of its time auditing code that a much cheaper model wrote.
We run the quartet in two places. GitHub Actions runs it on every pull request that carries the agent label. Agents also run it locally, before they open the PR, with a command-line tool. Two systems, one policy, and a policy that lives in two files drifts. So the seats, the severity rules, and the impact test now live in a single module, scripts/lib/review_policy.py, and both systems import it. A unit test asserts that the CI gate and the local tool resolve to the same four models. A version bump applied to one caller now fails a test instead of quietly leaving the other caller a release behind.
The panel today:
Grok is not on the panel. We measured its signal and it did not earn the seat.
The July post made the case from the research: a model reviewing code carries the priors of the model that wrote it, and evaluators score their own output higher. Since then the fleet gained a second Anthropic driver. The swarm runs on Opus 5. Interactive engineering sessions run on Fable 5, and those sessions author shipping branches too. A fixed Anthropic seat is therefore the author for one of them.
The seat is now resolved per run. The tool takes --author claude-fable-5 (or the QUARTET_AUTHOR_MODEL environment variable), and the Anthropic judge becomes a model that did not write the diff: Opus judges Fable's work, Fable judges Opus's. An undeclared author keeps the documented default, and the run log prints which seat was chosen, so a review is never silently a self-review. The rule is simple to state: a model reading its own diff is a second look, and the gate wants a second opinion.
A four-judge panel is only a four-judge panel when four judges actually answer. The tool now treats every way a judge can fail to answer as a failure of the run:
NO VERDICT, and when the cause is an output cap it says so, so the fix is to raise the token ceiling, never to shrug.ship or fix counts. Anything else, including a well-formed JSON object with no verdict field, does not.--range used to produce a clean pass on nothing.A gate that blocks on style is a gate people route around. In August we measured the CI gate against pull requests we knew were correct and tuned it. Two rules came out of that.
A blocker now has to carry an impact clause: <finding> || IMPACT: <who is affected and how>. The policy module applies the test; a blocker without a concrete impact is recorded as a major, which the author must address but which does not hold the merge on its own.
The panel verdict follows a consensus rule. Two judges independently raising a supported blocker fails the run. A single security or hard-rule blocker fails the run by itself: one is enough. Each judge also scores the diff from one to ten on five axes: correctness, security, house rules, simplicity and reuse, and the correctness of the pull request's own claims about itself. The scores are advisory. The blockers decide.
The biggest change is in what the quartet reviews. Our default build loop is now: a cheap frontier model writes the code, real tests run against it, the quartet audits it, and the loop repeats until the panel is clean. The writer is GLM 5.3 Flash, called directly with streaming and a capped reasoning budget. One large file takes about a minute; files run in parallel.
The case that set the pattern was a Postgres-backed task-queue service for our fleet of screen-driving agents. The first draft did not run. Four cheap passes and a few quartet rounds later, the same service carried passing integration tests and a ten out of ten security score from the Gemini seat. Hand-writing that code with the expensive model would have cost many times more and reviewed no better. The expensive model's job in this loop is to orchestrate, read the panel, and write the tests that prove the fix.
This week we shipped a change that moved our pending-build dispatcher from a Mac Mini cron to a serverless sweep with a database lock. It took eight rounds. Each round ran the quartet and, alongside it, three adversarial verifiers with different lenses (races, the state machine, the callers), each told to refute the author's claims and allowed to execute the real classifier against histories they invented.
The two kinds of review found different things. The quartet caught a select that omitted the one column the code below it tested on, which two judges flagged independently, and a 409 response that every consumer was reading as success. The verifiers caught what only execution shows: a build row that an operator re-queued being parked forever because its last run had completed months earlier, and attempt counting that spanned a reused row's entire history. Both went back into the code before merge. A panel reads the diff. An adversary runs it. We now do both on anything that touches money, auth, or a lock.
Four frontier reviews of a typical diff still cost low single-digit dollars. A full round with the three verifiers costs more than the panel does, and a project-level change earns it. Compile and test runners remain the larger line item.
# four-judge-gate (pseudo-config, September 2026)
policy: scripts/lib/review_policy.py # ONE file: seats, severity floor, impact test
panel:
- { seat: gpt, model: openai/gpt-5.6-sol, fallback: none } # dead id = skip = fail closed
- { seat: gemini, model: google/gemini-3.1-pro-preview, fallback: gemini-2.5-pro }
- { seat: qwen, model: qwen/qwen3.8-max }
- { seat: anthropic, model: resolved_per_run(author) } # never the model that wrote the diff
input: merge-base diff + title + body # claims to verify, never evidence
refuse: empty diff, oversized payload, unreadable file, missing verdict
blocker: must carry "|| IMPACT: <customer effect>" or it is a major
fail_if: two independent supported blockers, or one security / hard-rule blocker
loop: cheap model writes -> tests -> quartet -> fix -> repeat until cleanWe let AI write the code. We let cheaper AI write most of it. We still do not let it grade itself.
About the author. Steven Ojo is the founder of TheMVP.ai, the club app for youth sports: each club gets its own app with schedules, team chat, registration, live streaming, highlights and Reddi, an AI teammate. Before the 2026 rebrand the company was Affirmi and the product was AppSpace.