Building a Trustworthy AI Tutor, Part II: Shipping It to the Public

Not long ago, the tutor from Part I lived behind a login and a feature flag — proven, but private.

Today, anyone can open BossFx Academy, ask a trading question, and get a grounded answer with its sources, without ever creating an account.

Part I was about building a tutor I could trust. Part II is about what happened when it met production.

The work I promised to do

Part I ended on a promise I didn't enjoy making. The tutor's first honest score was a bruising 35.7%, and I said out loud that the number would only move after the work that justified it — no shortcuts, no quietly nudging the baseline to feel better.

So I did the work in the order Part I named it: filled the gaps in the curriculum the baseline had exposed, calibrated the retrieval floor from real data instead of a hopeful guess, and wired the evaluation gate into CI so that nothing merges if quality slips. Only then did I let myself re-measure.

It reached an 80.0% gate pass, with grounding failures sitting at zero across every archived run. I care less about the number than about how it got there — earned in sequence, and impossible to fake past CI. That distinction turned out to matter more than the score itself.

Then I opened the doors

Here's the decision I'm proudest of: for the first phase, the tutor is public with no login at all. That's a product choice, not a shortcut. The whole value of a trustworthy tutor collapses if a nervous beginner has to create an account before they can ask whether they've misunderstood leverage.

Opening it up safely meant treating access as a capability rather than an identity. Anonymous visitors reach only the public parts of the system, their requests are rate-limited, and nothing personal is stored. The authenticated path didn't change at all — anonymous mode is purely additive, and it sits behind a flag I can pull in a single move. It shipped as v1.1.0. The grounding guarantee from Part I didn't change a line; it just started answering strangers.

The humbling part

Getting it working on my machine was the easy 80%. Production was the other 80%, and it was humbling in the most ordinary ways.

A 502 that looked like a catastrophe turned out to be a single wrong value in the deploy configuration — the code was fine. Rows silently failing to save traced back not to the database logic but to seed data that lived in a local file and had never been applied to production, with a foreign key quietly depending on it. A client SDK politely refused to send a request without a login token, breaking the exact feature I most wanted to show off. And a handful of public pages ran the tutor but had never loaded its script, so they fell back to the old engine without a word.

None of these were clever bugs. I found every one the same way this project taught me to work: isolate a single variable, ask what changed since it last worked, and let the evidence — not my first, most flattering guess — pick the cause. Every fix was small and reversible, because I'd built the system so that they could be.

What I actually built

The evaluation gate that made the private tutor trustworthy is the same thing that made the public launch uneventful. There was no launch-day scramble over answer quality, because quality had been a blocking part of every merge for months.

Boring launches are a feature you build long before launch day.

Why I trust it

The code answering questions in production today isn't dramatically different from the code I had months ago. Same retrieval, the same grounding guarantee, much the same pipeline.

What changed is that I trust it now — and, more than that, I know exactly why.

Building the model was never the hard part. Building the engineering discipline around it was: the evaluation gate, the archived runs, the review that lets nothing reach a student unapproved, the habit of small reversible changes. Production trust didn't come from the system being clever. It came from the process being honest — and that's the part you can't bolt on at the end.

I set out to build a tutor that sounds smart. I'm happier that I built one that knows when to say "I don't know," and that I understand well enough to explain why it does.