Building a Trustworthy AI Tutor: An Engineering Case Study of the BossFx AI Platform

The problem

Retail forex education attracts exactly the people who can least afford a confident wrong answer. A beginner who is told the wrong thing about leverage or stop-losses can lose real money. So when I set out to build an AI learning assistant for BossFx students, the hardest requirement was not "answer questions" — it was "never answer beyond what the curriculum actually teaches, and say so plainly when it doesn't know."

That reframes the entire engineering problem. A chatbot optimizes for a good answer. A trustworthy tutor optimizes for the right answer — or an honest "I don't have material on that; ask your mentor." In this domain, a confident hallucination is not a bug. It is a liability.

Constraints that shaped every decision

• Solo founder.One engineer maintaining everything. "Bus factor" is a named risk, so the architecture favors boring, mainstream technology and the smallest structure that prevents drift — never the largest that looks impressive.
• Real-money domain. No output may read as personalized financial advice; the no-answer path is treated as the most important path in the system.
• Honesty as a feature. The system must be able to measure — and admit — what it cannot yet do.

Architecture in one paragraph

The platform is a TypeScript monorepo with a deliberately capability-agnostic core: a provider adapter layer (Anthropic and OpenAI behind one interface), a retrieval layer over Postgres + pgvector, a pipeline that orchestrates the retrieval-augmented generation (RAG) flow, and an evaluation harness that scores the whole thing. The "Learning Assistant" is simply the first capability riding on that substrate; future capabilities inherit the same retrieval, evaluation, and governance for free. Knowledge lives behind a dual-enforced editorial-review gate — nothing is retrievable until a human approves it.

Five engineering decisions worth defending

Each of these is an immutable Architecture Decision Record in the repository.

1. pgvector over a dedicated vector database (ADR-001). At the scale of a curriculum — hundreds to low-thousands of chunks — a separate vector store is operational weight a solo founder cannot justify. Cosine similarity through a SECURITY DEFINER Postgres function keeps retrieval, authorization, and data in one system.
2. A provider adapter, not an SDK dependency (ADR-002). No capability code touches a vendor SDK directly. This looked like over-abstraction until a real model migration proved its worth (below).
3. The editorial-review gate, enforced twice (ADR-011). Every knowledge document carries a reviewed_at timestamp. Retrieval filters on it in both the SQL function and the TypeScript retriever — defense in depth. Ingestion clears the flag on every re-ingest, so edited material must be re-approved by a human before it can ever be served again.
4. Evaluation as a first-class, blocking concern (ADR-004). The golden-set harness was built alongside the pipeline, not bolted on afterward. Anything touching prompts, retrieval, or model configuration is expected to pass a scored evaluation before it ships.
5. Prompts as immutable, versioned files (ADR-007). Prompts are production assets: versioned, snapshot-tested, and rolled back by git revert. Retrieved content and user input are always treated as untrusted data, never as instructions.

The centerpiece: why the first quality score was 35.7% — on purpose

The platform's first official evaluation against a 14-item hand-built golden set scored 35.7% (5 of 14 passing). It would have been trivial to make that number look better. I didn't, and that was the point.

The evaluation decomposed the failures honestly:

• Every fully-covered topic passed — leverage, stop-loss, spread, drawdown, session nuance.
• Six failures were content gaps, not engineering faults: the assistant correctly declined to answer questions (pips, lot sizes, demo-vs-live, risk-reward, market structure, an FAQ item) that the ingested Wave A curriculum simply doesn't cover yet. The grounding discipline worked exactly as designed.
• One failure exposed a real tuning need in retrieval (below).

A 35.7% baseline that quantifies exactly which curriculum content is worth acquiring next is far more useful than a 90% number that hides its own gaps. The score is the honest starting line the platform must beat — and it is now an immutable, archived artifact. Supporting metrics from the same run: grounding-failure rate 7.7%, retrieval precision/recall 22.6% / 42.9%, p50/p95 latency 4.5 s / 5.5 s, and a mean cost of $0.00225 per query against a hard per-query cost ceiling enforced in code.

A war story: the bug 86 green tests could not see

The evaluation's rubric judge — a Claude model that scores answers — failed on its very first real call: the Claude 5-family models reject a temperature parameter the adapter had always sent. Eighty-six passing unit tests never caught it, because unit tests mock the provider. Only a real API call could.

Two things saved the run. First, the judge was built to fail closed — a broken judge scored items as incorrect rather than silently passing them, so the failure was loud, not invisible. Second, the provider-adapter boundary (ADR-002) meant the fix was one change in one file, applied uniformly to both providers. The lesson is now a standing rule: any new model family gets one real smoke call before it is trusted anywhere.

Measure before you optimize

The retrieval layer shipped with an uncalibrated similarity floor of 0.25. The evaluation didn't just flag that it was wrong — it produced the distribution needed to fix it. Correct content matched at cosine similarities of 0.56–0.65; irrelevant content on out-of-scope questions still cleared 0.25 at up to ~0.47, because glossary chunks bundle many terms and dilute single-term matches. The two populations overlap in a 0.35–0.47 band that no single global threshold can cleanly separate today.

The recommendation — raise the floor to 0.35 now, then toward ~0.45–0.50 after content density improves — is deliberately not yet applied. Tuning a production threshold before the evidence justifies it is precisely the premature optimization the project's principles forbid.

The governance model: automation prepares, humans approve

The same discipline runs end to end. Automation ingests, chunks, embeds, retrieves, evaluates, and reports. A human performs every act that makes something official: editorial approval of knowledge, golden-set sign-off, prompt releases, and milestone freezes. This single principle — automation prepares, humans approve — began as one ingestion rule and became the constitution of the whole system.

Outcomes

• A production-grade RAG platform validated against real Supabase, real pgvector, and both real model providers — not mocks.
• An 11-document / 106-chunk curriculum corpus, every document human-approved before it could be retrieved, ingested for a total embedding cost of $0.00034.
• An honest, archived quality baseline and a five-milestone frozen history, built solo under CI gates, immutable ADRs, and 86 automated tests — with total model-provider spend across the whole knowledge-and-evaluation milestone of roughly $0.10.

What's next

The platform is intentionally gated behind honest readiness criteria: acquire the curriculum content the baseline proved is missing, calibrate the retrieval floor from the collected distribution, wire the evaluation gate into CI, then re-measure. The number only earns the right to go up after the work that justifies it is done.