The Features of Ofuma, and Why Each One Exists

There is a moment, early in every programmer's life, that teaches you why version control exists. You change one working file, the program breaks, and you cannot for the life of you remember what it used to say. So, you sit there trying to un-type your way back to a thing that ran perfectly an hour ago. After that one moment, git stops being a chore someone imposed on you and becomes obviously, permanently necessary.

I built Ofuma because building with language models is full of those exact moments, and almost none of the tools to survive them existed yet.

So let me do something specific. Rather than list Ofuma's features like items on a menu, I want to walk through them the way they actually arrived: each one as the answer to a problem that hurt somebody first. If I do this right, by the end you will see why an app built on top of an LLM needs roughly the same discipline we long ago decided ordinary software needs, and why "prompts are just text, how hard can it be" is a trap.

The problem hiding under all the other problems

Start with the thing that goes wrong first.

In most LLM applications, the prompt (the actual instructions that shape how the model behaves) lives buried deep inside the application code. To change a single sentence of how your AI talks, an engineer has to open a source file, edit it, get it reviewed, and ship a whole deployment.

Think about who that locks out. The person who best understands how your support bot should answer is probably not the engineer. It is the content designer, the support lead, the domain expert. But they cannot touch the prompt, because the prompt is trapped in code they are not allowed to edit. So they file a ticket, wait for an engineer, and iteration slows to the speed of the deployment pipeline. Meanwhile the feedback you most need, which is simply "was that answer any good?", is scattered across thumbs-down clicks, support tickets, and Slack complaints, none of them connected to anything.

That is the knot Ofuma is built to untie. And the guiding idea behind every single feature is one sentence: treat prompts like code. Not as a cute analogy. As an operating principle, with all the machinery that "like code" implies: versioning, testing, gated releases, monitoring, rollback.

Here is how that principle turns into features.

Decoupling the prompt from the code

The first two pieces exist to pull the prompt out of the codebase and give it a home of its own.

The Prompt SDK is the thin client an engineer actually calls. Instead of pasting a wall of instructions into their code, they write something close to call(prompt_id="answer_question", inputs={...}). They reference a prompt by name, and they do not own its contents. Under the hood the SDK fetches the current version of that prompt, fills in the variables, sends it off to the model, and quietly logs what happened. To the engineer it is one function call. That simplicity is the whole point: it means the prompt's wording can change underneath them without a single line of their code changing.

The Prompt Service is the home that name points to. It is where prompts are written, edited, versioned, and published, all through a UI so a non-engineer can actually use it, and where they get promoted from development to staging to production. This is the part that breaks the knot. The content designer edits the prompt directly. No ticket, no engineer, no deployment. The change is still tracked and versioned behind the scenes, exactly like a code change, but the gate that used to require an engineer is gone.

This separation is what the design calls modular ownership: the prompt, the model choice, and the evaluation rules can each be owned and changed by the right person, independently. It is the precondition for everything else. I hope at this point you are not confused, because this next part is where it gets fun.

Knowing whether a change actually helped

Decoupling the prompt creates a brand new danger. Now that anyone can change a prompt quickly, anyone can also break it quickly. Speed without safety is just a faster way to ship regressions. So the next cluster of features exists to answer one question: did this change make things better or worse?

The Logging and Feedback Collector is the foundation. Every interaction flows through it: which prompt, what input, what the model said, which model, when, and for whom. And critically, it captures feedback: the thumbs-up and thumbs-down, the free-text complaints, and the quiet gold of implicit feedback, like a user editing the AI's answer into what they actually wanted. That edit is a perfect labeled example of right-versus-wrong, and most systems just throw it in the bin. Ofuma funnels all of these scattered signals into one schema, so feedback stops being noise in ten different inboxes and becomes data you can slice by prompt and by version.

The Golden Set Store is your test suite for prompts. In ordinary software, before you ship, you run your unit tests. The golden set is the equivalent: a curated collection of representative inputs paired with the answer you consider correct. You start small, handwritten by experts, and grow it over time, and the most valuable way it grows is by absorbing real failures. When the model botches something in production, you capture that exact case, add it with the right answer, and now every future version is forever tested against that mistake. Failures get memorized so they cannot quietly sneak back in.

The Eval and Judge Service runs your prompts against the golden set and scores them. For some tasks scoring is easy: did the output contain the right number or not? But most LLM work is open-ended, where there is no exact answer to match against. For those, Ofuma uses a technique called LLM-as-a-judge: a second model, given a rubric encoded as a prompt, grades the first model's output for correctness, tone, format, whatever your rubric cares about. This is how you evaluate "is this a good support answer?" without a human reading every single one. (That judge is itself something you have to learn to trust, which is its own calibration problem with real statistics behind it, so maybe more on that in another article 🤔.)

Changing things safely

So now you can change a prompt easily and tell whether the change helped. The next features make the act of rolling it out safe.

The Experiment Orchestrator brings A/B testing to prompts. Instead of swapping a new prompt in for everyone and praying, you route a slice of traffic, say ten percent, to the new version and leave the rest on the old one. Then you compare them on real metrics: thumbs-up rates, error rates, judge scores. This is the whole difference between believing your new prompt is better and actually knowing it. It can also run the entire golden set through a new model offline and lay the results side by side.

The Release Manager is the governance layer. It tracks which version of every prompt and model is live in each environment, and it enforces promotion rules: a new prompt might be required to hit a minimum pass rate on the golden set, with no critical failures, before it is even allowed near production. And here is the one that actually lets you sleep at night: it supports rollback. If a release misbehaves, you revert to the last known-good version in one move. The same instinct that gave us blue-green deployments and feature flags, pointed straight at prompts.

The CI/CD Pipeline is what ties all of those together into automation. Edit a prompt, change a model, tighten a rubric, add golden cases: any of these kicks off an evaluation run that scores the change against the baseline. If quality drops past your threshold, the build fails, exactly like a failing unit test blocks a merge. Bad changes get caught before a single user ever sees them, and good changes flow right through. This is the literal mechanism of CI/CD-style safety: nothing reaches production without passing through a gate.

Watching it out in the wild

The last piece accepts a genuinely humbling truth: your golden set will never cover everything. Reality will always find a case you didn't imagine. So you need to watch production directly.

Observability Dashboards turn the firehose of logs and feedback into something a human can actually read at a glance: success rates per prompt, the trend of happy versus unhappy users, latency, cost, and alerts when a metric suddenly lurches. If a new release sends the thumbs-down rate climbing, the dashboard shows it, the team investigates, and the Release Manager rolls it back. Observability is the smoke detector that tells you to go use the fire extinguisher you already built.

Two principles I haven't named yet

Threaded through all of this are two ideas worth pulling out on their own. I hope you are still with me 🥱.

The first is human-in-the-loop fallback. When the model is uncertain, or a guardrail trips, the system does not just shrug and ship a bad answer. It can route to a human, or fall back to a safe deterministic response instead. The AI's confidence is treated as a real signal, and low confidence triggers a safety net instead of a coin flip. The machine handles the easy ninety percent, and a person handles the hard, weird, high-stakes remainder.

The second is feedback-driven iteration, and it is the loop that makes the whole thing a system rather than just a pile of tools. User feedback flows into the collector. Painful failures become new golden cases. Golden cases gate the next release through CI. A better prompt ships. Observability watches the result, which produces even more feedback. Round and round it goes. Each turn of the loop, the product learns a little more from the very people using it.

The honest summary

None of Ofuma's features are exotic, if I am being honest. Version control, tests, gated releases, A/B experiments, monitoring, rollback: software teams have leaned on every one of these for decades. What is genuinely new is the realization that a prompt is not a harmless little string you can edit on a whim. It is the behavior of your product, written in plain English, and it deserves the exact same discipline as the behavior written in code.

Ofuma is, in the end, that discipline made concrete. Every feature is an answer to a moment where someone changed a prompt, broke something, and then couldn't remember what it used to say.

Test all things; hold fast what is good. - 1 Thessalonians 5:21