A marketer asks mParticle Agent for an audience of customers likely to make a purchase in the next 30 days. The Agent has to determine whether the answer already exists in the customer’s data, whether an existing prediction fits the goal, or whether a new prediction is actually needed. Those paths may sound similar in conversation, but they represent materially different product decisions. The foundation model alone does not know which one mParticle considers the right next step.
Our product team at mParticle had to decide how the agent makes that choice long before any marketer asked. Agents inherit the judgment of the teams that build them. That was a lesson we first learned building mParticle Agent, and adding Predictions made its implications much more concrete. The hard part is expressing that judgment at the right level of detail, specific enough to guide the agent and general enough to cover situations no one has tested.
In my last post, I argued that when an agent takes on an expert’s work, the expert’s judgment has to move into the product, and I covered where that judgment lives inside a multi-agent system. This post is about where it comes from. Our product team turned expert instincts into principles the agent follows, then built evals, repeatable test conversations with a known right answer, to check that it does.
Removing expertise from the user means building it into the agent
With mParticle Agent, marketers can start from a business goal instead of bringing the expertise Predictions used to require. That expertise didn’t disappear. Every class of judgment a marketer or expert previously brought to the workflow, from whether prediction fits the goal to whether the result is appropriate for the intended use, became behavior our product team had to specify.
You can see those decisions in how the agent behaves. If nothing in a marketer’s data cleanly represents the behavior they describe, the agent says what’s missing instead of substituting a similarly named event, because quietly building on the wrong event does more harm than admitting the data isn’t there. Before creation, the Agent checks whether the proposed configuration satisfies the relevant prerequisites and readiness requirements. Once results exist, it separately evaluates whether the prediction appears suitable for the marketer’s intended use. Once usable results are available, it can recommend a target range using the marketer’s stated goal and the available prediction evidence, while leaving the material targeting decision with the marketer. Each of those behaviors started as a judgment call someone on our team made.
Expert instincts have to become explicit principles
Before writing a prompt, we started with a person. We pictured someone who knew when a prediction would add information and when existing customer data already had the answer. That person would check what a marketer already had before recommending a model, surface the evidence that mattered for the goal, and leave the campaign decision with the marketer. An expert does those things without thinking. An agent needs each one spelled out.
When subject-matter experts flagged a response in testing, we asked what general principle would have led them to answer differently, not what wording they preferred. Fixing one bad answer helps one conversation. Fixing the principle behind it helps every conversation like it.
Model quality is where the level of detail mattered most. A tempting implementation would be a simple cutoff, where a high Strength score means proceed and a Moderate one means stop. As the first post explained, that breaks down because the same prediction can be good enough for a cheap test and fall short for an expensive launch. The principle we landed on was to evaluate the available prediction in the context of the marketer’s intended decision, surface the evidence relevant to that decision, and avoid universal quality thresholds. Moderate Strength may be useful for one purpose and insufficient for another; Strong Strength is favorable evidence, not a guarantee of campaign performance. Because it describes what to weigh rather than where to draw a line, it gives the agent something to reason from in situations we never wrote evals for. A cutoff would only have constrained the agent. The principle made it more capable, and that distinction became our test for every proposed change.
The prompt is an implementation, not the specification
Principles like that need somewhere to live. The obvious place is the system prompt, the instructions the model receives on every turn. But when the prompt is the only definition of behavior, every bad response adds another sentence, until the prompt reads like a record of past failures instead of a description of the product.
So our product team kept a separate behavioral specification and wrote the prompt from it. Principles described the judgment we wanted, like preserving the marketer’s intent and staying grounded in available evidence. Patterns described how each principle applied to a type of conversation, along with failures we’d already seen. For Predictions, one pattern covered every place a prediction appears and listed mistakes like describing a customer in the top 10% as having a 90% chance of buying. Evals tested whether the agent actually behaved that way.
Keeping the specification separate also changed how we fixed problems, because we could ask which part of the system actually owned the failure before reaching for the prompt. If the Agent proposes a new prediction even though an existing one should have been reused, the issue might be behavioral guidance, but it could also be retrieval that failed to surface the prediction or an evaluation that assumed context the Agent never had. Similarly, a correctly reasoned proposal can still fail because of rendering, validation, permissions, or product state. A prompt change would be the wrong fix for those problems. Sometimes the right answer was to change nothing. The spec also made the judgment portable. When we moved from a single prompt to a routed system of specialist agents, as I covered in the first post, the runtime architecture changed, but the underlying behavioral principles remained.
Evals judge the decision before the wording
A spec only matters if you can check the agent against it. An agent can write a polished answer and still make the wrong product decision, so every eval named the right decision, such as reusing an existing prediction, proposing a new one, or flagging a quality concern, before any wording was judged. A response can be phrased many ways, while the set of acceptable product decisions is usually much narrower.
We grouped evals into families that held one request constant and varied the conditions that should change the answer. A simplified illustrative family built around the request from the start of this post might look like this:
- A strong purchase prediction already exists — Reuse it instead of creating a new one
- Only a weaker prediction exists, and the marketer mentions a low-cost test — Explain what that strength score means for a test and recommend a range that fits it
- Only a weaker prediction exists, and the marketer mentions an expensive launch — Flag the quality concern before proceeding to audience construction
- No suitable prediction exists — Propose a new prediction and explain when the audience will be ready
- The marketer actually wants recent purchasers — Answer from existing customer data without a model
Rewording the request adds almost nothing to coverage. Changing what’s in the account changes the right answer. Filling in the right-hand column also forced our team to agree on the right call in each case, which is exactly the judgment the agent inherits.
Graders, the models that score eval results, carry judgment too. A grader can reject a good answer because it differs from the sample or pass a polished one that took the wrong path, so ours was told to identify the correct decision first and accept any valid way of expressing it. For Predictions, that could mean checking whether the Agent surfaced recall when it was available and materially relevant to the marketer’s goal. We calibrated the grader against human reviewers before trusting its scores.
Predictions also taught us that good behavioral specifications often preserve distinctions a model might otherwise collapse. Calculation readiness is not model quality. A recommendation is not user acceptance. A selected chart range is not a confirmed targeting decision. Predicted lift is not causal impact. Creating a prediction is not the same as having usable results. Each distinction sounds small, but collapsing one can produce a confident answer that is materially wrong. Part of specifying behavior is deciding which distinctions the product must never smooth away.
Behavioral artifacts ship like code
If the agent’s judgment comes from our product team, the spec, evals, and graders that hold it are part of the product. Changing the model-quality principle changes whether the agent tells a marketer a prediction is good enough for their campaign. So we treated these artifacts like code. Product managers wrote them in familiar docs and spreadsheets, and engineering synced them into GitHub, where every change went through a pull request with review, version history, and rollback. For a product manager, that means the documents you write go straight into production.
Specifying judgment is the product work
Go back to the marketer from the start of this post. Whether the Agent uses known data, reuses an existing prediction, proposes a new one, or raises a quality concern depends on behavioral principles our product team defined, tested, and continues to maintain. Getting those calls right means the marketer gets guidance that better reflects their goal, their data, and the decision in front of them. Agents inherit the judgment of the teams that build them, so the quality of that judgment is the quality of the product.
If you’re building something similar, start with one expert decision your users struggle with. Find the principle beneath the expert’s decision, build an eval family that varies the conditions that should change the answer, and grade the product decision before the wording. Doing that once will teach you more about your agent’s judgment than any prompt rewrite.
.webp)
