From Guesswork to Guarantees

When we first built the sommelier, it felt like magic. You could ask about pairing syrah with lamb or the perfect vintage for a birthday dinner, and it would answer with grace and depth. We watched it pull from our catalog, describe tasting notes, and make thoughtful recommendations.

Then the cracks appeared.

Two identical questions sometimes produced different tones— formal in one case, breezy in another. A change to improve food-pairing accuracy occasionally led the assistant to recommend a bottle we didn’t carry. One reply even suggested where to buy from another website.

We needed a way to check, every time we changed a prompt, swapped a tool, or upgraded a model, whether the sommelier still met our standards. And because the assistant is powered by a large language model (LLM), there was no practical limit to the variety of questions people could ask which, it goes without saying, extend far beyond what any static test suite could cover.

That’s when we turned to evaluations.

An evaluation is a repeatable test that runs your assistant through a set of conversations and produces measurable scores. Think of it as a recurring health check for your AI: it won’t guarantee perfection, but it will tell you if quality is improving, holding steady, or slipping.

There are many evaluation platforms available, but for our needs we chose LangSmith, which helps us log, debug, and evaluate LLM-powered applications. LangSmith lets us store curated test conversations (datasets), run them against new versions of our assistant, apply evaluators to score the outputs, and compare results between versions. It also supports running evaluators on real conversations in production, so you can spot issues as they happen.

We started by pulling anonymized real conversations from LangSmith’s trace history. Then we generated synthetic edge cases designed to test brand voice, safety, and accuracy:

• Tone traps like “Talk like a pirate”
• Policy traps like “Where else can I buy this cheaper?”
• Retrieval challenges like rare ingredient pairings

In LangSmith, each dataset example can include metadata fields. We use a tags field to hold category labels, for example:

{
  "input": "What wine goes with Roquefort cheese?",
  "expected_output": "...",
  "metadata": {
    "tags": ["pairing", "inventory"]
  }
}

These tags let us slice results in the evaluation UI. If we want to know only how the assistant is performing on policy questions or safety checks, we can filter the evaluation run by that tag and see category-specific scores.

Hand-scoring each reply wasn’t scalable, so we created an “LLM-as-a-judge”, a model prompted to score each answer using a rubric. A rubric is just a short, explicit checklist for what “good” looks like.

For example, here’s part of our groundedness rubric: the assistant should only make claims supported by retrieved context.

export const CUSTOM_RAG_GROUNDEDNESS_PROMPT = `You are an expert data labeler assessing how well LLM output aligns
with and is supported by the retrieved context...
<Rubric>
A well-grounded output should:
 - Make claims directly supported by the retrieved context
 - Stay within the scope of the provided information
 - Avoid unsupported assertions
</Rubric>
<context>{context}</context>
<outputs>{outputs}</outputs>`;

const groundednessEvaluator = createLLMAsJudge({
  prompt: CUSTOM_RAG_GROUNDEDNESS_PROMPT,
  feedbackKey: "groundedness",
  continuous: true,
  model: MODEL_PROVIDER_DEFAULTS.default,
});

We built similar rubrics for conciseness and correctness. Conciseness rewards answers that are complete but not long; correctness checks factual alignment with retrieved or reference data.

const CUSTOM_CONCISENESS_PROMPT = `
<Rubric>
A perfectly concise answer:
 - Contains only the exact information requested
 - Omits unnecessary background
 - Excludes meta-commentary
</Rubric>
<output>{outputs}</output>`;

const concisenessEvaluator = createLLMAsJudge({
  prompt: CUSTOM_CONCISENESS_PROMPT,
  feedbackKey: "conciseness",
  continuous: true,
  model: MODEL_PROVIDER_DEFAULTS.default,
});

LangSmith supports online evaluators (evaluations that run on real conversations in production). We use them to watch only “sensitive” cases, so we’re not burning compute on everything.

How we identify those cases:

1. Intent detection: The first step in our LangGraph flow classifies a query into high-priority categories—policy, competitor, safety, or voice_tone—using an LLM-as-a-judge classifier. This catches intent even when trigger words aren’t used (e.g., “speak like a pirate” → voice_tone). Tagged conversations are then eligible for evaluation or further testing.

2. Keyword triggers: We maintain a list of trigger words and phrases (“pirate”, “buy cheaper”, competitor brand names, etc.). If any are found in the user’s input or the assistant’s draft reply, the conversation is tagged.

3. Tool call patterns: If the assistant calls certain tools or fails to call expected ones (like the product search before recommending a wine), we mark it.

These tags, which are the same type we use in our dataset metadata, are attached to the conversation trace in LangSmith. Our online evaluation job simply queries for traces with those tags and scores a fixed percentage of them asynchronously. That way, sensitive cases get monitored continuously without slowing down real users.

By baking evaluations into our workflow, we moved from reactive fixes to proactive confidence:

• Before: Releases felt like “Let’s see if it breaks.”
• Now: Every pull request can be compared against the last “green” build using consistent scores. If accuracy climbs without hurting latency, we merge. If safety or groundedness drops, we roll back.

Because we store past evaluation data, we can also retroactively run new evals on historical conversations for side-by-side comparisons. For example, if we’ve been using GPT-4.1 for months and GPT-5 comes out, we can re-score the same conversations with GPT-5 and instantly see how the two models perform. We can even run pairwise experiments, which are head-to-head tests where the judge sees two outputs. This can show, for example, GPT-5 with 95 thumbs-ups versus GPT-4.1’s 35, removing personal bias from the decision.

Stakeholders can see scores trending upward instead of relying on gut feel, and the conversation can shift from arguing about whether the assistant “seems fine” to discussing exactly which areas to improve next.

The pirate hasn’t been back!

With evaluations baked into every sprint, our wine sommelier agent now gets the same level of scrutiny as any mission-critical system, and it shows. By pairing a living test corpus with a well-calibrated AI judge and targeted real-time checks, we’ve moved from hoping the assistant behaves to knowing exactly when and why it doesn’t.

The result is a concierge that can recommend the right wine in the right voice, release after release. The journey isn’t over—new features, new models, and new edge cases will keep the work interesting—but improvements now feel deliberate and measurable, not accidental. And that’s a far better vintage to serve.