Goals and experiments

Goals work with any host that reports them. curb build writes goals and experiments into the bundle (curb.bundle.json), and the host reports per call which goals it reached and which variant it had. On KugelAudio, the results appear in the dashboard; hosting there is in closed alpha and switched on per organization.

A goal says what counts as a successful call, for example “the caller booked a slot”. An experiment splits calls between variants of your agent, for example a formal and a casual greeting, so you can see which variant reaches your goals more often. You declare both in your agent’s code, next to the agent itself, and they go live with your next deployment.

Declare goals and experiments

Add goals and experiments to defineAgent in src/agent/agent.tsx:

import { defineAgent } from "curb";

export const agent = defineAgent({
  app: BookingAgent,
  goals: {
    booked: { description: "The caller booked a webinar slot", tool: "book_slot" },
    felt_easy: {
      description: "Booking felt easy",
      judge: "The caller says booking with the assistant was easy or pleasant.",
    },
  },
  experiments: {
    greeting: {
      description: "Formal vs casual opening",
      variants: { formal: 1, casual: 1 },
      fromClient: "greeting",
    },
  },
});

A goal is one of two kinds:

• Tool goal (tool): reached when the agent ran that tool during the call and the tool succeeded. Name one of your agent’s tools. npx curb check warns when no tool of your agent has that name.
• Judge goal (judge): a sentence describing what should be true of the conversation, 1 to 1000 characters. After the call, a language model reads the transcript and decides whether the sentence is true of the call. See How judge goals are judged.

An experiment lists 2 to 6 variants, each with a weight from 1 to 100. The weights are relative: { formal: 1, casual: 1 } gives each variant half the calls, and { formal: 1, casual: 3 } gives casual three calls in four. Each call gets one variant of every experiment when it starts and keeps it until it ends. The same call always gets the same variant, so you can trace any call back to the variant it had.

The limits:

• Goal, experiment and variant names start with a lowercase letter and have at most 40 lowercase letters, digits and underscores, such as booked or casual_v2.
• At most 20 goals and 5 experiments per agent.
• A description is optional and holds at most 200 characters. It is for the people reading the results; the agent never sees it.

npx curb check reports goals and experiments outside these limits and names every problem, and a deployment with them fails to build, so a mistake never reaches a live call.

How judge goals are judged

Shortly after a call ends, a language model hosted in the EU, at Scaleway in Paris, reads the call’s transcript and answers whether your sentence is true of the call: yes (Reached) or no (Not reached), with a one-sentence reason such as “The caller said booking with the assistant was easy.” The call’s page shows the reason with the result.

The model receives only the transcript and your sentence: nothing else about the call, your agent or your project. Each call is judged once, and its result does not change later.

• The model judges from the transcript alone, and the caller’s words in it are what speech recognition heard. Write the sentence about what was said in the conversation; to check that something happened, such as a booking, use a tool goal.
• The call’s conversation has to be stored. A call whose conversation is not stored has no transcript to judge, and its judge goals are Not evaluable.
• For a very long call the model reads the first 10,000 and the last 30,000 characters of the transcript, about forty minutes of conversation in total.
• When the model cannot answer for a call (it did not answer in time, it was unavailable, or its answer could not be read), that goal is Not evaluable for the call and the reason says why. It never counts as Not reached, and the call’s other goals are decided as usual.

Use the variant in your agent

useVariant returns the variant the current call got, so the agent can behave differently per variant:

import { Instruction, useVariant } from "curb";

function Greeting() {
  return useVariant("greeting") === "casual" ? (
    <Instruction>Greet the caller casually and use their first name when you know it.</Instruction>
  ) : (
    <Instruction>Greet the caller formally.</Instruction>
  );
}

Render it inside your <Agent>, like any other instruction. The variant is set before the agent’s first words, so the greeting can depend on it. useVariant returns undefined for an experiment your agent does not declare.

Choose a variant from the page

fromClient names a key of the context your web page passes to the call. When the call’s context has that key and its value is one of the experiment’s variants, the call gets that variant instead of a random one. Any other value is ignored and the call gets a variant by weight as usual.

With fromClient: "greeting", a widget tag with data-context-params="greeting" lets a link choose the variant: https://example.com/call?greeting=casual always starts a call with the casual greeting. Use this to send one audience to one variant, or to try a variant yourself on the live agent. How a page passes context is in Tell the agent where a call comes from.

Try each variant locally

Every conversation in npx curb dev gets its own variants, chosen the same way as on a live call, so a new conversation can get another variant. To try one variant, add the experiment’s fromClient key to the preview’s address:

http://127.0.0.1:4000/?client.greeting=casual

In a test scenario, experiments fixes the variant for that conversation. An experiment the scenario leaves out gets a fixed variant chosen from the scenario’s file and name, so the same scenario runs with the same variants every time:

import { scenario } from "curb/testing";

export const casualGreeting = scenario({
  experiments: { greeting: "casual" },
  turns: ["Hallo, ich möchte einen Termin buchen."],
  expect: ({ called }) => [called("book_slot")],
});

Keep one scenario per variant for every conversation that has to work in each of them, and run them with npx curb test.

See the results

The dashboard shows goals and experiments; you change them only in code.

• Each call on your agent’s Calls page shows the variant it got in every experiment and, shortly after the call ends, the result of every goal. The calls table has a Goals column and an A/B test filter that lists only the calls of one variant.
• The agent’s Goals page shows each goal of the live version with how it is checked and how often calls reached it over the last 30 days, the same rates for every version, and for each experiment the share of calls, the number of calls and the goal rates of every variant.

Each rate comes with a 95% range. While the ranges of two variants or versions overlap, the page says there are too few calls to tell them apart rather than naming a winner. Calls whose goals are not judged or not evaluable are left out of the rates and counted beside them.

A goal’s result is one of:

Changing a goal or an experiment takes effect with the next deployment. Calls keep the results and variants they had, so the calls of an earlier deployment still show what they were measured against.

What is allowed

• A goal has exactly one of tool or judge. A tool goal is reached when the agent ran that tool and it succeeded; curb check warns (AGENT_GOAL_TOOL_NOT_DECLARED) when no <Tool> has that name. A judge goal is a criterion of 1 to 1000 characters.
• An experiment has 2 to 6 variants with whole-number weights from 1 to 100.
• Names of goals, experiments and variants are a lowercase letter, then up to 39 lowercase letters, digits or underscores. An agent has at most 20 goals and 5 experiments, and a description holds at most 200 characters. defineAgent refuses anything else (AGENT_INVALID_AUTHORING_PROP), naming every problem.

useVariant returns undefined for an experiment the agent does not declare, and in curb say and curb start, which assign no variants. The variant is the session info experiment.<name>, so a firstTurn: { say } function reads it as info["experiment.greeting"].

Next steps

• Deploying: ship the goals and experiments with curb deploy.
• Testing: keep one scenario per variant.