Evals

Test an agent against real situations, including ones it has already been in.

An eval suite is a set of scenarios and how they are scored. A run places a real call per scenario against the agent, with a tester on the other end improvising towards a goal, and scores the transcript against the suite's criteria. A scripted caller only ever tests the path someone already thought of, which is why a scenario is a goal and a persona rather than a script.

The scenarios worth having come from production. Pick a source, look at real completed calls with the sentiment and resolution the analysis pipeline already found, import the interesting ones, and let triage work out what scenario each would make. Triage is batched and resumable because it is a model call per transcript.

Runs are asynchronous: starting one returns a run at pending. Poll it, or subscribe to eval.run.completed and be told.

List suites#

Eval suites for an agent.

GET/api/v1/agents/{agentId}/evals

Path parameters

agentIdstringrequired: Agent id.

Responses

The suites.

Request

curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals" \
  -H "Authorization: Bearer YOUR_API_KEY"

Response200

{
  "data": {
    "evals": [
      {
        "id": "9d02…",
        "name": "Pharmacy follow-up",
        "status": "ready",
        "iterationsPerScenario": 3,
        "scoringCriteria": [
          {
            "key": "accuracy",
            "name": "Accuracy",
            "weight": 0.4
          }
        ],
        "runCount": 12,
        "latestRunScore": 8.1
      }
    ]
  },
  "meta": {
    "requestId": "req_…"
  }
}

Create a suite#

A suite starts empty; add scenarios by hand or import them from real calls.

POST/api/v1/agents/{agentId}/evals

Path parameters

agentIdstringrequired: Agent id.

Body

application/json

name: string; required
description: string
evalType: voice | text
iterationsPerScenario: number
maxCallDurationSeconds: number

Responses

The created suite.

Request

curl -X POST "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Pharmacy follow-up",
    "iterationsPerScenario": 3
  }'

Response201

{
  "data": {
    "id": "9d02…",
    "name": "Pharmacy follow-up",
    "status": "draft"
  },
  "meta": {
    "requestId": "req_…"
  }
}

Get a suite#

One suite, including its scoring criteria and its latest score.

GET/api/v1/agents/{agentId}/evals/{evalId}

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.

Responses

The suite.

Request

curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}" \
  -H "Authorization: Bearer YOUR_API_KEY"

Response200

{
  "data": {
    "id": "9d02…",
    "name": "Pharmacy follow-up",
    "status": "ready",
    "latestRunScore": 8.1
  },
  "meta": {
    "requestId": "req_…"
  }
}

Update a suite#

Change a suite's settings. `noise` puts the tester's line under background sound; pass null to switch it off.

PATCH/api/v1/agents/{agentId}/evals/{evalId}

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.

Body

application/json

name: string
description: string
iterationsPerScenario: number
maxCallDurationSeconds: number
status: draft | ready | archived
noise: { type: cafe|street|static|office, level: light|moderate|heavy } | null

Responses

The updated suite.

Request

curl -X PATCH "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "status": "ready",
    "noise": {
      "type": "office",
      "level": "moderate"
    }
  }'

Response200

{
  "data": {
    "id": "9d02…",
    "status": "ready"
  },
  "meta": {
    "requestId": "req_…"
  }
}

Delete a suite#

Remove a suite and everything under it.

DELETE/api/v1/agents/{agentId}/evals/{evalId}

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.

Responses

Deleted.

Request

curl -X DELETE "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}" \
  -H "Authorization: Bearer YOUR_API_KEY"

Response200

{
  "data": {
    "deleted": true
  },
  "meta": {
    "requestId": "req_…"
  }
}

Add a scenario#

A situation to put the agent in, expressed as a goal the tester pursues.

POST/api/v1/agents/{agentId}/evals/{evalId}/scenarios

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.

Body

application/json

name: string; required
description: string
goal: string; required
persona: string
expectedOutcome: string
context: string
category: happy_path | edge_case | error_handling | adversarial

Responses

The created scenario.

Request

curl -X POST "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/scenarios" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Prescription not ready",
    "persona": "A busy pharmacy technician",
    "goal": "Say the prescription is not ready and refuse to give a date.",
    "category": "edge_case"
  }'

Response201

{
  "data": {
    "id": "31fa…",
    "name": "Prescription not ready"
  },
  "meta": {
    "requestId": "req_…"
  }
}

Import sources#

Campaigns and call centers whose real calls can become scenarios.

GET/api/v1/agents/{agentId}/evals/{evalId}/imports/sources

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.

Responses

The sources.

Request

curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/imports/sources" \
  -H "Authorization: Bearer YOUR_API_KEY"

Response200

{
  "data": {
    "outboundCampaigns": [
      {
        "type": "campaign",
        "id": "a1…",
        "name": "September refill reminders",
        "direction": "outbound",
        "completedCalls": 812,
        "lastCallAt": "2026-09-30T17:42:10.000Z",
        "workspaceName": "Pharmacy"
      }
    ],
    "inboundCampaigns": [],
    "callCenters": [
      {
        "type": "call_center",
        "id": "cc1…",
        "name": "Member services",
        "direction": null,
        "completedCalls": 4302,
        "lastCallAt": "2026-10-01T09:15:03.000Z",
        "workspaceName": null
      }
    ]
  },
  "meta": {
    "requestId": "req_…"
  }
}

Candidate calls#

Completed calls from one source that could become scenarios, with the sentiment, intent and resolution already found, so you can take the unresolved and the unhappy rather than a random sample. Calls already imported into this suite are still listed, with `alreadyImported` set, so pages stay stable while you import. An unreadable `from`, `to` or `direction` is refused rather than ignored.

GET/api/v1/agents/{agentId}/evals/{evalId}/imports/candidates

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.

Query parameters

sourceTypestringrequired: campaign or call_center.; Examplecampaign
sourceIdstringrequired: The source's id.
fromstring: ISO 8601 lower bound, inclusive.
tostring: ISO 8601 upper bound, exclusive.
minDurationnumber: Seconds.
directionstring: inbound or outbound.
searchstring: Free text.
pagenumber: Page number.; Default1
pageSizenumber: Up to 100.; Default25

Responses

A page of candidates.

Request

curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/imports/candidates" \
  -H "Authorization: Bearer YOUR_API_KEY"

Response200

{
  "data": {
    "candidates": [
      {
        "id": "5f1c…",
        "externalRef": "8b7e…",
        "direction": "outbound",
        "phoneNumber": "+15551234567",
        "contactName": "Dana Reyes",
        "durationSeconds": 184,
        "startedAt": "2026-09-29T15:02:44.000Z",
        "endReason": "hangup",
        "sentiment": "negative",
        "callerIntent": "prescription_status",
        "resolved": false,
        "alreadyImported": false
      }
    ],
    "total": 41,
    "page": 1,
    "pageSize": 25
  },
  "meta": {
    "requestId": "req_…"
  }
}

List scenarios#

Every scenario in a suite, written or imported.

GET/api/v1/agents/{agentId}/evals/{evalId}/scenarios

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.

Responses

The scenarios.

Request

curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/scenarios" \
  -H "Authorization: Bearer YOUR_API_KEY"

Response200

{
  "data": {
    "scenarios": [
      {
        "id": "31fa…",
        "name": "Prescription not ready",
        "goal": "Say the prescription is not ready and refuse to give a date.",
        "category": "edge_case",
        "isEnabled": true,
        "isAutoGenerated": false
      }
    ]
  },
  "meta": {
    "requestId": "req_…"
  }
}

List imports#

Batches of real calls pulled into this suite, and how far triage has got. `status` is pending, triaging, ready or failed.

GET/api/v1/agents/{agentId}/evals/{evalId}/imports

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.

Responses

The imports.

Request

curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/imports" \
  -H "Authorization: Bearer YOUR_API_KEY"

Response200

{
  "data": {
    "imports": [
      {
        "id": "77c0…",
        "source": "production",
        "label": "September refill reminders",
        "status": "ready",
        "totalCalls": 18
      }
    ]
  },
  "meta": {
    "requestId": "req_…"
  }
}

Import calls#

Record which calls to read, up to 300. If any id is not a completed call of the source, the whole import is refused with a 422 whose `details.rejectedCallIds` names them.

POST/api/v1/agents/{agentId}/evals/{evalId}/imports

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.

Body

application/json

sourceType: campaign | call_center; required
sourceId: string; required
callIds: string[] (1 to 300); required

Responses

The import.

Request

curl -X POST "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/imports" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "sourceType": "campaign",
    "sourceId": "a1…",
    "callIds": [
      "5f1c…",
      "6a2d…"
    ]
  }'

Response201

{
  "data": {
    "id": "77c0…",
    "status": "pending",
    "totalCalls": 2,
    "triagedCalls": 0
  },
  "meta": {
    "requestId": "req_…"
  }
}

Triage a batch#

Read the next batch of imported calls and work out what scenario each would make. Call it until `remaining` is zero; each batch commits, so a retry resumes rather than restarts. An empty body uses the defaults; an invalid one is refused.

POST/api/v1/agents/{agentId}/evals/{evalId}/imports/{importId}/triage

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.
importIdstringrequired: Import id.

Body

application/json

batchSize: number (1 to 20, default 6)
retryFailed: boolean

Responses

Progress.

Request

curl -X POST "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/imports/{importId}/triage" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "batchSize": 6
  }'

Response200

{
  "data": {
    "import": {
      "id": "77c0…",
      "triagedCalls": 6
    },
    "processed": 6,
    "remaining": 12
  },
  "meta": {
    "requestId": "req_…"
  }
}

Review triaged calls#

What triage found for each imported call (the summary, the category, and the scenario it suggests) so it can be reviewed before anything is created.

GET/api/v1/agents/{agentId}/evals/{evalId}/imports/{importId}/scenarios

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.
importIdstringrequired: Import id.

Responses

The imported calls.

Request

curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/imports/{importId}/scenarios" \
  -H "Authorization: Bearer YOUR_API_KEY"

Response200

{
  "data": {
    "calls": [
      {
        "id": "b1…",
        "triageStatus": "done",
        "triage": {
          "category": "edge_case",
          "summary": "Pharmacy would not commit to a date.",
          "suggestedScenario": {
            "name": "Prescription not ready"
          }
        },
        "scenarioId": null
      }
    ]
  },
  "meta": {
    "requestId": "req_…"
  }
}

Create scenarios from calls#

Turn triaged calls into scenarios. A call already turned into one is skipped rather than duplicated. GET the same path first to review what triage found.

POST/api/v1/agents/{agentId}/evals/{evalId}/imports/{importId}/scenarios

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.
importIdstringrequired: Import id.

Body

application/json

callIds: string[]; required

Responses

What was created.

Request

curl -X POST "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/imports/{importId}/scenarios" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "callIds": [
      "b1…",
      "b2…"
    ]
  }'

Response201

{
  "data": {
    "created": 2,
    "scenarioIds": [
      "31fa…",
      "31fb…"
    ]
  },
  "meta": {
    "requestId": "req_…"
  }
}

Start a run#

Place a call per enabled scenario per iteration. Returns immediately; the run takes minutes. A suite with no enabled scenarios is refused with a 409.

POST/api/v1/agents/{agentId}/evals/{evalId}/runs

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.

Responses

The run, at pending.

Request

curl -X POST "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/runs" \
  -H "Authorization: Bearer YOUR_API_KEY"

Response202

{
  "data": {
    "id": "e91b…",
    "runNumber": 13,
    "status": "pending",
    "totalIterations": 9
  },
  "meta": {
    "requestId": "req_…"
  }
}

List runs#

Every run of a suite, newest first.

GET/api/v1/agents/{agentId}/evals/{evalId}/runs

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.

Responses

The runs.

Request

curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/runs" \
  -H "Authorization: Bearer YOUR_API_KEY"

Response200

{
  "data": {
    "runs": [
      {
        "id": "e91b…",
        "runNumber": 13,
        "status": "completed",
        "agentVersion": 8,
        "overallScore": 8.1
      }
    ]
  },
  "meta": {
    "requestId": "req_…"
  }
}

Get a run#

One run, with the per-criterion breakdown behind its headline score. A score with no breakdown says nothing about what to fix.

GET/api/v1/agents/{agentId}/evals/{evalId}/runs/{runId}

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.
runIdstringrequired: Run id.

Responses

The run and its aggregates.

Request

curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/runs/{runId}" \
  -H "Authorization: Bearer YOUR_API_KEY"

Response200

{
  "data": {
    "run": {
      "id": "e91b…",
      "status": "completed",
      "agentVersion": 8,
      "overallScore": 8.1,
      "completedIterations": 9,
      "failedIterations": 0
    },
    "aggregates": {
      "overallScore": 8.1,
      "criterionAggregates": [
        {
          "criterionKey": "accuracy",
          "criterionName": "Accuracy",
          "avgScore": 8.6,
          "minScore": 7,
          "maxScore": 10,
          "stdDev": 1.1,
          "weight": 0.4
        }
      ],
      "totalResults": 9,
      "completedResults": 9,
      "failedResults": 0,
      "avgDurationSeconds": 102.4,
      "totalCost": 1.87
    }
  },
  "meta": {
    "requestId": "req_…"
  }
}

Delete a run#

Remove a run and its results.

DELETE/api/v1/agents/{agentId}/evals/{evalId}/runs/{runId}

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.
runIdstringrequired: Run id.

Responses

Deleted.

Request

curl -X DELETE "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/runs/{runId}" \
  -H "Authorization: Bearer YOUR_API_KEY"

Response200

{
  "data": {
    "deleted": true
  },
  "meta": {
    "requestId": "req_…"
  }
}

Run results#

One result per scenario per iteration: the transcript of the call that was placed, what it scored on each criterion, and why. A score nobody can audit is a score nobody should act on.

GET/api/v1/agents/{agentId}/evals/{evalId}/runs/{runId}/results

Path parameters

agentIdstringrequired: Agent id.
evalIdstringrequired: Suite id.
runIdstringrequired: Run id.

Responses

The results.

Request

curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/runs/{runId}/results" \
  -H "Authorization: Bearer YOUR_API_KEY"

Response200

{
  "data": {
    "results": [
      {
        "id": "aa10…",
        "evalScenarioId": "31fa…",
        "iterationNumber": 1,
        "status": "completed",
        "overallScore": 8.5,
        "durationSeconds": 96,
        "scores": [
          {
            "criterionKey": "accuracy",
            "criterionName": "Accuracy",
            "score": 9,
            "maxScore": 10,
            "weight": 0.4,
            "reasoning": "Gave the correct pickup window and did not invent a date.",
            "evidence": "I can't give you a date yet, but the pharmacy will text you."
          }
        ],
        "scenario": {
          "id": "31fa…",
          "name": "Prescription not ready",
          "goal": "Say the prescription is not ready and refuse to give a date.",
          "category": "edge_case"
        }
      }
    ]
  },
  "meta": {
    "requestId": "req_…"
  }
}