API Docs

Evals

Test an agent against real situations, including ones it has already been in.

An eval suite is a set of scenarios and how they are scored. A run places a real call per scenario against the agent, with a tester on the other end improvising towards a goal, and scores the transcript against the suite's criteria. A scripted caller only ever tests the path someone already thought of, which is why a scenario is a goal and a persona rather than a script.

The scenarios worth having come from production. Pick a source, look at real completed calls with the sentiment and resolution the analysis pipeline already found, import the interesting ones, and let triage work out what scenario each would make. Triage is batched and resumable because it is a model call per transcript.

Runs are asynchronous: starting one returns a run at `pending`. Poll it, or subscribe to `eval.run.completed` and be told.

List suites

Eval suites for an agent.

GET/api/v1/agents/{agentId}/evals

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

Responses

{
  "data": {
    "evals": [
      {
        "id": "9d02…",
        "name": "Pharmacy follow-up",
        "status": "ready",
        "iterationsPerScenario": 3,
        "scoringCriteria": [
          {
            "key": "accuracy",
            "name": "Accuracy",
            "weight": 0.4
          }
        ],
        "runCount": 12,
        "latestRunScore": 8.1
      }
    ]
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals" \
  -H "Authorization: Bearer YOUR_API_KEY"

Create a suite

A suite starts empty; add scenarios by hand or import them from real calls.

POST/api/v1/agents/{agentId}/evals

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

Responses

{
  "data": {
    "id": "9d02…",
    "name": "Pharmacy follow-up",
    "status": "draft"
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X POST "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Pharmacy follow-up",
    "iterationsPerScenario": 3
  }'

Get a suite

One suite, including its scoring criteria and its latest score.

GET/api/v1/agents/{agentId}/evals/{evalId}

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

Responses

{
  "data": {
    "id": "9d02…",
    "name": "Pharmacy follow-up",
    "status": "ready",
    "latestRunScore": 8.1
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}" \
  -H "Authorization: Bearer YOUR_API_KEY"

Update a suite

Change a suite's settings. `noise` puts the tester's line under background sound; pass null to switch it off.

PATCH/api/v1/agents/{agentId}/evals/{evalId}

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

Responses

{
  "data": {
    "id": "9d02…",
    "status": "ready"
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X PATCH "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "status": "ready",
    "noise": {
      "type": "office",
      "level": "moderate"
    }
  }'

Delete a suite

Remove a suite and everything under it.

DELETE/api/v1/agents/{agentId}/evals/{evalId}

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

Responses

{
  "data": {
    "deleted": true
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X DELETE "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}" \
  -H "Authorization: Bearer YOUR_API_KEY"

Add a scenario

A situation to put the agent in, expressed as a goal the tester pursues.

POST/api/v1/agents/{agentId}/evals/{evalId}/scenarios

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

Responses

{
  "data": {
    "id": "31fa…",
    "name": "Prescription not ready"
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X POST "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/scenarios" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Prescription not ready",
    "persona": "A busy pharmacy technician",
    "goal": "Say the prescription is not ready and refuse to give a date.",
    "category": "edge_case"
  }'

Import sources

Campaigns and call centers whose real calls can become scenarios.

GET/api/v1/agents/{agentId}/evals/{evalId}/imports/sources

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

Responses

{
  "data": {
    "outboundCampaigns": [
      {
        "id": "a1…",
        "name": "September refill reminders",
        "callCount": 812
      }
    ],
    "inboundCampaigns": [],
    "callCenters": [
      {
        "id": "cc1…",
        "name": "Member services",
        "callCount": 4302
      }
    ]
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/imports/sources" \
  -H "Authorization: Bearer YOUR_API_KEY"

Candidate calls

Completed calls from one source that could become scenarios, with the sentiment, intent and resolution already found — so you can take the unresolved and the unhappy rather than a random sample. Calls already imported are excluded.

GET/api/v1/agents/{agentId}/evals/{evalId}/imports/candidates

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

Query parameters

ParameterTypeDescription
sourceTyperequired
string

campaign or call_center.

sourceIdrequired
string

The source's id.

from
string

ISO date lower bound.

to
string

ISO date upper bound.

minDuration
number

Seconds.

direction
string

inbound or outbound.

search
string

Free text.

page
number

Page number.

Default: 1

pageSize
number

Up to 100.

Default: 25

Responses

{
  "data": {
    "calls": [
      {
        "id": "5f1c…",
        "direction": "outbound",
        "durationSeconds": 184,
        "sentiment": "negative",
        "callerIntent": "prescription_status",
        "resolved": false
      }
    ],
    "total": 41
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/imports/candidates" \
  -H "Authorization: Bearer YOUR_API_KEY"

List scenarios

Every scenario in a suite, written or imported.

GET/api/v1/agents/{agentId}/evals/{evalId}/scenarios

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

Responses

{
  "data": {
    "scenarios": [
      {
        "id": "31fa…",
        "name": "Prescription not ready",
        "goal": "Say the prescription is not ready and refuse to give a date.",
        "category": "edge_case",
        "isEnabled": true,
        "isAutoGenerated": false
      }
    ]
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/scenarios" \
  -H "Authorization: Bearer YOUR_API_KEY"

List imports

Batches of real calls pulled into this suite, and how far triage has got.

GET/api/v1/agents/{agentId}/evals/{evalId}/imports

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

Responses

{
  "data": {
    "imports": [
      {
        "id": "77c0…",
        "source": "production",
        "label": "September refill reminders",
        "status": "triaged",
        "totalCalls": 18
      }
    ]
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/imports" \
  -H "Authorization: Bearer YOUR_API_KEY"

Import calls

Record which calls to read. Ids that do not belong to the source are rejected rather than imported.

POST/api/v1/agents/{agentId}/evals/{evalId}/imports

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

Responses

{
  "data": {
    "id": "77c0…",
    "status": "pending",
    "totalCalls": 2,
    "triagedCalls": 0
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X POST "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/imports" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "sourceType": "campaign",
    "sourceId": "a1…",
    "callIds": [
      "5f1c…",
      "6a2d…"
    ]
  }'

Triage a batch

Read the next batch of imported calls and work out what scenario each would make. Call it until `remaining` is zero; each batch commits, so a retry resumes rather than restarts.

POST/api/v1/agents/{agentId}/evals/{evalId}/imports/{importId}/triage

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

importIdrequired
string

Import id.

Responses

{
  "data": {
    "import": {
      "id": "77c0…",
      "triagedCalls": 6
    },
    "processed": 6,
    "remaining": 12
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X POST "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/imports/{importId}/triage" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "batchSize": 6
  }'

Review triaged calls

What triage found for each imported call — the summary, the category, and the scenario it suggests — so it can be reviewed before anything is created.

GET/api/v1/agents/{agentId}/evals/{evalId}/imports/{importId}/scenarios

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

importIdrequired
string

Import id.

Responses

{
  "data": {
    "calls": [
      {
        "id": "b1…",
        "triageStatus": "done",
        "triage": {
          "category": "edge_case",
          "summary": "Pharmacy would not commit to a date.",
          "suggestedScenario": {
            "name": "Prescription not ready"
          }
        },
        "scenarioId": null
      }
    ]
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/imports/{importId}/scenarios" \
  -H "Authorization: Bearer YOUR_API_KEY"

Create scenarios from calls

Turn triaged calls into scenarios. A call already turned into one is skipped rather than duplicated. GET the same path first to review what triage found.

POST/api/v1/agents/{agentId}/evals/{evalId}/imports/{importId}/scenarios

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

importIdrequired
string

Import id.

Responses

{
  "data": {
    "created": 2,
    "scenarioIds": [
      "31fa…",
      "31fb…"
    ]
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X POST "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/imports/{importId}/scenarios" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "callIds": [
      "b1…",
      "b2…"
    ]
  }'

Start a run

Place a call per scenario per iteration. Returns immediately — the run takes minutes.

POST/api/v1/agents/{agentId}/evals/{evalId}/runs

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

Responses

{
  "data": {
    "id": "e91b…",
    "runNumber": 13,
    "status": "pending",
    "totalIterations": 9
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X POST "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/runs" \
  -H "Authorization: Bearer YOUR_API_KEY"

List runs

Every run of a suite, newest first.

GET/api/v1/agents/{agentId}/evals/{evalId}/runs

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

Responses

{
  "data": {
    "runs": [
      {
        "id": "e91b…",
        "runNumber": 13,
        "status": "completed",
        "agentVersion": 8,
        "overallScore": 8.1
      }
    ]
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/runs" \
  -H "Authorization: Bearer YOUR_API_KEY"

Get a run

One run, with the per-criterion breakdown behind its headline score. A score with no breakdown says nothing about what to fix.

GET/api/v1/agents/{agentId}/evals/{evalId}/runs/{runId}

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

runIdrequired
string

Run id.

Responses

{
  "data": {
    "run": {
      "id": "e91b…",
      "status": "completed",
      "agentVersion": 8,
      "overallScore": 8.1,
      "completedIterations": 9,
      "failedIterations": 0
    },
    "aggregates": {
      "criteria": [
        {
          "criterionKey": "accuracy",
          "averageScore": 8.6,
          "maxScore": 10
        }
      ]
    }
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/runs/{runId}" \
  -H "Authorization: Bearer YOUR_API_KEY"

Delete a run

Remove a run and its results.

DELETE/api/v1/agents/{agentId}/evals/{evalId}/runs/{runId}

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

runIdrequired
string

Run id.

Responses

{
  "data": {
    "deleted": true
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X DELETE "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/runs/{runId}" \
  -H "Authorization: Bearer YOUR_API_KEY"

Run results

One result per scenario per iteration: the transcript of the call that was placed, what it scored on each criterion, and why. A score nobody can audit is a score nobody should act on.

GET/api/v1/agents/{agentId}/evals/{evalId}/runs/{runId}/results

Path parameters

ParameterTypeDescription
agentIdrequired
string

Agent id.

evalIdrequired
string

Suite id.

runIdrequired
string

Run id.

Responses

{
  "data": {
    "results": [
      {
        "id": "aa10…",
        "evalScenarioId": "31fa…",
        "iterationNumber": 1,
        "status": "completed",
        "overallScore": 8.5,
        "durationSeconds": 96
      }
    ]
  },
  "meta": {
    "requestId": "req_…"
  }
}
Request
curl -X GET "https://voice.evryhealth.com/api/v1/agents/{agentId}/evals/{evalId}/runs/{runId}/results" \
  -H "Authorization: Bearer YOUR_API_KEY"