Case study · 03
DriftWatch
A self-observing AI agent SDK.
- Role
- Creator & maintainer
- Period
- Jul 2026 – present
- Stack
- TypeScript, OpenTelemetry, Prometheus, Vercel AI SDK, SigNoz, pnpm workspaces
- Status
- Open source
01 The problem
An agent that worked yesterday can behave differently today. The tool-call mix shifts, errors creep up, latency and token spend grow, and nobody notices until a user does.
Traditional monitoring sees requests and errors, not behaviour. DriftWatch treats the agent's own steps as telemetry and watches them for drift.
02 What I built
- agent (vercel ai sdk)
agent (vercel ai sdk): Your agent, your providers
- @driftwatch/sdk
@driftwatch/sdk: Every tool call and model step as an OpenTelemetry span
- otel export
otel export: SigNoz or any OTLP backend
- drift judge
drift judge: Prometheus window deltas (tool mix, errors, p95 latency, tokens), judged by a model
- @driftwatch/autopilot
@driftwatch/autopilot: Notify, pause, roll back or throttle
- human approval
human approval: Required for destructive actions
- server + console
server + console: @driftwatch/server, @driftwatch/console
DriftWatch wraps an agent built on the Vercel AI SDK. Every tool call and model step becomes an OpenTelemetry span, so the agent's behaviour shows up in SigNoz or any OTLP backend next to the rest of the system.
The drift detector queries a Prometheus-compatible backend for two time windows and computes the change in tool-call mix, error rate, p95 latency and token spend. A model (any AI SDK model you pass in) then judges whether that change is drift and returns a schema-typed verdict with a severity, reasons and a recommended action.
When drift crosses a policy, @driftwatch/autopilot responds. It can notify, pause, roll back or throttle, and destructive actions wait for a human to approve them. @driftwatch/server and @driftwatch/console give the team a place to see and act on it.
03 Decisions and trade-offs
Decision 01
Chose Layering over the Vercel AI SDK over shipping provider SDKs.
Why: Teams keep the providers and models they already use; DriftWatch brings no provider SDKs of its own.
Cost: Agents built on other frameworks need an adapter.
Decision 02
Chose OpenTelemetry as the trace format over a custom event schema.
Why: Traces land in backends teams already run, such as SigNoz.
Cost: Agent concepts have to be mapped onto generic span attributes.
Decision 03
Chose Human approval for destructive autopilot actions over fully automatic remediation.
Why: Pausing or rolling back an agent in production should never surprise its owners.
Cost: Slower response when a person is not around.
04 Results
What shipped
- MIT-licensed pnpm monorepo (Node ≥ 22) with @driftwatch/sdk, @driftwatch/autopilot, @driftwatch/server and @driftwatch/console.
- The SDK reads no environment variables itself; configuration is explicit.
- Docs at drift-watch-docs.vercel.app and two articles on drift detection with SigNoz.
Want this kind of work on your team?