The Problem:
Knowledge Is Scattered

• Data tables — metric logs, dimensional breakdowns, time series across tenant tiers and connector types
• DS domain knowledge — metric formulas, co-movement patterns, what "normal" looks like per segment — lives in people's heads
• Eng domain knowledge — what systems changed, connector health, model rollouts, experiment ramps — scattered across teams

Every investigation re-assembles the same puzzle from scratch. The structured legwork repeats, while the judgment calls that matter get squeezed.

Why a 4-Stage Pipeline?

Because that's how a senior DS actually investigates. The pipeline encodes that discipline.

Demo: A Real Investigation

Click Quality drops 6.2% for Standard tier. The tool runs:

Real Output: Investigation Report

Structured output ready to drop into Slack or an incident review — root cause, evidence, confidence, and next steps.

Full Transparency:
Execution Trace

Every decision is logged. Reviewers can expand any phase to see exactly what the tool did and why.

Total: 12.3s · 4 phases · 11 steps · Full audit trail for every investigation

The Domain:
Enterprise Search

A multi-layered system where a change at any level ripples through every metric downstream.

The tool encodes domain knowledge as structured YAML — metric formulas, co-movement patterns, known incident signatures, hypothesis priority order — not just prompt instructions.

How It Works Under the Hood

The Builder Journey

Not vibe coding. This system was built by giving AI the right context — specs, tests, and domain knowledge — so it could build something complex on my behalf.

Domain Expert on Demand

Domain-specific AI agents fail without expert pressure-testing. Generic LLMs get easy cases right and hard cases catastrophically wrong.

One review caught that the system's quality checks were weakest at the most critical stage — the final diagnosis. That single finding reshaped the entire v2.

v2: Better Diagnosis Quality,
Structurally Enforced

v1 worked but relied on the AI "following instructions." v2 makes quality checks structural — the system validates its own reasoning at every stage.

Building Rigor, Honestly

Rigorous offline and online eval is still being built. Here's where we are.

Why AI-Driven DS
Solutions Are Hard

Building it is 30%. Getting people to trust and use it is 70%.

The critical path: show the work, earn IC trust, start with structured legwork, add judgment later.

What's Next

North star: a tool that earns trust by being transparent, domain-accurate, and honest about what it doesn't know.

What the Tool Knows
About Your System

Domain knowledge is encoded as structured YAML — not prompt instructions. The tool reasons from data, not vibes.

# The AI adoption trap — encoded as a rule, not a suggestion
pattern: ai_answers_working
signal:
  click_quality: DOWN
  search_quality_success: STABLE or UP
  ai_trigger: UP
verdict: POSITIVE — AI answers cannibalizing clicks by design
action: Do NOT treat as regression

# Connector outage signature — skip 30min of decomposition
pattern: connector_auth_expiry
signal:
  zero_result_rate: SPIKE
  connector_health: FAILURES in single connector
  onset: gradual
shortcut: Jump to connector root cause directly

5 YAML knowledge files: metric definitions, co-movement patterns, historical incidents, hypothesis priority, architecture tradeoffs.

v2 Architecture:
How Enforcement Works

Quality rules run at stage boundaries — not as prompt instructions the AI might ignore.

# Seam validator: runs between HYPOTHESIZE → DISPATCH
rule: hypotheses_consistent_with_co_movement
check: "If AI metrics stable, no hypothesis may
        attribute click_quality drop to AI adoption"
on_violation: REJECT hypothesis, log to trace

# Seam validator: runs at SYNTHESIZE output
rule: effect_size_proportionality
check: "Root cause segment contribution must be
        proportional to claimed severity (P0/P1/P2)"
on_violation: RETRY once, then SOFT WARN

• 11 rules across 4 stage boundaries — from data quality gates to narrative coherence checks
• Tiered enforcement — hard fail early (data parsing), retry + soft warn late (final diagnosis)
• Full trace — every decision logged: which direction was "bad," which hypotheses tested, what conclusion drawn

Build Your Own:
The Reusable Pattern

This isn't a one-off project. It's a pattern any eng team can follow to build domain-specific AI agents.