Two years ago, I had a problem. I was building AI-powered features into SaaS products and every time something went wrong in production, I was flying blind. The API returned HTTP 200 OK. The logs showed success. But the model hallucinated a function that does not exist, or looped through the same tool call 400 times burning $50 in tokens.
Standard monitoring tools treat LLMs like slow databases. They track latency, token usage, and uptime. But they cannot tell you why the model made a specific decision, whether its output is actually grounded in reality, or whether the reasoning chain is about to spiral into an infinite loop.
I built Observyze to solve this. Not as a side project - as a production platform that real teams use to monitor, evaluate, and govern their AI systems. Here are the lessons that mattered.
Lesson 1: The SDK is the Product
The most important engineering decision was not the backend architecture - it was the developer experience. A developer should be able to add observability to their existing codebase in under 2 minutes. Zero refactoring. One line of code:
import { ObservyzeClient } from "@observyze/sdk"
import OpenAI from "openai"
const nw = new ObservyzeClient({ apiKey: process.env.OBSERVYZE_API_KEY! })
const openai = nw.wrap(new OpenAI())
// Every call is now automatically traced
The wrap() method intercepts the OpenAI client, captures input/output/latency/tokens/cost, and ships the trace asynchronously. It supports OpenAI, Anthropic, Google Gemini, Vercel AI SDK, LangChain, and any OpenAI-compatible endpoint.
The Performance Constraint
The SDK must add less than 1ms of overhead per LLM call. At high throughput, TypeScript GC pauses and event-loop contention become real problems. The solution: write the core buffering layer in a systems language, compile it to a native Node.js addon, and handle serialization, batching, and retry logic there. The TypeScript layer stays thin - just the developer-facing API.
Every architectural decision serves this goal: the binary trace format instead of JSON, the memory-mapped ring buffer for zero-copy handoff, the exponential backoff with jitter on network failures. If the SDK adds latency, developers will remove it. If it is invisible, they will keep it.
Lesson 2: Most AI Problems Have Non-AI Solutions
When I started building the hallucination detection engine, the instinct was to use GPT-4 to grade GPT-4 output. The industry standard. It fails for three reasons:
-
The Implementation Detail Paradox: In code generation, introducing valid implementation details not present in the prompt is NOT a hallucination - it is the entire point. A naive judge flags
import httpxas hallucinated because the user never mentionedhttpx. -
Cost and Latency: Running GPT-4 to evaluate every trace doubles your token bill and adds seconds of latency. You are spending $10 to verify $5 of work.
-
Prompt Injection: An adversary can craft outputs that manipulate the judge model into giving high scores.
The breakthrough was realizing that most evaluations do not need a language model at all. Syntax validation catches the majority of code hallucinations at zero cost. Natural Language Inference (NLI) models - small, local, fast - handle the bulk of RAG grounding checks without calling any API. Only the hard 20% of cases needs expensive frontier models.
This layered approach - deterministic checks first, local models second, frontier models last - reduced evaluation cost by 90% while improving accuracy. The lesson applies broadly: before reaching for an LLM, ask whether a simpler tool solves the problem.
Lesson 3: Security Architecture Emerges from Constraints
The proxy architecture was not in the original plan. It emerged when an enterprise customer said: we cannot route our API keys through a third-party proxy. Fair. But we still needed the safety checks.
The solution was credential sharding: the SDK swaps the provider API key for an Observyze token, moves the original key to a separate header, and the gateway validates the Observyze token before forwarding to the provider. The provider key is used in-memory only and never stored in logs or traces.
This constraint produced a better architecture than the original design. It led to Compliance Mode - a configuration where the proxy is bypassed entirely, PII redaction runs locally in the customer VPC, and no raw prompts or completions leave their infrastructure. Traces contain only metadata: latency, tokens, cost, model, timestamps.
The lesson: security features that emerge from real customer constraints are more robust than features designed in isolation. Build the constraint into the architecture, not around it.
Lesson 4: The Neutral Class Changes Everything
In hallucination detection, the industry default is binary: the output is either grounded or hallucinated. This creates a 90%+ false positive rate because it conflates two very different situations:
- Contradicted: The output directly conflicts with the source documents. This is a real hallucination.
- Neutral: The source documents simply do not contain the answer. The model might be correct - it is just not verifiable from the provided context.
A RAG system that retrieves three documents about cloud pricing might generate a correct answer about a feature not mentioned in those documents. Binary evaluation flags this as a hallucination. A three-class system correctly identifies it as neutral - needs review, not a failure.
This single architectural change - adding a neutral class - eliminated the majority of false positives and made the evaluation results actually actionable. Teams stopped ignoring the alerts because most of them were real.
Lesson 5: Circuit Breakers Are the Real Safety Feature
Tracing and evaluation are observability. Circuit breakers are governance.
When the evaluation engine detects a persistent problem - a specific model or prompt consistently producing contradicted outputs - it can automatically trip a circuit breaker. Subsequent requests in the affected scope are rejected. Alerts fire to Slack or Discord. The circuit resets after a cooldown period.
This prevents a failing agent from burning through your budget while the team investigates. It is the difference between knowing something went wrong (observability) and stopping it from going wrong (governance).
The design tradeoff: circuit breakers add latency on the check path and can cause false rejections if thresholds are too aggressive. The solution is configurable thresholds per org, with clear reason codes on every rejection so teams can tune without guessing.
Lesson 6: Compliance is a Spectrum, Not a Toggle
Enterprise customers need SOC2-ready observability. But compliance is not a binary switch - it is a spectrum of controls:
- Client-side PII redaction: Deterministic regex pipeline scrubs emails, credit cards, SSNs, API keys before data leaves the server
- Server-side gatekeeping: A secondary parser catches anything the client-side missed
- Zero-Data-Leak Mode: No raw data ever leaves the customer VPC - metadata only
- Cold storage encryption: Archived traces encrypted at rest
- Access control: Role-based access with SSO/SAML for large teams
- Audit trail: Every evaluation result stores the exact inputs, model versions, and rubric used - full reproducibility
The key insight: no single control makes you compliant. Compliance is defense in depth - multiple layers where each one assumes the others might fail. Build every layer independently, and let customers choose which combination matches their requirements.
What I Would Do Differently
Start with the evaluation engine, not the dashboard
I built the dashboard first because it was visually satisfying. But the dashboard is the easy part. The hard part - and the real value - is the evaluation engine. If I started over, I would build the evaluation pipeline first, prove it works on real traces, and then build the dashboard to visualize its results.
Invest in deterministic checks earlier
Every month I delayed deterministic validation (syntax checks, import resolution, format verification) was a month of unnecessary LLM costs. These checks are fast, free, and catch the majority of issues. They should be the foundation, not an afterthought.
Make compliance mode the default
I treated Compliance Mode as an enterprise feature. It should have been the default from day one. Most teams care about data governance, not just the Fortune 500. Making compliance the default - with an opt-out for teams that do not need it - would have accelerated enterprise adoption.
The Stack (High Level)
I am intentionally not publishing the exact microservice breakdown, internal service names, or infrastructure specifics. That is implementation detail that competitors can copy and customers do not need.
What I will say: the platform runs on a distributed architecture with separate services for API routing, trace ingestion, evaluation, cost attribution, and real-time streaming. The SDK is open-source on npm. The evaluation engine uses a hybrid approach combining deterministic checks, local NLI models, and multi-model consensus for high-stakes decisions.
Try It
Observyze is live at observyze.com. The Node.js SDK is open-source:
npm install @observyze/sdk
If you are building AI features in production and need to know whether your model is actually telling the truth, give it a try.
Built as a full-stack project: TypeScript API gateway, Python evaluation engine, native SDK core, Next.js dashboard, all deployed on AWS. View the architecture at observyze.com/docs or get in touch to discuss AI infrastructure.