# Vishvdeep Dashadiya (AI with VD), Lead AI Engineer

> I design, build, and deploy intelligent products with rigorous engineering, clear metrics, and a calm delivery cadence. From agentic workflows to resilient MLOps, we move from prototype to scale without chaos.

Portfolio, services, case studies and technical blog. Core focus: Agentic AI, real-time ML systems, and cloud-native infrastructure. Areas of expertise: Machine Learning, Generative AI, LLM Engineering, Agentic Systems, MLOps, RAG, LangGraph, FastAPI.

## About

I design, build, and deploy intelligent products with rigorous engineering, clear metrics, and a calm delivery cadence. From agentic workflows to resilient MLOps, we move from prototype to scale without chaos.

Core focus: Agentic AI, real-time ML systems, and cloud-native infrastructure.

- Website: https://aiwithvd.com
- Email: contact@aiwithvd.com
- LinkedIn: https://www.linkedin.com/in/vishvdeep/
- GitHub: https://github.com/aiwithvd

Engagements:
- Strategy sprints and AI roadmap design
- Production-grade LLM applications
- MLOps architecture and reliability

## Expertise

### Machine Learning
Production-grade ML systems built for measurable business impact.
- Problem framing and model strategy
- CV and NLP solution design
- Performance tuning and reliability

### GenAI Engineering
LLM-powered products that scale from idea to deployment.
- LLM integration and evaluation
- Agentic workflows and copilots
- Content and image generation pipelines
- End-to-end AI product delivery

### MLOps
Operational excellence for models in the wild.
- Automated ML pipelines and CI/CD
- Monitoring, versioning, and governance
- Cloud-native deployment strategy

## Experience

### Lead AI Engineer, WeyBee Solutions Pvt Ltd
Aug 2026 - Present · On-site
Leading AI engineering efforts, architecting intelligent systems and driving end-to-end delivery of production AI products.

### Senior Software Engineer - AI/ML, BankBenchers
Nov 2025 - Jul 2026 · Vadodara, Gujarat, India - Remote
Leading AI/ML initiatives with a focus on agentic systems, cloud-native deployment, and scalable production workflows.

### Board of Studies - External Industry Expert, Atmiya University
May 2024 - Present · Rajkot, Gujarat, India - Hybrid
Advising academic curriculum and industry alignment for AI and data science programs, bridging classroom outcomes with real-world needs.

### Senior AI Engineer, hypREspace
Jul 2025 - Oct 2025 · Remote
Delivered AI consulting and platform capabilities, emphasizing AWS-based deployments and applied generative AI solutions.

### Senior Software Engineer, 7Span
2024 - 2025 · India - Remote
Spearheading GenAI and ML applications in Python, building scalable pipelines, and integrating AI solutions into enterprise systems.

### Data Science Engineer & Manager, fxis.ai
2022 - 2024 · India - Remote
Led ML engineering teams through delivery, focusing on data quality, model optimization, and stakeholder alignment.

### Software Engineer, Accenture
2019 - 2022 · India
Drove feature development, impact analysis, and cross-team delivery in Agile environments.

## Services

### AI-Assisted Custom Software Development

URL: https://aiwithvd.com/services/ai-assisted-custom-software-development

Web, API and data products built with AI-assisted engineering and real code review.

Deliverables:
- Written specification and architecture before code
- Typed API and data model, containerised and deployable on your cloud
- Automated tests and CI on every commit
- Handover documentation and a recorded walkthrough

## What I build

Products that need a real backend, a clean API and a frontend that does not fall over: dashboards, internal tools, SaaS features, document pipelines and the services that sit behind AI features. I write the specification first, so we agree on scope before code exists.

## How I work

Every project runs on one branch strategy, one CI pipeline and one definition of done. AI tools help me draft code and tests quickly; a type checker, linting and my own review decide what stays. You see progress on a preview deployment, not in a status report.

## When to call me

You have a product idea with a clear owner and no engineering team yet, or you have a team that needs a senior engineer to design and ship a well-bounded piece of the system.

---

### Agentic Workflow Development

URL: https://aiwithvd.com/services/agentic-workflow-development

LangGraph agents with tools, memory and evaluation gates that hold up in production.

Deliverables:
- Agent architecture with explicit state, tools and failure paths
- Tool layer over your existing APIs, often as an MCP server
- Evaluation set and harness so behaviour changes are measured
- Tracing and observability from the first deployment

## What I build

Agents that do real work: answer analytics questions over your data, generate personalised content on a schedule, run research with citations, or orchestrate multi-step tasks across your systems. Each one has explicit state, typed tools and a trace for every run.

## How I work

I start with the failure cases, not the happy path. We write down what the agent must never do, build an evaluation set from real queries, and only then wire tools. Prompt content is served dynamically so each request carries only the instructions it needs.

## When to call me

Your prototype agent works in a demo and fails in production, or you have a workflow with clear inputs and outputs that a person currently does by hand.

---

### LLM Fine-tuning, Inference and Deployment

URL: https://aiwithvd.com/services/llm-fine-tuning-inference-deployment

SFT and LoRA fine-tuning, evaluation harnesses, and serving that fits the budget.

Deliverables:
- Gold evaluation dataset and scoring harness
- Curated training set mined from production traces
- Fine-tuned model (SFT, LoRA or QLoRA) with a before-and-after report
- Serving setup sized to your traffic and cost target

## What I build

Smaller, cheaper models that behave like your best prompt. I mine successful trajectories from production logs, curate them, fine-tune with SFT and LoRA, and prove the result on a held-out evaluation set before anything is deployed.

## How I work

Evaluation first. Nothing changes until there is a number to compare against. Then I iterate on data quality rather than hyperparameters, because that is where the gains are. Serving is chosen last, once we know how small the model can be.

## When to call me

Your LLM bill scales faster than your usage, your agent needs reliable tool calls, or you need a model that runs inside your own infrastructure.

---

### MLOps

URL: https://aiwithvd.com/services/mlops

Pipelines, monitoring, versioning and cloud deployment for models in the wild.

Deliverables:
- Reproducible training and data pipelines
- Model and dataset versioning with a registry
- Deployment on AWS or your cloud, containerised, with rollback
- Monitoring for drift, latency, cost and quality

## What I build

The operational layer that keeps models useful after launch: pipelines that rebuild datasets and models on demand, registries that record what is running where, deployments with rollback, and monitoring that catches drift before users do.

## How I work

I treat a model like any other release artefact. It has a version, a test suite (the evaluation set), a deployment pipeline and a dashboard. Cost is a first-class metric, tracked per request from day one.

## When to call me

You have a model in production that nobody can confidently redeploy, or you are about to launch one and want it to survive its first month.

---

### RAG Applications

URL: https://aiwithvd.com/services/rag-applications

Retrieval systems with hybrid search, reranking and traceable answers.

Deliverables:
- Ingestion pipeline with chunking, metadata and change detection
- Hybrid retrieval (keyword plus vector) with reranking
- Answer generation with citations and a verification gate
- Evaluation set for retrieval quality and answer faithfulness

## What I build

Question-answering and research systems over your documents, tickets, contracts or regulations. Retrieval is hybrid and reranked, answers carry citations, and a verification layer checks that each claim is supported before the user sees it.

## How I work

I measure retrieval before generation. A small gold set of questions with known source passages tells us whether chunking and search are working. Only then do I tune prompts. Every answer keeps a trace of what was retrieved and why.

## When to call me

Your users ask questions that your search box cannot answer, or a first RAG prototype is confidently wrong too often to trust.

---

### Document Intelligence

URL: https://aiwithvd.com/services/document-intelligence

OCR plus LLM extraction pipelines that turn documents into structured data.

Deliverables:
- Extraction API with async processing and status tracking
- OCR and layout handling for scans, PDFs and images
- Typed output schemas per document type, with templates for new ones
- Local or cloud LLM backend, switchable by configuration

## What I build

Pipelines that take messy documents in and give typed JSON out: invoices, contracts, ESG reports, forms. OCR handles the scans, an LLM fills a strict schema, and an async worker layer keeps the API responsive while the slow work happens.

## How I work

Schemas first. We define the fields, their types and their validation rules, then measure extraction accuracy on a labelled sample before any tuning. New document types become a template, not a project.

## When to call me

A team is retyping documents into a system by hand, or an existing OCR setup gives you text but not data.

---

### Voice AI

URL: https://aiwithvd.com/services/voice-ai

Real-time voice assistants, self-hosted when privacy demands it.

Deliverables:
- Real-time voice pipeline (VAD, speech to text, LLM, text to speech)
- WebRTC media layer and a web or mobile client
- Provider-swappable configuration, local or cloud per stage
- Latency budget and resilience wrappers for each stage

## What I build

Voice assistants you can actually talk to: low latency, interruptible, and deployable where your data has to stay. The pipeline is four swappable stages behind a WebRTC media layer, with a web client included.

## How I work

Latency is the product. I set a budget per stage, measure it continuously, and choose models to fit. Privacy requirements decide which stages stay local; the configuration system makes that a setting, not a rebuild.

## When to call me

You want a voice interface for a product or an internal tool, and cloud voice APIs are either too slow, too costly or not allowed.

---

### AI Strategy and Consulting

URL: https://aiwithvd.com/services/ai-strategy-consulting

Roadmaps, feasibility checks and architecture reviews before you commit budget.

Deliverables:
- Written assessment of the problem, data and constraints
- Architecture proposal with build-versus-buy options and costs
- Evaluation plan that defines what "working" means
- Phased roadmap sized for your team

## What I do

Help founders and product leaders decide what to build with AI, what to buy, and what to skip. The output is a document: the problem stated precisely, the data you have and lack, an architecture with costs, and a roadmap in phases that each deliver something usable.

## How I work

I ask for real examples of the task, not a slide about it. I look at the data before I propose a model. I write down how we will know it works before anyone writes code. Recommendations come with the trade-offs I rejected and why.

## When to call me

Before a budget is committed, when an AI initiative has stalled, or when you need an independent review of a proposal from a vendor or an internal team.

## Case Studies

### hypREspace: Cutting Token Costs 90% While Raising Tool-Calling Accuracy to 95%

URL: https://aiwithvd.com/case-studies/hyprespace-agent-cost-optimization
Client: hypREspace
Industry: Commercial real estate, transaction management SaaS
Outcomes:
- 90%: Token cost reduction, measured per query
- 75.8% to 95%: Tool-calling accuracy on the gold eval set
- ~10K: Verified golden trajectories used for SFT + LoRA
- Amazon Nova: Fine-tuned model replacing a large prompt-steered one
Stack: LangGraph, LangChain, FastMCP, FastAPI, Amazon Nova, Unsloth (SFT + LoRA), AWS Bedrock

### At a Glance

| Field | Detail |
|---|---|
| Client | hypREspace |
| Sector | Corporate real estate transaction management |
| Engagement | Reduce LLM spend and raise tool-calling reliability on a live agent |
| Platform | LangGraph agent, custom MCP layer, gold-standard evaluation dataset, Amazon Nova fine-tuning (SFT + LoRA via Unsloth) |
| Focus | Paying only for the prompt each query actually needs |
| Outcome | 90% token cost reduction and 95% tool-calling accuracy, measured, not estimated |

### The Product

hypREspace is an AI-driven transaction management platform for corporate real estate, trusted by firms including Cushman & Wakefield, which selected it as their transaction management technology provider in 2025. Its agentic AI features run in production, on real user traffic, every day.

Production is where agent economics get honest. The agent worked, but every query carried a large static prompt: full workflow instructions, tool guidance, and formatting rules, whether the query needed them or not. Token costs scaled with traffic, and tool-calling mistakes, the wrong tool, wrong arguments, or the wrong order, meant wasted retries on top. The mandate was direct: make the agent cheaper and more accurate at the same time, and prove both with numbers.

The engagement centered on four objectives:

- Cut token spend by making prompt content dynamic instead of static.
- Raise tool-calling accuracy: right tool, right arguments, right order.
- Build an evaluation harness so both improvements are measured, not assumed.
- Fine-tune a model on the agent's own successful behavior to lock the gains in.

### The Challenge

The core tension of this project was capability versus cost. Everything you put in a prompt makes the agent theoretically more capable and definitely more expensive, and past a point, bigger prompts make tool calling worse, not better.

1. **The static prompt tax.** Every query paid for the full instruction set, even when it used a fraction of it. A simple lookup carried the same prompt weight as a complex multi-step workflow. At production volume, that overhead compounds into real money.

2. **Accuracy you cannot see is accuracy you cannot fix.** Tool-calling failures were anecdotal: a wrong argument here, a skipped step there. Without a ground-truth benchmark there was no way to know the real accuracy, which meant no way to know if any change helped.

3. **Prompt bloat degrades tool selection.** Long prompts bury the instructions that matter for the current query. The agent had to find the relevant workflow guidance inside everything else, and sometimes it did not.

4. **Improvements had to hold at lower cost.** The end goal was not just a tuned prompt on an expensive model. It was locking reliable behavior into the system so quality and cost improve together, including through fine-tuning.

### My Approach

I ran this as an evaluation-first engagement. Nothing shipped without a number attached.

**A gold evaluation dataset before any changes.** I built a gold-standard eval set of real query patterns, each annotated with the correct tool calls: which tool, with which arguments, in which order. Accuracy became a score, not an impression. Token usage was instrumented the same way, with a token counter measuring every query before and after each change.

**Dynamic prompt injection through the custom MCP layer.** Instead of one giant static prompt, I restructured instructions into portions served by the MCP layer. The agent calls a tool to fetch exactly the workflow or prompt section the current query needs, when it needs it. A simple query pulls a small prompt. A complex workflow pulls its full instructions. The prompt became pay-per-use.

**Prompt tuning against the eval set.** With the harness in place, prompt variants were iterated and scored against the gold dataset. Changes that raised tool-calling accuracy stayed, changes that did not were discarded, and every decision had a before-and-after number.

**Fine-tuning to lock it in.** The final layer was training the behavior into the model itself. I curated roughly 10,000 verified golden trajectories from production logs, filtered for correct tool calls and completed workflows, and fine-tuned Amazon Nova with SFT and LoRA using Unsloth. Verification ran on two tracks: programmatic verifiers checking tool names and arguments, and LLM-as-judge evaluation on a held-out set. A model trained on the agent's exact tool-call format emits valid calls consistently, without carrying an 80,000-token instruction manual into every query.

### Engineering Highlights

- **Gold eval dataset as the source of truth.** Tool-calling accuracy scored on right tool, right arguments, and right call order against annotated ground truth.
- **Token accounting on every query.** Before-and-after token counts and costs measured with a token counter, so the savings figure is arithmetic, not a guess.
- **MCP-served prompt portions.** Workflow instructions decomposed into fetchable sections; the agent retrieves only what the query requires.
- **Prompt tuner loop.** Systematic prompt iteration scored against the eval set instead of tweaked by feel.
- **~10K golden trajectories.** The SFT dataset came from the agent's own verified production successes, filtered for correct tool calls and completed workflows, so the model learned the system's real behavior, not synthetic approximations.
- **Amazon Nova fine-tuned with Unsloth.** SFT with LoRA adapters, keeping training fast and affordable while staying inside the AWS ecosystem the client already runs on.
- **Two-track verification.** Programmatic verifiers checked tool names and arguments; LLM-as-judge evaluation on a held-out set caught the failures rule-based checks miss.
- **Compounding gains.** Injection cut the cost per query, tuning raised accuracy per call, and fine-tuning locked both into the weights.

### Results

| Metric | Result |
|---|---|
| Token cost and prompt volume | 90% reduction, measured by per-query token counting before and after (prompts that ran up to 80K tokens per question cut to a fraction) |
| Tool-calling accuracy | Lifted from 75.8% to 95%, scored against the gold eval dataset on tool choice, arguments, and call order |
| Failure diagnosis | Eval harness pinpoints which tool call diverges from ground truth |
| Serving economics | Fine-tuned Amazon Nova replaces a large prompt-steered model for tool decisions |


### Tech Stack and Deliverables

- **Platform:** LangGraph, LangChain, custom MCP layer, FastAPI
- **Fine-tuning:** Amazon Nova, SFT with LoRA via Unsloth, ~10K curated golden trajectories from production logs
- **Evaluation:** Gold-standard tool-calling eval dataset, token counting instrumentation
- **Deliverables:** Evaluation harness and gold dataset, dynamic prompt injection architecture, tuned production prompts, trajectory-mined SFT dataset, fine-tuned model and serving setup, before-and-after cost and accuracy report

---

### hypREspace: Rearchitecting NL-to-SQL into an Agentic Analytics Engine

URL: https://aiwithvd.com/case-studies/hyprespace-nl-to-sql-agent
Client: hypREspace
Industry: Commercial real estate, transaction management SaaS
Outcomes:
- One agent: Replaced a multi-hop prompt chain end to end
- Every step traced: Failures diagnosed from logs in minutes, not days
- Config-driven: New capabilities added by editing an MCP whitelist
Stack: LangGraph, LangChain, FastMCP, FastAPI, Claude on AWS Bedrock

### At a Glance

| Field | Detail |
|---|---|
| Client | hypREspace |
| Sector | Corporate real estate transaction management |
| Engagement | Rearchitect the AI chat and analytics feature end to end |
| Platform | LangGraph ReAct agent, Claude on AWS Bedrock, custom MCP layer (FastMCP) on the existing FastAPI backend |
| Focus | Replacing a brittle multi-hop prompt chain with one observable agent |
| Outcome | Complex queries answered reliably, with every step of the reasoning visible and debuggable |

### The Product

hypREspace is an AI-driven transaction management platform for corporate real estate, built by real estate transaction professionals. In April 2025, Cushman & Wakefield, one of the largest commercial real estate services firms in the world, selected hypREspace as its transaction management technology provider. Part of what makes the platform stand out is its AI chat interface: transaction professionals ask questions about their portfolio in plain English and get answers and charts back, drawn from four core tables covering Projects, Properties, Leases, and Lease Options.

That chat feature was the problem. Under the hood it ran on a chain of hard-coded prompts, and as usage grew, the chain kept breaking in ways nobody could see into. I was brought in to rearchitect it.

The engagement centered on four objectives:

- Replace the multi-hop prompt chain with a single agent that reasons through queries end to end.
- Make every step of query handling observable, so failures are diagnosable instead of silent.
- Reuse the existing FastAPI backend rather than rebuilding data access from scratch.
- Handle the hard cases the old system dropped: complex multi-table questions, currency and unit conversion, and chart generation.

### The Challenge

The core tension of this project was flexibility versus control. A pipeline of fixed prompts is predictable until the question does not fit the pipeline, and then it fails without explanation.

1. **A five-hop chain with no visibility.** The old system ran a fixed sequence: analyze the user query, map it to the right table, select the appropriate fields, generate the SQL, then post-process results for the user's preferred currency and measurement units. Chart questions added a separate Python code generator on top. When a complex query failed, and they often did, sometimes with no response at all, there was no way to know which hop broke.

2. **Hard-coded knowledge of four tables.** Table mapping and field selection were baked into prompts. Every schema change meant prompt surgery, and any question spanning Projects, Properties, Leases, and Lease Options at once pushed the chain past what its fixed steps could express.

3. **Enterprise data cannot be handed to an LLM raw.** hypREspace serves enterprise clients. The agent could not be given open database access. It needed a controlled, auditable surface where the backend team decides exactly what the agent can touch.

4. **Post-processing complexity.** Multi-currency conversion to each user's preferred currency and multiple measurement units were bolted on after SQL generation, which meant the reasoning step and the formatting step could silently disagree.

### My Approach

I treated this as an agent architecture problem, not a prompt improvement problem. The fixed chain was replaced by a LangGraph ReAct agent that decides its own path through each query.

**A custom MCP layer over the existing backend.** Instead of giving the agent SQL access, I built a custom MCP server with FastMCP directly on top of hypREspace's existing FastAPI backend. Because the backend is FastAPI, every endpoint already carries typed input and output schemas and descriptions, and the MCP layer passes those to the agent as tool definitions automatically. Exposure is config-driven: only endpoints explicitly whitelisted in configuration become tools. The backend team controls the agent's entire surface area from a config file.

**A ReAct agent instead of a fixed chain.** The LangGraph ReAct agent, running Claude on AWS Bedrock, reasons about each question and picks the tools it needs, in the order it needs them. Table mapping, field selection, query construction, currency conversion, and chart generation stopped being separate hard-coded stages and became decisions the agent makes per query. A question the old chain could not route now simply takes a different path through the same tools.

**Observability as a first-class outcome.** Every reasoning step and every tool call is a discrete, logged event. When something goes wrong, the trace shows exactly which step failed and why. The days of "the chat did not respond and we do not know why" ended with the chain.

**The agent as its own service.** The agent runs as a separate FastAPI service alongside the main backend, so it scales and deploys independently and the existing product surface stays untouched.

### Engineering Highlights

- **Config-driven tool exposure.** A custom FastMCP layer where whitelisted FastAPI endpoints become agent tools, schemas and descriptions included, with zero manual tool definitions to maintain.
- **LangGraph ReAct orchestration.** One agent replaces five hard-coded stages, choosing its own tool sequence per query.
- **Claude on AWS Bedrock.** Enterprise-grade inference inside the client's AWS environment, keeping data within their cloud boundary.
- **Schema changes without prompt surgery.** When the backend adds or changes an endpoint, the tool definition updates with it. No prompts to rewrite.
- **Currency and units inside the loop.** Conversion to the user's preferred currency and measurement units happens through tools the agent calls as part of its reasoning, not as a bolt-on afterthought.
- **Chart generation unified.** Visualization requests flow through the same agent instead of a separate code-generation pipeline.
- **Full trace logging.** Every query leaves a step-by-step record of reasoning and tool calls for debugging and audit.
- **Independent agent service.** A dedicated FastAPI service for the agent, deployable and scalable apart from the core backend.

### The Impact

- **Complex queries stopped failing silently.** Questions spanning multiple tables, currencies, and units, the exact cases that broke the old chain, are now handled by the agent's own reasoning.
- **Failures became diagnosable.** When something does go wrong, the trace points to the exact step, turning multi-day debugging mysteries into minutes of log reading.
- **The backend team kept control.** The config-driven MCP layer means the agent's capabilities grow by editing a whitelist, not by writing integration code.
- **A foundation, not a patch.** New tables, new endpoints, and new capabilities plug into the same agent, so the analytics feature scales with the product instead of against it.


### Tech Stack and Deliverables

- **Platform:** LangGraph, LangChain, FastAPI, AWS Bedrock (Claude)
- **Agent layer:** Custom MCP server built with FastMCP over the existing FastAPI backend, config-driven endpoint exposure
- **Deliverables:** Agent architecture design, custom MCP layer, LangGraph ReAct agent service, currency and unit handling inside the agent loop, unified chart generation, trace logging and observability, deployment as an independent FastAPI service

---

### AI Research Platform for a Regulated Accounting Profession

URL: https://aiwithvd.com/case-studies/regulated-accounting-research-platform
Client: Compliance-focused accounting technology startup (under NDA)
Industry: Accounting technology, tax and compliance SaaS
Outcomes:
- 95%: Accuracy target the architecture is built to
- Zero: Tolerance for fabricated citations, enforced by a hard-fail gate
- 8 skills: Agent Skills framework with progressive loading
- In-region: Tenant-isolated processing for regulated firms
Stack: LangGraph, FastAPI, Weaviate, PostgreSQL (RLS), Mem0 + Neo4j, Celery, Dagster, Claude, AWS ECS

### At a Glance

| Field | Detail |
|---|---|
| Client | Accounting technology startup (under NDA) |
| Sector | Tax research and compliance software for accounting professionals |
| Engagement | End-to-end AI and backend architecture for the MVP |
| Platform | LangGraph, FastAPI, Weaviate, PostgreSQL, Celery, Dagster, Claude, AWS (region-locked) |
| Focus | Cited, verified answers in a domain where a wrong citation is a liability event |
| Outcome | An architecture built to a 95% accuracy target with zero tolerance for fabricated citations |

### The Product

The client is building a research and workflow platform for accounting and tax practitioners in a heavily regulated market. Practitioners ask complex tax questions and get cited answers drawn from authoritative government sources, upload spreadsheets for AI-driven analysis, and generate client advice letters with embedded citations, exportable to PDF and Word.

The defining constraint is the profession itself. For an accountant, a fabricated ruling number is not a UX bug, it is a professional indemnity issue. The discovery process set non-negotiables: 95%+ accuracy on gold-standard queries, zero fabricated citations, all data and LLM inference inside the country's cloud region, a knowledge base never more than seven days behind the regulator, and full audit reproducibility for every answer.

The engagement centered on four objectives:

- Design an agent architecture where unverified output structurally cannot reach the user.
- Build a RAG layer over regulatory sources with citation-grade metadata and freshness guarantees.
- Deliver multi-tenant isolation and in-region deployment fit for regulated firms from day one.
- Ship eight domain skills as a framework, so new skills are content additions, not code changes.

### The Challenge

The core tension of this project was capability versus defensibility. Generative AI is fluent by default and accurate by engineering, and in this domain, fluency without verification is a lawsuit.

1. **Hallucination as a hard failure, not a quality issue.** Most RAG systems tolerate occasional wrong answers. Here, a single invented section reference could expose a firm professionally. Verification could not be a scoring layer; it had to be a gate that fails closed.

2. **A knowledge base that ages by the week.** Tax rulings get superseded. An answer citing last year's ruling as current is wrong even if retrieval worked perfectly. The pipeline needed change detection, version supersession, and a freshness ceiling measured in days.

3. **Multi-tenancy in a trust-sensitive market.** Every firm brings its own policies and precedents. Firm A's documents must be structurally invisible to Firm B, while both share the same regulatory knowledge, in the same infrastructure.

4. **Regulated deployment boundaries.** All processing had to stay in a single national cloud region, with LLM inference through in-region endpoints, immutable audit trails meeting statutory retention, and a documented path from managed services to self-hosted infrastructure for the most conservative clients.

### My Approach

I designed the system as a layered architecture where each layer has one responsibility and compliance is enforced at the platform level, not sprinkled through business logic.

**A state machine, not a free-form loop.** The agent is an explicit LangGraph state machine: parse, route, execute, verify, compose. A regulated profession demands deterministic, inspectable control flow, so every step is replayable and individually testable, and no path through the graph can bypass verification.

**RAG built for citations, not just retrieval.** The knowledge layer is a single Weaviate instance with namespace isolation: a shared global namespace holding rulings, legislation, and guidance, plus one isolated namespace per firm. Retrieval is hybrid, dense semantic search plus BM25, with cross-encoder reranking from top-50 to top-10. Every chunk carries citation-grade metadata: ruling ID, section path, effective date, supersession links, and status, and only current-status chunks can back a primary citation.

**A four-tier verification layer.** Generated claims are extracted atomically, checked against retrieved sources by a local NLI cross-encoder, escalated to an LLM judge for borderline cases, and hard-failed if anything is classified as fabricated. The user sees a verification error, never a hallucination.

**Skills as data, not code branches.** Eight domain skills, covering areas like shareholder loan analysis, trust distributions, fringe benefits, and capital gains events, ship as filesystem-based Agent Skill packages following the Anthropic specification. Each package bundles its metadata, retrieval strategy, output schema, and scripts, with progressive loading so metadata costs about 100 tokens per skill until a skill is actually invoked. Adding a ninth skill requires no agent changes.

**Async by default, scaled with Celery and Dagster.** Advice letter generation runs as Celery jobs, rendering firm-customized letters with footnoted citations to PDF, Word, and Excel without blocking the API. Dagster orchestrates the daily ingestion pipeline: browser-automated scraping of the regulator's site, change detection, chunking, embedding, and supersession management, keeping the knowledge base within its seven-day freshness commitment.

**Compliance as infrastructure.** FastAPI services on AWS ECS in the required region, PostgreSQL with row-level security for operational state, complete persistence of every query's retrieved chunks, prompt, model version, and verification result for audit reproducibility, and a self-hosted memory layer (Mem0 with Weaviate and Neo4j) so institutional research memory stays inside the client's boundary.

### Engineering Highlights

- **Verification-gated LangGraph state machine.** Structurally impossible to ship unverified output; hard failure is a first-class graph node with incident logging.
- **Hybrid RAG with citation metadata.** Dense plus BM25 retrieval, cross-encoder reranking, and chunk-level ruling IDs, effective dates, and supersession tracking.
- **Namespace-per-tenant isolation.** One shared regulatory namespace, one isolated namespace per firm, merged at query time with sources distinguished in citations.
- **Agent Skills framework.** Eight skills as discoverable filesystem packages with progressive token loading, extendable without touching the agent.
- **FastAPI plus Celery at scale.** Synchronous answers within tight latency targets (P50 under 8 seconds, P95 under 20), with slow work, letter generation and bulk imports, pushed to async workers.
- **Dagster ingestion pipeline.** Daily automated scraping with change detection and version supersession, holding the seven-day freshness guarantee.
- **Region-locked AWS deployment.** All compute and inference pinned in-region, row-level security, immutable audit trail meeting statutory retention.
- **Self-hosted memory layer.** Mem0 with Weaviate and Neo4j giving firms a searchable, relationship-aware record of past research without data leaving their environment.

### The Impact

- **Trust became an architecture, not a promise.** The verification gate and audit trail give the client something rare in AI products: an answer they can defend to a professional standards body.
- **Regulated clients are servable from day one.** In-region processing, tenant isolation, and the self-hosted path mean the most conservative firms are in scope, not a future phase.
- **The product scales by adding content.** New skills, new firms, and new rulings all slot into existing structures without engineering work.
- **Costs stay controlled.** Progressive skill loading, prompt caching, and a small local NLI model keep per-query economics viable at the platform's price points.


### Tech Stack and Deliverables

- **Platform:** LangGraph, LangChain, FastAPI, Claude (in-region endpoints), AWS ECS
- **Knowledge and data:** Weaviate (hybrid search, multi-namespace), PostgreSQL with row-level security, Mem0 with Neo4j for graph memory
- **Pipelines and workers:** Celery for advice letter generation and bulk imports, Dagster for daily regulatory ingestion with change detection
- **Verification:** Claim extraction, local NLI cross-encoder, LLM judge, hard-fail citation gate
- **Deliverables:** Full MVP architecture, LangGraph agent, RAG and verification layers, eight-skill Agent Skills framework, multi-tenant and compliance infrastructure design, ingestion pipeline, cost and scaling model

---

### Agentic Question Generation for a Family Connection App

URL: https://aiwithvd.com/case-studies/family-app-question-generator
Client: Premium family connection platform (under NDA)
Industry: Consumer mobile, family and social connection
Outcomes:
- Daily: Personalised prompts per member, fully automated
- Stable JSON: Typed contract the mobile team integrated without friction
- Traced: LangSmith observability on every generation
Stack: LangGraph, Agentic RAG, OpenAI GPT, LangSmith, FastAPI, Docker, AWS EC2

### At a Glance

| Field | Detail |
|---|---|
| Client | Family connection mobile platform (under NDA) |
| Sector | Consumer social, private family networks |
| Engagement | Design, build, and integrate a personalized daily question engine |
| Platform | LangGraph, Agentic RAG, OpenAI GPT, FastAPI, AWS EC2, Docker |
| Focus | Structured JSON personalization served to a mobile app in real time |
| Outcome | Increased daily user engagement through personalized storytelling prompts |

### The Product

The client runs a premium, invite-only mobile platform that helps families stay genuinely connected: no ads, no algorithms, no strangers. Its signature feature is a daily group prompt. Every day, family members receive a question designed to spark a story, a memory, or a conversation, and their answers, in text, voice, or video, build a shared family archive over time.

The daily question is the heartbeat of the product. If it lands, families talk. If it feels generic, the streak dies and so does the habit. The client wanted every member to receive questions that felt written for them personally, and they needed it delivered as a clean API their mobile app could consume without friction.

The engagement centered on four objectives:

- Generate daily questions personalized to each user's profile, preferences, and moment in time.
- Match the emotional tone of prompts to the person and the occasion, not just the topic.
- Deliver everything as structured JSON through fast, reliable REST endpoints the mobile app consumes directly.
- Deploy a production service the client's team could operate, monitor, and evolve.

### The Challenge

The core tension of this project was personal versus repeatable. A human writer can craft a touching question for one family; the system had to do it for every user, every day, automatically, and hand it to a mobile app as clean data.

1. **Generic prompts kill engagement.** A question bank on rotation feels like a question bank on rotation. Personalization had to run deep: a user who chose humor packs should get playful prompts, one focused on spirituality or self-reflection should get depth, and a dad should get questions that fit that role.

2. **Timing is part of the message.** The right question on a random Tuesday is different from the right question before a birthday or heading into a weekend. Temporal context, birthdates, day of the week, upcoming weekends, had to shape generation, not just profile data.

3. **An LLM behind a mobile app cannot improvise formats.** The app renders prompts natively, which means the API must return strict, structured JSON every single time. One malformed response is a broken screen in a paying member's hands.

4. **Production from day one.** This was not a demo. The service had to run reliably on the client's AWS infrastructure, be observable when something misbehaved, and integrate into an existing mobile release cycle.

### My Approach

I built the feature as an agentic service with a hard contract at its edge: creative generation inside, strict JSON outside.

**An agentic generator, not a prompt template.** The question generator is a LangGraph agent using Agentic RAG over user context. For each user it reasons over profile attributes, selected preference packs such as dad-pack, humor, spirituality, and self-reflection, and temporal signals like birthdate, day of the week, and upcoming weekends, then composes a question aligned with the user's emotional tone and moment. The same machinery produces a playful weekend prompt for one member and a reflective one for another, from one codebase.

**Structured output as a contract.** Generation is constrained to a defined JSON schema that the mobile app renders natively. The app team integrates against a stable shape, and the AI's creativity stays inside the fields.

**FastAPI as the integration surface.** The agent is served through FastAPI REST endpoints purpose-built for mobile consumption: fast responses, clean status codes, and predictable payloads. FastAPI's typed request and response models double as living documentation for the mobile team, so integration questions were answered by the schema instead of by meetings.

**Deployed and observable on AWS.** The service runs Dockerized on AWS EC2. LangSmith tracing covers every generation end to end, and LangGraph Studio was used throughout development to inspect and debug the agent's reasoning paths before they reached users.

### Engineering Highlights

- **LangGraph agent with Agentic RAG.** Context-aware generation reasoning over profile, preference packs, and temporal signals per user, per day.
- **Preference-pack personalization.** Dad-pack, humor, spirituality, and self-reflection styles shape tone and subject, so the same feature feels different to every member.
- **Temporal awareness.** Birthdays, weekdays, and upcoming weekends steer question selection, making prompts feel timely instead of scheduled.
- **Strict structured JSON.** Schema-constrained output the mobile app renders directly, with no parsing gymnastics on the client side.
- **Mobile-first FastAPI design.** Typed Pydantic models, predictable payloads, and response times fit for an in-app daily experience.
- **Dockerized EC2 deployment.** A containerized service on the client's own AWS account, fitting their existing operations.
- **Full generation tracing.** LangSmith instrumentation on every request, so any odd question can be traced back through the agent's reasoning.
- **Studio-driven development.** LangGraph Studio used to visualize and debug agent paths during development, shortening the iteration loop.

### The Impact

- **Daily engagement went up.** Personalized storytelling prompts gave members a reason to open the app every day, and the client credits the feature with increasing daily user engagement.
- **The mobile team shipped without friction.** A stable JSON contract and typed FastAPI endpoints meant the app integration was a consumer of clean data, not a negotiation with an AI system.
- **Personal at scale.** Every member receives questions shaped by who they are and what day it is, from a single automated service.
- **A pattern to build on.** The agent, schema, and API structure generalize to other personalized content surfaces the platform adds next.


### Tech Stack and Deliverables

- **Platform:** LangGraph, Agentic RAG, OpenAI GPT, LangSmith, LangGraph Studio
- **Backend:** FastAPI, Docker, AWS EC2
- **Deliverables:** Agentic question generation service, personalization logic over preference packs and temporal context, structured JSON output schema, mobile-facing REST API, Dockerized AWS deployment, tracing and observability setup

---

### Document Extraction Pipeline: Scaling AI Workloads with FastAPI and Celery

URL: https://aiwithvd.com/case-studies/document-extraction-pipeline
Client: Product build
Industry: Document processing, back-office automation
Outcomes:
- 10x: Faster than manual data entry
- 95%+: Extraction accuracy on structured documents
- 3 schemas: Invoice, legal and ESG, plus custom templates
- Local or cloud: Ollama or OpenAI with zero code changes
Stack: FastAPI, Celery, Redis, PostgreSQL (JSONB), MinIO, MinerU OCR, OpenAI GPT-4o-mini, Ollama, Docker

### At a Glance

| Field | Detail |
|---|---|
| Client | Product build (portfolio) |
| Sector | Document automation: invoices, legal documents, ESG reports |
| Engagement | Design and build a production-grade extraction service |
| Platform | FastAPI, Celery, Redis, PostgreSQL, MinIO, MinerU OCR, OpenAI / Ollama |
| Focus | Async-first architecture that keeps the API fast while AI does slow work |
| Outcome | 10x faster than manual entry with 95%+ extraction accuracy |

### The Product

Businesses drown in documents that carry structured data trapped in unstructured formats: invoices, contracts, ESG reports. Manual data entry is slow, error-prone, and expensive, and most extraction tools fail the moment a layout changes.

This pipeline turns PDFs and images into clean, structured JSON. Upload a document, get back typed fields matching a schema. OCR handles the reading, an LLM handles the understanding, and the architecture handles the part most AI demos skip: doing this reliably, concurrently, and securely in production.

The engagement centered on four objectives:

- Extract structured data from PDFs and images across multiple document types.
- Keep the API responsive while OCR and LLM inference run for seconds or minutes per document.
- Support both cloud LLMs and fully local inference for cost and privacy flexibility.
- Ship production concerns complete: authentication, rate limiting, storage, and job tracking.

### The Challenge

The core tension of this project was heavy AI inference inside a responsive API. OCR and LLM extraction take seconds to minutes per document, and an API that blocks on them collapses under its second concurrent user.

1. **Slow work behind a fast interface.** A single document can take longer to process than any sane HTTP timeout. The architecture had to accept work instantly and deliver results asynchronously, without losing jobs in between.

2. **Layout chaos.** Invoices, legal documents, and ESG reports share nothing structurally. Tables, multi-column layouts, scanned images, and mixed quality inputs all had to resolve into the same clean schemas.

3. **The cost and privacy fork.** Some workloads justify a cloud LLM; others demand data never leaves the machine. One codebase had to serve both without branching into two products.

4. **Production is a feature.** Authentication, per-user rate limiting, object storage, retry behavior, and job status tracking are where prototypes go to die. They were in scope from the first commit.

### My Approach

I built the system async-first, with a clean separation between the API that accepts work and the workers that do it.

**FastAPI at the front, Celery at the back.** The API validates uploads, persists files to MinIO object storage, enqueues a Celery job through Redis, and returns a job ID immediately. Clients poll for results while workers process in the background. Throughput scales by adding workers, not by making users wait.

**A two-stage extraction pipeline.** MinerU, running its vlm-auto-engine, converts documents into structured Markdown that preserves tables and layout, the context an LLM needs to extract fields correctly. The LLM stage then maps that Markdown into typed schemas, with pre-built templates for Invoice, Legal, and ESG document types and support for custom extraction templates.

**Provider-agnostic LLM layer.** The extraction stage runs against OpenAI GPT-4o-mini for managed inference or a local Ollama model for fully private, zero-API-cost processing. Switching is configuration, not code.

**Production hardening throughout.** JWT authentication with per-user rate limiting, PostgreSQL with JSONB for flexible result storage, Redis doubling as broker and cache, MinIO for S3-compatible storage, and the whole stack containerized with Docker for reproducible deployment.

### Engineering Highlights

- **Async job architecture.** Upload returns a job ID in milliseconds; Celery workers absorb the heavy lifting, and the API never blocks on inference.
- **MinerU OCR front end.** Layout-aware conversion to structured Markdown, preserving tables that naive OCR flattens into noise.
- **Schema-driven LLM extraction.** Pre-built Invoice, Legal, and ESG schemas with customizable templates, returning typed JSON instead of free text.
- **Cloud or local inference.** OpenAI and Ollama behind one interface, letting cost and privacy requirements pick the backend per deployment.
- **JSONB result storage.** PostgreSQL stores heterogeneous extraction results queryably without schema migrations per document type.
- **JWT auth and rate limiting.** Per-user access control and request budgets built in, not bolted on.
- **S3-compatible object storage.** MinIO handles originals and artifacts, keeping the database lean and the files durable.
- **Fully containerized.** Docker Compose brings up the entire stack, API, workers, Redis, Postgres, and MinIO, in one command.

### Results

| Metric | Result |
|---|---|
| Processing speed | 10x faster than manual data entry |
| Extraction accuracy | 95%+ on structured extractions |
| Document types | 3 pre-built schemas (Invoice, Legal, ESG) plus custom templates |
| Deployment | Cloud LLM or fully local via Ollama, zero code changes |

### Tech Stack and Deliverables

- **Platform:** FastAPI, Celery, Redis, PostgreSQL (JSONB), MinIO, Docker
- **AI layer:** MinerU (vlm-auto-engine) OCR, OpenAI GPT-4o-mini, Ollama for local inference
- **Deliverables:** Async extraction API, worker pipeline, three document schemas with template system, JWT auth and rate limiting, containerized deployment

---

### Self-Hosted Voice AI: Real-Time Conversation with Zero Cloud Dependency

URL: https://aiwithvd.com/case-studies/self-hosted-voice-ai
Client: Product build
Industry: Conversational AI, privacy-sensitive deployments
Outcomes:
- < 2s: Voice-to-voice latency on Apple Silicon
- Zero: Cloud dependency, all four stages local
- $0: API cost in fully local mode
- 4 stages: VAD, STT, LLM, TTS, each independently swappable
Stack: LiveKit Agents, FastAPI, Silero VAD, faster-whisper, Ollama, Edge-TTS, Next.js, Docker, WebRTC

### At a Glance

| Field | Detail |
|---|---|
| Client | Product build (portfolio) |
| Sector | Voice AI for privacy-sensitive and on-premise use cases |
| Engagement | Design and build a complete real-time voice assistant stack |
| Platform | LiveKit Agents, FastAPI, Next.js 15, Whisper, Ollama, Edge-TTS, Docker |
| Focus | Sub-2s conversational latency with every model running locally |
| Outcome | Real-time voice conversation, zero API costs, no data leaving the machine |

### The Product

Voice assistants usually come with a hidden clause: your audio goes to someone else's cloud. For regulated industries, internal tools, and privacy-conscious deployments, that clause is a dealbreaker, and the alternative has traditionally been no voice interface at all.

This project proves the alternative exists. A complete conversational voice assistant, speech in, intelligent response out, running entirely on local hardware. WebRTC delivers real-time audio, a four-stage AI pipeline handles understanding and response, and not a single byte of conversation leaves the machine. No API keys, no per-minute billing, no data processing agreements.

The engagement centered on four objectives:

- Achieve natural, low-latency voice conversation over real-time WebRTC transport.
- Run the entire AI pipeline, voice detection, transcription, reasoning, and speech synthesis, locally.
- Keep the stack swappable, so any stage can move to a cloud provider when a deployment allows it.
- Make consumer hardware sufficient, with Apple Silicon GPU acceleration doing the heavy lifting.

### The Challenge

The core tension of this project was latency versus locality. Cloud voice APIs are fast because of data-center GPUs; matching conversational latency on local hardware means every stage of the pipeline has to earn its milliseconds.

1. **Four models, one deadline.** A voice turn chains voice activity detection, speech-to-text, LLM inference, and text-to-speech. Their latencies add up, and past roughly two seconds, conversation stops feeling like conversation.

2. **Real-time audio is unforgiving.** HTTP request-response cannot carry natural dialogue. The system needed WebRTC transport, utterance boundary detection, and interruption handling, the plumbing that separates a voice assistant from a voice memo.

3. **Local hardware, real constraints.** Everything had to run on a MacBook, which meant choosing models that fit in memory, exploiting Metal GPU acceleration, and resisting the temptation to solve every problem with a bigger model.

4. **Flexibility without forking.** Some deployments will want Groq or OpenAI for one stage. The pipeline had to swap providers per stage through configuration, keeping one codebase across fully local and hybrid setups.

### My Approach

I built the system on LiveKit Agents, with a clean separation between media transport, the agent worker, and the AI pipeline.

**WebRTC transport via LiveKit.** A Dockerized LiveKit server handles media routing. The browser client, built with Next.js 15 and React 19, requests a JWT from a FastAPI token service and connects over WebRTC. LiveKit dispatches each session to an agent worker that runs the conversation.

**A four-stage local pipeline.** Silero VAD detects speech segments and utterance boundaries. faster-whisper (large-v3-turbo) transcribes locally with Metal acceleration. Ollama running llama3.2:3b generates responses. Edge-TTS synthesizes natural-sounding speech. Each stage streams into the next, keeping end-to-end latency under two seconds on Apple Silicon.

**Resilience wrappers on every stage.** SafeSTT, SafeLLM, and SafeTTS wrappers give each stage graceful error handling, so a single model hiccup degrades the turn instead of killing the session.

**Configuration-driven providers.** Every stage reads its provider from environment configuration. Swap the LLM to Groq, the STT to OpenAI, or the TTS to Cartesia by editing .env; the system auto-detects provider types at startup. Local-first, cloud-optional.

### Engineering Highlights

- **LiveKit Agents orchestration.** SDK 1.5 voice pipeline with proper utterance detection and turn handling, not a walkie-talkie loop.
- **Fully local model stack.** Silero VAD, faster-whisper, Ollama, and Edge-TTS with no cloud dependency and no API spend.
- **Sub-2s conversational latency.** Streaming handoffs between stages, tuned on Apple Silicon with Metal GPU acceleration.
- **Safe wrappers per stage.** Graceful degradation on STT, LLM, and TTS failures keeps sessions alive through transient errors.
- **Provider swapping via .env.** Any stage moves to Groq, OpenAI, Cartesia, or any OpenAI-compatible API without code changes.
- **JWT-secured sessions.** FastAPI service issuing tokens and health checks, keeping room access controlled.
- **One-command media layer.** Dockerized LiveKit server for reproducible WebRTC infrastructure.
- **Modern web client.** Next.js 15, React 19, and Tailwind CSS 4 frontend with live conversation UI.

### Results

| Metric | Result |
|---|---|
| Response latency | Sub-2s voice-to-voice on Apple Silicon |
| Cloud dependency | Zero: all four pipeline stages run locally |
| API costs | Zero in fully local mode |
| Pipeline | 4 stages (VAD, STT, LLM, TTS), each independently swappable |

### Tech Stack and Deliverables

- **Platform:** LiveKit Agents SDK 1.5, FastAPI, Python 3.12, Docker, WebRTC
- **AI pipeline:** Silero VAD, faster-whisper (large-v3-turbo), Ollama (llama3.2:3b), Edge-TTS
- **Frontend:** Next.js 15, React 19, Tailwind CSS 4
- **Deliverables:** Complete voice assistant stack, four-stage local pipeline, provider-swapping configuration system, resilience wrappers, JWT token service, Dockerized media infrastructure, web client

## Projects

### Adaptive RAG: Toggle-Driven Hybrid Retrieval With Skills

URL: https://aiwithvd.com/projects/adaptive-rag
GitHub: https://github.com/aiwithvd/adaptive-rag
Technologies: LangGraph, FastAPI, Milvus, Redis, Ollama, bge-reranker, Langfuse, DeepEval, Python

A self-hosted LangGraph and FastAPI RAG service for a regulated finance and legal practice. Vector search always runs; allowlisted web search is added only when the request toggles it on, never by an LLM router. Four self-reflection gates with hard-capped retry loops decide whether an answer ships. Slash-invocable skills run as isolated sub-agents over the same graph, and opt-in Langfuse tracing plus async DeepEval scoring observe every run without adding latency.

Key features:
- Human-Chosen Route: The vector store is searched on every request. Web search is additive and switched on by a per-request toggle, so the route is a field in the trace rather than a model decision nobody can audit.
- Four Self-Reflection Gates: Per-document relevance grading, a groundedness check, an answer-quality check and a query-rewrite retry. Both retry loops are hard-capped, so the graph always terminates or gives up honestly.
- Skills as Sub-Agents: A SKILL.md and a skill.yaml parametrize four points of the shared graph: system prompt, source filter, web allowlist and web mode. Each run is isolated on a derived thread.
- Trace and Score Without Latency: A per-request Langfuse callback records every node and LLM call. DeepEval GEval scoring starts after the stream closes, on a judge model decoupled from the graph LLM.

Architecture:
- FastAPI endpoint streams thread, status, sources, token and done events over SSE
- LangGraph pipeline with SQLite checkpointing per thread; skill runs on thread_id::skill
- Milvus (HNSW, cosine) with per-skill metadata filters; content-hash refresh on ingest
- Allowlist enforced three times: site: filter, pre-fetch host check, post-redirect host check
- In-process cross-encoder reranker runs on a worker thread off the event loop
- Citations deep-link to the exact passage with URL text fragments
- Opt-in Langfuse v2 profile; fire-and-forget DeepEval scoring that cannot fail the request

Impact: Practitioners get answers grounded in an allowlisted knowledge base, with every route decision, retrieved chunk and quality score recorded on a single trace, and citations that open the source page scrolled to the cited sentence.

---

### Fine-Tuned Tool-Calling Agent: A Small Model Taught to Use 40+ Tools

URL: https://aiwithvd.com/projects/tool-agent-finetune
GitHub: https://github.com/aiwithvd
Technologies: Gemma 3 4B, QLoRA, SFT, PEFT, TRL, LangGraph, MCP, Python, PyTorch

A LangGraph ReAct agent over a custom MCP backend exposing 40+ tools, with the decision model replaced by a fine-tuned Gemma 3 4B. Successful multi-turn tool trajectories from production were filtered and reformatted into an SFT dataset, then used to train Gemma 3 4B with QLoRA — cutting cost and latency while making tool-call formatting far more reliable.

Key features:
- Fine-Tuned Decision Model: Gemma 3 4B trained with SFT + QLoRA on the agent’s own successful tool trajectories, replacing an expensive prompt-steered API model.
- 40+ Tool MCP Backend: A custom MCP server exposes the existing backend as 40+ tools the ReAct agent calls during multi-step tasks.
- Trajectory-Mined Dataset: Production multi-turn runs filtered to verified successes, deduplicated, and formatted with the model’s own chat template and multi-turn response masking.
- Reliable Tool Calling: Training on the exact tool-call format makes the model emit valid name + JSON-args calls consistently, so the harness parses them reliably.

Architecture:
- LangGraph ReAct agent orchestrates the multi-turn tool-calling loop
- Custom MCP server exposes 40+ backend operations as tools
- Gemma 3 4B fine-tuned with QLoRA (4-bit NF4 base, BF16 LoRA adapters)
- LoRA adapters target attention and MLP projections for tool + domain behavior
- Paged optimizer to survive training memory spikes on a single GPU
- Identical chat template at training and serving via apply_chat_template

Impact: Replaced a large prompt-steered model with a small fine-tuned one: lower cost and latency, and far more reliable tool-call formatting across multi-step tasks.

---

### Document Extraction Pipeline: Turn Documents into Structured Data with AI

URL: https://aiwithvd.com/projects/document-extraction
GitHub: https://github.com/aiwithvd/document-extraction-pipeline
Technologies: FastAPI, Celery, PostgreSQL, Redis, MinIO, OpenAI, Ollama, Docker, MinerU

A production-ready FastAPI service that extracts structured data from PDFs and images using OCR and LLM technology. Features async processing, JWT authentication, and support for multiple document types including invoices, legal documents, and ESG reports.

Key features:
- Multi-Format Support: Upload PDF or image files (PNG, JPG, TIFF) up to 10MB. Automatic format detection and preprocessing.
- Async Processing: Non-blocking API calls with Celery workers. Submit documents and poll for results without waiting.
- OCR + LLM Extraction: MinerU (vlm-auto-engine) converts documents into structured Markdown preserving tables and layouts, then OpenAI GPT-4o-mini or local Ollama models extract structured fields.
- Structured Output: Pre-built schemas for Invoice, Legal, and ESG documents with customizable extraction templates.

Architecture:
- FastAPI for high-performance async API endpoints
- Celery workers for background document processing
- PostgreSQL with JSONB for flexible result storage
- Redis as Celery broker and caching layer
- MinIO for S3-compatible object storage
- JWT authentication with per-user rate limiting

Impact: Processes documents 10x faster than manual data entry with 95%+ accuracy on structured extractions.

---

### LangChain DeepAgent: Production-Ready AI Agent with Specialized Skills

URL: https://aiwithvd.com/projects/deepagent
GitHub: https://github.com/aiwithvd/langchain_deepagent
Technologies: FastAPI, LangGraph, LangChain, Ollama, Redis, Python, Docker, SSE

A FastAPI service powering a LangChain DeepAgent backed by Ollama llama3.2:3b. Features four specialized skills for complex reasoning tasks, SSE streaming for real-time responses, Redis rate limiting, and multi-turn session memory.

Key features:
- 4 Specialized Skills: Think (reasoning), Plan (task breakdown), Web Search (DuckDuckGo), Write Report (structured output). Each skill has its own SKILL.md instruction file.
- Real-Time Streaming: Server-Sent Events (SSE) streaming for live agent responses. Watch the agent think and execute in real-time.
- Session Memory: Multi-turn conversation support with context preservation. The agent remembers previous interactions within a session.
- Rate Limiting: Redis-based rate limiting (10 requests/60s per IP) with health checks and graceful degradation.

Architecture:
- FastAPI with middleware stack (CORS, Logging, RateLimiter)
- LangGraph for orchestrating agent workflows
- ChatOllama with llama3.2:3b for local LLM inference
- 4 specialized tools with SKILL.md instruction files
- Redis for rate limit counters and session storage
- SSE streaming for real-time client updates

Impact: Executes complex multi-step tasks autonomously with observable reasoning steps and verifiable outputs.

---

### BoxCricket Umpier: AI-Powered Cricket Scoring for Gully Cricket

URL: https://aiwithvd.com/projects/boxcricket
GitHub: https://github.com/aiwithvd/boxcricket_umpier
Technologies: Next.js 14, TypeScript, Tailwind CSS, Zustand, Vitest, React Testing Library

A Next.js 14 mobile-optimized cricket scoring application for box cricket and gully cricket matches. Features ball-by-ball scoring, intelligent rules engine, live statistics, and smart undo functionality.

Key features:
- Ball-by-Ball Scoring: Intuitive interface for recording runs (0-6) and extras. Track every ball with a single tap.
- Smart Rules Engine: Configurable No Ball, Wide Ball, and Bouncer tracking with free hit logic and auto-conversions.
- Live Statistics: Real-time Current Run Rate (CRR), over progression, wickets tracking, and score updates.
- Smart Undo: Full state reversion including runs, balls, wickets, extras, and free hit status. Never lose track.

Architecture:
- Next.js 14 App Router for optimal performance
- TypeScript for type safety across the codebase
- Tailwind CSS for responsive mobile-first design
- Zustand for lightweight state management
- Vitest + React Testing Library for comprehensive testing
- Dark theme optimized for outdoor visibility

Impact: Used in local cricket tournaments with 100+ matches scored. Replaces pen-and-paper scoring with digital accuracy.

---

### Voice AI Demo: Self-Hosted Conversational Voice AI Assistant

URL: https://aiwithvd.com/projects/voiceai
GitHub: https://github.com/aiwithvd/voiceai
Technologies: LiveKit Agents, FastAPI, Python 3.12, Next.js 15, React 19, Tailwind CSS 4, Whisper, Ollama, Edge-TTS, Docker, WebRTC

A fully self-hosted, open-source conversational voice AI assistant powered by LiveKit Agents. Features real-time WebRTC audio transport, a configurable AI pipeline with VAD, STT, LLM, and TTS, a polished Next.js web UI, and Apple Silicon Metal GPU acceleration — all running locally with zero cloud dependencies.

Key features:
- Real-Time Voice Conversation: Low-latency WebRTC audio transport via LiveKit. Speak naturally and hear responses in real-time with sub-2s latency.
- 100% Self-Hosted: All AI models run locally — Silero VAD, faster-whisper STT, Ollama LLM, and Edge-TTS. No cloud dependencies, no API keys, no data leaving your machine.
- Configurable AI Pipeline: Swap STT, LLM, and TTS providers at runtime via .env. Use local models or switch to Groq, OpenAI, Cartesia, or any OpenAI-compatible API.
- Apple Silicon Optimized: Metal GPU acceleration for Whisper, Llama, and other models. Runs efficiently on MacBook hardware with minimal resource usage.

Architecture:
- LiveKit Agents SDK 1.5 for voice pipeline orchestration
- FastAPI for JWT token generation and health check endpoints
- Silero VAD for speech segment and utterance boundary detection
- faster-whisper for local STT transcription (large-v3-turbo)
- Ollama with llama3.2:3b for local LLM inference
- Edge-TTS for speech synthesis with natural-sounding voices
- Docker-based LiveKit WebRTC SFU for media routing
- SafeSTT/SafeLLM/SafeTTS wrappers for graceful error handling

Impact: Real-time voice conversation with full privacy — all models run locally, zero API costs, sub-2s response latency on Apple Silicon.

## Blog

### Tracing and Scoring a RAG Pipeline Without Slowing It Down

URL: https://aiwithvd.com/blog/tracing-and-scoring-rag-without-slowing-it-down
Published: 2026-09-05
Tags: Langfuse, DeepEval, LLM Evaluation, Observability, RAG, LangGraph, Production AI

# Tracing and Scoring a RAG Pipeline Without Slowing It Down

The question that decides whether evaluation survives contact with production is not "which metrics?" It is "when does the judge run?"

Run it inline and every answer waits for a second model to grade the first. Run it in a nightly batch and you learn about a bad week on Monday. The pipeline from [the first post](/blog/its-not-adaptive-rag-toggle-driven-hybrid-retrieval) does neither. **Score after you answer.** The judge starts the moment the response stream closes, on the same event loop, and it attaches its scores to a trace that was recording the whole time.

I wrote about *which* metrics matter in [Evaluating Generative AI in Production](/blog/evaluating-generative-ai-production). This post is the wiring.

## Trace first: one callback per request

Langfuse is opt-in. It runs as a Docker Compose profile (`--profile observability`) with its own Postgres, and it is off unless three environment variables are set. When it is on, the chat endpoint builds one callback handler per request and passes it into the graph's config:

```python
handler = make_chat_handler(thread_id, skill, payload.web_search)
if handler is not None:
    config["callbacks"] = [handler]
```

LangGraph propagates that handler to every node and every nested LLM call. The result is a single trace per request: `contextualize` → `retrieve_vector` → `rerank_vector` → `grade_documents` (one generation span per document) → `generate` → `reflect` (two spans) → `finalize`, each with its prompt, its tokens and its latency. The trace is grouped by `session_id = thread_id`, so a whole conversation reads as one session, and tagged with the skill and whether web search was on.

The same callback works whether the LLM is a local Ollama model or a hosted one. That was a requirement, not an accident: the graph LLM is pluggable and the tracing must not care.

Every node also logs `node=<name> elapsed_ms=<n>` and emits a `timing` event on the SSE stream. When something is slow, you do not guess. The [reranker fix from post one](/blog/its-not-adaptive-rag-toggle-driven-hybrid-retrieval) came straight out of those numbers.

## The stream, and the moment it closes

The SSE stream has a fixed order: `thread` (the id, so a client that omitted one can capture it), then interleaved `status`, `sources` and `token` events, then a terminal `done`. Tokens are filtered to the `generate` node only, so the grading and reflection calls never leak into the answer.

When the stream ends, the endpoint reads the final persisted state for the thread with `graph.aget_state(config)` rather than trusting what it accumulated. That gives it the answer, the source list, and which `[[n]]` ids the answer actually cited. It writes the citation map onto the trace, yields `done`, and only then does anything about scoring happen.

![Evaluation flow: the judge runs after the response closes, and its scores attach to the same trace](/diagrams/adaptive-rag-evaluation.svg)

## Fire and forget, with three safety nets

```python
task = asyncio.create_task(evaluate_and_score(trace_id, question, answer, chunks))
_eval_tasks.add(task)
task.add_done_callback(_eval_tasks.discard)
```

Three things in those lines are load-bearing.

**It is a task, not an await.** The response has already gone out. The judge runs on the event loop after `done`, and because judge calls are I/O-bound awaits, it does not starve other requests.

**The task is held in a module-level set.** A bare `create_task` with no reference can be garbage-collected mid-flight. The set keeps it alive; the done-callback removes it. This is the standard fire-and-forget pattern in asyncio and it is easy to get wrong.

**It cannot fail the request.** The scheduling call is wrapped in `try/except`. The body of `evaluate_and_score` is wrapped in `try/except` and logs. Each metric inside it is wrapped in `try/except`, so one metric failing does not drop the others. DeepEval itself is imported lazily inside the function, so if the package is missing or broken the chat path never notices. I looked hard for a way scoring could affect the response and did not find one.

## The judge is not the graph model

The graph runs on whatever `LLM_PROVIDER` says, often a small local model. Judging with the same small model produces scores that are indicative at best. So the judge has its own settings, `EVAL_PROVIDER` and `EVAL_MODEL`, falling back to the graph LLM only if unset. Score on something stronger, or on something local to protect a metered quota; the app does not care.

Two implementation details worth stealing:

- **Retry inside the judge.** Hosted providers return transient 5xx errors. The judge wrapper retries twice with linear backoff before giving up on a metric, so one blip does not fail a whole score.
- **Refuse the schema kwarg on purpose.** DeepEval will ask the judge for structured output when it can. Small and hosted models are unreliable at that. The judge's `generate` deliberately does not accept the `schema` argument, which raises a `TypeError` that DeepEval catches, falling back to plain text it parses itself. It is a documented hack, and it works.

## Three live metrics, no reference answer

On live traffic there is no ground truth, so the metrics are referenceless. DeepEval's GEval takes a natural-language rubric and returns a 0 to 1 score with a reason:

| Metric | Judges | Question it answers |
|---|---|---|
| `geval_retrieval` | input + retrieved chunks | Are the chunks relevant and sufficient to answer? |
| `geval_answer` | input + answer + chunks | Is the answer correct and faithful to the context, with no unsupported claims? |
| `contextual_relevancy` | input + chunks | What share of the context was actually useful? |

The third one is the tuning signal. A low relevancy score with a high retrieval score means the right chunks are in there but so is a lot of noise: chunk size or top-K is too generous. It can be switched off (`EVAL_CONTEXTUAL_RELEVANCY=false`) to save judge calls on a metered provider.

Scores land on the request's trace by id, so in Langfuse you open a run and see the nested spans, the prompts, the latency, the citation map and the three scores in one place. When a practitioner says "that answer was wrong", you have everything.

## Two metrics live traffic cannot compute

Recall and precision need a reference answer. "Did retrieval capture everything the correct answer needs?" is unanswerable without knowing the correct answer. So those two run offline, against a golden set per skill:

```bash
python -m app.eval.offline --skill regulatory-research
python -m app.eval.offline --limit 5          # quick subset
```

Each golden item is a question plus an `expected_output`. The harness runs each one through the *real* compiled graph, scores `contextual_recall` (did retrieval get everything the reference needs, which points at embeddings, chunking and top-K) and `contextual_precision` (are relevant chunks ranked above irrelevant ones, which is the reranker's report card), prints a table, and records a Langfuse dataset run so you can diff two commits.

The harness and the API share one `build_runtime()`. Whatever the app wires at startup, the offline evaluator wires identically. Evaluating a different graph than the one you ship is a classic way to be confidently wrong.

## What to watch

- **Judge cost.** Every live request adds judge calls. That is why it is off by default and why the judge is decoupled. Turn it on for a sample, not for everything, once you trust the numbers.
- **Small judges flatter.** On a tiny local judge the scores are directional. Point `EVAL_MODEL` at something strong before you put a number in a report.
- **A yes/no gate that defaults to yes.** The graph's own relevance and groundedness gates parse a one-word answer and default to "yes" when the model rambles. That bias is deliberate, it favours progress over spinning, but it means the gates get softer as the model gets smaller. The GEval scores are how you notice.

## The takeaway

- **Score after you answer.** The judge starts when the stream closes. Zero added latency, and the client never waits on a grader.
- **One callback, one trace.** Attach Langfuse per request and let LangGraph propagate it; every node and LLM call lands in a nested trace grouped by thread.
- **Fire and forget correctly.** `create_task`, hold the reference, wrap every layer. Scoring must not be able to fail the request.
- **Decouple the judge.** Separate provider and model settings, retries for transient errors, plain-text fallback for models that cannot do structured output.
- **Live for referenceless, offline for the rest.** GEval and relevancy on real traffic; recall and precision against a golden set, through the same runtime the app uses.

Previous: [Skills as Sub-Agents](/blog/skills-as-sub-agents-one-graph-many-personas). That closes the three-part series on the [adaptive-rag](https://github.com/aiwithvd/adaptive-rag) system: the human-chosen route and the capped self-reflection loop, the skill layer over one shared graph, and the observability that runs beside it without slowing it down.

---

### Skills as Sub-Agents: One Graph, Many Personas

URL: https://aiwithvd.com/blog/skills-as-sub-agents-one-graph-many-personas
Published: 2026-09-04
Tags: LangGraph, Skills, Sub-Agents, RAG, Allowlist, Regulated AI, System Design

# Skills as Sub-Agents: One Graph, Many Personas

The first version of the system had one pipeline and one system prompt. Then the practice asked for a second persona: not a general researcher but a retirement-planning strategist, with a different voice, a narrower set of sources, and a different idea of when to reach for the web.

The tempting move is to copy the graph. Two pipelines, two prompts, two sets of bugs. I did not want two of anything. What I wanted was for the *same* graph to behave like a different specialist depending on who invoked it.

That is what a skill is. **Same graph, different contract.**

## A skill is a folder

```
skills/regulatory-research/
  SKILL.md       # YAML frontmatter + markdown body
  skill.yaml     # sources, allowlist, web mode, ingestion seeds
  references/    # optional curated markdown
```

`SKILL.md` has a frontmatter block with a `name` and a `description` (the description powers the `/` menu in the UI), and a markdown body. The body *is* the system prompt for a run under that skill. Everything about voice, output format, source hierarchy and how to handle uncertainty lives there, in prose a domain expert can edit without touching Python.

`skill.yaml` is the contract:

```yaml
name: regulatory-research
web_mode: fallback
vector_sources: [regulator.example, legislation.example, caselaw.example]
web_allowlist:  [regulator.example, legislation.example, caselaw.example,
                 professional-body-one.example, professional-body-two.example]
ingest:
  references: true
  topics:
    - act_section: "Related-party loan rules"
      topic: "loans from a private company to its owners"
      query: "private company loan to shareholder deemed distribution complying loan"
```

The `references/` bundle is a set of paraphrased practitioner notes. It is ingested into the vector store tagged `skill=regulatory-research` and `source_type=reference`, so retrieval ranks curated notes alongside crawled official pages, and the citation can still deep-link to the canonical URL declared in the note's frontmatter.

A `SkillRegistry` scans `skills/*/SKILL.md` once at startup. Adding a persona is: create the folder, restart, ingest. No code.

## Invoking one

Two ways, and they resolve to the same thing:

```json
{ "question": "/regulatory-research how do the related-party loan rules work?", "thread_id": "t-1", "web_search": false }
{ "question": "how do the related-party loan rules work?", "thread_id": "t-1", "skill": "regulatory-research" }
```

A leading `/slug` token is parsed off the question, exactly like a slash command. The explicit `skill` field wins if both are present. An unknown slug is a 400, not a silent fallback to the generic pipeline, because a practitioner who typed `/retirement-planning` and got a generic answer would not notice, and that is worse than an error.

## Four points on one graph

![The full architecture. The skill routing band at the top parametrizes the pipeline below it](/diagrams/adaptive-rag-architecture.svg)

The graph does not know skills exist. Its nodes look up `state["skill"]` in the registry and read four values:

| Node | What the skill supplies |
|---|---|
| `contextualize` | `web_mode`, which decides how the router treats the web toggle |
| `retrieve_vector` | a Milvus filter built from `vector_sources`, so the skill only sees its own references plus official pages from its declared hosts |
| `web_search` | `web_allowlist`, the only hosts the fallback may fetch |
| `generate` | the `SKILL.md` body as the system prompt |

Everything else, the reranker, the four gates, the loop caps, the citation numbering, is identical for every skill. Fix a bug once and every persona gets the fix.

The filter in `retrieve_vector` deserves a sentence. It is a boolean expression over the collection's metadata: `(source_type == "reference" and skill == "<slug>") or (source_type == "vector" and domain in [...])`. A skill can therefore be regulator-only. The retirement-planning skill declares the same three official hosts but a different professional body, so its web fallback reads a specialist industry association the research skill never sees.

## Isolation: `thread_id::skill`

A skill run happens on a derived thread. If the client thread is `t-1`, the research skill's memory lives at `t-1::regulatory-research` in the SQLite checkpointer, and the plain chat thread at `t-1` never sees it.

This matters because `contextualize` condenses prior turns into the current question. Without isolation, a follow-up in the generic thread would be rewritten against a specialist's conversation, and a specialist's follow-up against generic chat. Each persona keeps its own history and the client keeps one id.

The response's `done` event carries the skill name back, so the frontend can label the thread.

## Web-credible fallback

The base `/chat` path runs web search *alongside* vector whenever the toggle is on. Skills flip that to `fallback`: web runs only when the vector documents cannot answer (nothing survives the relevance gate) **and** the toggle is on. The knowledge base is primary and the web is a safety net, which is exactly how the practitioner thinks about it.

A one-shot guard (`web_searched`) means the fallback fires at most once per turn. After a rewrite it will not fire again, even though the toggle is still on. That was deliberate, and the graph's termination proof depends on it.

## The allowlist, enforced three times

The allowlist is the safety property of the whole system. The glossary in the repo said it was "enforced twice", and for months I believed it.

**Check 1, on the query.** The search runs once per allowed domain with a `site:` filter appended, so the search engine only returns candidates from those hosts.

**Check 2, before the fetch.** Every returned URL's host is checked against the allowlist: an exact match or a subdomain. Search-engine noise and lookalike domains are dropped here.

**Check 3, after the fetch.** This one did not exist. The HTTP client followed redirects, and the host was verified on the URL we *asked for*, never on the URL we *got*. An allowlisted page that returned a 302 to an arbitrary host would have had its content fetched, extracted, and quoted back to a practitioner with an official-looking citation.

The fix is a few lines: check the host of `response.url` after the redirect chain, drop and log if it is off-list, and cite the final URL rather than the pre-redirect one so the citation points where the content actually came from. It came with a test that mocks a 302 to an off-list host and asserts the document is dropped, a test that a same-host redirect is kept with the final URL recorded, and a test for lookalike domains.

The docstring and the glossary now say "three times". The gap between what documentation claims and what code does is where these bugs live.

## Adding a persona in practice

When the second skill arrived, the whole change was two files, a restart and one ingestion command:

```bash
python -m app.ingestion.run_ingest --skill retirement-planning
curl -s localhost:8000/api/v1/skills   # both listed
```

Ingestion for a skill pulls its `references/*.md` (tagged reference) and runs each `ingest.topics` seed through the *same* allowlisted search-and-fetch path the live fallback uses. So what the vector store holds is exactly what a live web fallback would have fetched. There is one code path for reading the web, and the allowlist sits in it.

## The takeaway

- **A skill is data, not code.** `SKILL.md` for the persona, `skill.yaml` for the contract, an optional references bundle. The registry loads it at startup.
- **Four parametrization points, one graph.** Web mode, source filter, allowlist, system prompt. Everything else is shared, so every fix lands everywhere.
- **Isolate the thread.** `thread_id::skill` keeps each persona's memory apart while the client holds one id.
- **Fallback, not alongside.** For a specialist, the knowledge base is primary and the web is a one-shot safety net.
- **Verify the URL you got, not the one you asked for.** Redirects are where an allowlist quietly stops being one.

Previous: [It's Not Adaptive RAG](/blog/its-not-adaptive-rag-toggle-driven-hybrid-retrieval). Next: [Tracing and Scoring a RAG Pipeline Without Slowing It Down](/blog/tracing-and-scoring-rag-without-slowing-it-down), how every run becomes a nested trace and gets scored by a judge model after the client already has its answer.

---

### It's Not Adaptive RAG: Why I Let the Human Choose the Route

URL: https://aiwithvd.com/blog/its-not-adaptive-rag-toggle-driven-hybrid-retrieval
Published: 2026-09-03
Tags: RAG, LangGraph, Self-Reflection, Retrieval, Regulated AI, FastAPI, Milvus

# It's Not Adaptive RAG: Why I Let the Human Choose the Route

The repo is called `adaptive-rag`. The first line of its README says it is not adaptive RAG. Both are true, and the gap between them is the most useful design decision in the project.

Classic adaptive RAG puts an LLM at the front door. The model reads the question, decides whether to answer from the vector store or go to the web, and picks one route. It sounds smart. In a regulated finance and legal practice it is a compliance problem waiting to happen: a model that silently decides to leave the knowledge base and read the open web, with nobody able to say afterwards why it did.

So I moved the decision. **The router is a person.**

## Vector always, web only when asked

Every request to `POST /chat` carries three things: the question, a `thread_id`, and a boolean `web_search`. The vector store is searched on every single request, no exceptions. Web search is *additive*. It runs only when the toggle is on, and it never replaces the vector results, it sits beside them.

That gives you three properties you cannot get from a model-driven router:

1. **The route is on the request**, so it is in the trace. When someone asks "did this answer touch the web?", the answer is a field, not a guess.
2. **Off by default means nothing leaves the knowledge base unasked.** The user opts in, per question.
3. **The allowlist does the rest.** When web search does run, it can only read a short list of official sources: the regulator, the legislation register and the case-law database configured for the deployment. Nothing else. I'll come back to how that list is enforced in [the next post](/blog/skills-as-sub-agents-one-graph-many-personas).

The trade-off is honest: the user has to know when to flip the switch. In practice they know far better than the model does. A practitioner asking about a rule they use every week wants the curated knowledge base. The same practitioner asking about a ruling published last month wants the web. They can tell the difference. The router cannot.

## The graph, and where the model actually gets a say

The model does make decisions, just not about *where* to look. It decides whether what came back is *good enough*. That is the self-reflection part, and it is the reason the pipeline is a graph rather than a chain.

![The compiled LangGraph, exported from the live code](/diagrams/adaptive-rag-langgraph.svg)

Twelve nodes. The spine is `contextualize` (condense the thread history into a standalone question), `retrieve_vector` (Milvus, HNSW, cosine), `rerank_vector` (a cross-encoder cuts the candidate pool to the top N), then the first of four gates.

**Gate 1, `grade_documents`.** Each retrieved chunk gets one LLM call: is this relevant to the question, yes or no? Anything graded no is dropped. If nothing survives and the toggle is off, the graph does not generate. It routes to `rewrite_query`, tries retrieval once more with a keyword-richer question, and if that also comes back empty it routes to `give_up`, which returns a plain "I don't have enough in the knowledge base to answer this reliably" and no fabricated citations.

**Gates 2 and 3, `reflect`.** After `generate` streams an answer, two more calls: is the answer grounded in the context it was given? Does it actually address the question? Not grounded routes to `prepare_regen` and generates again. Off topic routes back to `rewrite_query` and retrieves again.

**Gate 4 is the rewrite itself.** It is a retry with a better question, and it sits inside both of the other loops.

## Why it always terminates

Two loops that can each send the graph backwards is how you get a pipeline that spins forever on a bad question and a small model. The fix is boring and it is the part I am proudest of.

Both loops are hard-capped. `MAX_REWRITES` and `MAX_REGEN` both default to 1. The counters are incremented in exactly one node each (`rewrite_query` and `prepare_regen`), both routing functions read the counter before they route, and nothing resets a counter mid-turn. `contextualize` zeroes them once at the start of every request and never again.

I traced every path. With the defaults, the worst case is two extra passes, then `finalize`. There is no input that makes it loop. That property is worth more than any prompt.

## The answer the user actually sees

Two details at the output end matter more than they look.

The context handed to `generate` is source-labelled. Every chunk arrives as `[REFERENCE]` (a curated practitioner note), `[VECTOR]` (an ingested official page) or `[WEB]` (a live fetch), numbered `[[1]]`, `[[2]]` and so on. The model cites by number. The client receives a `sources` event before the first token, so it can resolve every `[[n]]` the moment it appears.

And each citation is a deep link. For vector and web sources the URL carries a text fragment (`#:~:text=`) built from the cited passage, so clicking it opens the official page scrolled to the exact sentence. In a domain where "show me where it says that" is the whole job, that one feature earned more trust than anything the model said.

## What I got wrong and fixed

Reviewing the code for this post turned up two things I would not have admitted a month ago.

**The reranker was blocking the event loop.** The cross-encoder is synchronous, CPU-bound torch work, and I was calling it inline from an async node. Under concurrent load, one request's reranking froze every other request's token stream for the duration. The DuckDuckGo call in the same file was already wrapped in `asyncio.to_thread`. The reranker now is too. The README had a whole section on diagnosing latency with per-node `elapsed_ms` logs, and the biggest source of it was a missing `to_thread`.

**The rate limiter could be reset by the caller.** The Redis key was `{client_ip}:{thread_id}`, and `thread_id` is client-supplied. A caller that sends a fresh thread id per request never accumulates a count. The key is now the client alone. While I was there, a Redis blip used to turn into a 500 for every chat request; the limiter now fails open with a warning in the log, because losing the limiter for a minute is better than losing the service.

Both fixes are in the [repo](https://github.com/aiwithvd/adaptive-rag) with tests.

## The takeaway

- **Let the human choose the route.** In a regulated domain, a model that silently decides to leave the knowledge base is an audit problem. A toggle on the request is a field in the trace.
- **Vector always, web additive.** Web search runs beside vector results, never instead of them, and only over an allowlist.
- **Four gates, two loops, both capped.** Relevance, groundedness, answer quality and a rewrite retry, with counters that live in one node each. The graph cannot spin.
- **Give up honestly.** When nothing relevant survives and the toggle is off, return "not in the knowledge base" and no citations.
- **Cite the sentence, not the page.** Source-labelled context plus text-fragment links is what makes a practitioner trust the output.

This is the first of three posts on the system. Next: [Skills as Sub-Agents](/blog/skills-as-sub-agents-one-graph-many-personas), how one `SKILL.md` and one `skill.yaml` parametrize four points of this same graph, run on an isolated thread, and carry their own allowlist.

The regulated-domain thread continues from [the verification layer](/blog/building-verification-layer-llm-outputs) post. That one was about catching fabricated citations after generation. This one is about retrieving well enough that there is less to catch.

---

### Fine-Tuning a Tool-Calling Agent: SFT + QLoRA on Gemma 3 4B

URL: https://aiwithvd.com/blog/fine-tuning-tool-calling-agent-sft-qlora-gemma
Published: 2026-06-15
Tags: Fine-Tuning, SFT, QLoRA, LangGraph, ReAct Agent, MCP, Tool Calling, Gemma

# Fine-Tuning a Tool-Calling Agent: SFT + QLoRA on Gemma 3 4B

The agent worked. It also cost too much and drifted.

It was a **LangGraph ReAct agent** sitting over a **custom MCP server that exposed 40+ tools** into an existing backend. The model was a large general-purpose API model, prompted with the full tool catalog on every turn. It was capable — but each multi-step task burned a long context of tool schemas, latency was high, and the tool-call formatting wobbled enough that the harness occasionally failed to parse a call.

The fix wasn't a better prompt. It was teaching a *small* model to use *these specific tools* well — **SFT + QLoRA on Gemma 3 4B**. This post is the case study that ties together [the two axes](/blog/two-axes-of-fine-tuning), [the transformer block](/blog/inside-a-transformer-block), [SFT](/blog/sft-what-the-model-predicts), and [LoRA & QLoRA](/blog/lora-qlora-fine-tuning-one-gpu) — applied to one real system.

## Why fine-tune instead of prompt

Three reasons made fine-tuning the right call here, not premature optimization:

1. **The tool surface was fixed and large.** 40+ tools with stable schemas. Re-describing them in the prompt every turn is paying, in tokens and latency, for knowledge the model could just *have*.
2. **The behavior was repetitive and verifiable.** The agent did the same families of multi-step tasks. There were plenty of successful runs to learn from.
3. **Format reliability mattered.** A flaky tool-call format breaks the whole trajectory. A model trained on the exact format emits it far more consistently than one steered by a prompt.

On [the two axes](/blog/two-axes-of-fine-tuning): the signal is **SFT** (imitate successful trajectories) and the weight method is **QLoRA** (fit Gemma 3 4B comfortably on one GPU). A clean "one from each axis" choice.

## Step 1: the dataset is the product

The agent had been running in production, which meant it was already generating exactly the data I needed: **multi-turn tool-use trajectories.** The work was turning that exhaust into a clean training set.

![Building the fine-tuning dataset from agent tool trajectories](/diagrams/agent-finetune-data-pipeline.svg)

The pipeline:

1. **Filter.** Keep only successful, verified runs — trajectories where the task actually completed correctly. Drop failures, dead-ends, and near-duplicates. A failed trajectory teaches the model to fail.
2. **Format.** Render each trajectory as a multi-turn conversation using **Gemma's own chat template**, with tool calls and tool results in the exact structure the harness uses. Then apply masking (next section).
3. **Dataset.** The output is a curated, on-distribution multi-turn SFT set. This is "golden trajectories" — hand-verified perfect runs — plus rejection-sampled completions (generate many, keep only the ones that pass).

Most of the effort lives in steps 1 and 2. Garbage in, garbage out: the dataset *is* the product.

## Step 2: masking in multi-turn data

This is where the [SFT post's](/blog/sft-what-the-model-predicts) masking detail gets sharper. In a single Q&A row you mask the prompt and compute loss on the response. In a **multi-turn agent trajectory**, there are many turns — user messages, assistant tool calls, tool results, more assistant turns — and you have to mask the *right* ones.

The rule: **every assistant turn carries loss; every user and tool turn is masked to `-100`.**

```text
<user>      task description            → masked (-100)
<assistant> tool_call: lookup(args)     → LOSS  (learn to call the tool)
<tool>      {result: ...}               → masked (-100)
<assistant> tool_call: update(args)     → LOSS  (learn the next step)
<tool>      {ok: true}                  → masked (-100)
<assistant> final answer                → LOSS  (learn to finish)
```

Get this wrong — mask the assistant turns, or compute loss on tool results — and the model learns to *generate the user's side of the conversation* or *hallucinate tool outputs*. Both are exactly the failure you don't want in an agent. Tools like TRL's collators handle multi-turn response masking, but you must verify it on a real tokenized row, not assume it.

## Step 3: train with QLoRA

Gemma 3 4B is small enough that QLoRA makes training comfortable on a single GPU. From the [LoRA & QLoRA post](/blog/lora-qlora-fine-tuning-one-gpu): a 4-bit NF4 frozen base, BF16 LoRA adapters, a paged optimizer for memory spikes.

```python
from peft import LoraConfig
from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype="bfloat16",
)

lora_config = LoraConfig(
    r=16,                       # modest rank — matched to dataset size
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=[            # attention AND MLP — tool behavior + domain knowledge
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    task_type="CAUSAL_LM",
)
# optimizer: paged_adamw_8bit  ·  1–3 epochs  ·  base stays frozen
```

Two decisions worth calling out, both straight from earlier posts:

- **Target the MLP, not just attention.** From [the transformer-block post](/blog/inside-a-transformer-block): the tools carry domain vocabulary and behavior, so the adapters touch `gate_proj`, `up_proj`, `down_proj` too — not just `q/k/v/o`. Attention-only would have produced a fluent agent that didn't really learn the tools.
- **Modest rank, few epochs.** Rank 16, 1–3 epochs. Cranking rank on a finite trajectory set overfits phrasing; few epochs protect the base model's general ability (and QLoRA's frozen base resists catastrophic forgetting by construction).

## Step 4: serve it back behind the agent

The fine-tuned model drops into the same ReAct loop it learned from — nothing about the orchestration changes.

![The tool-calling loop a ReAct agent runs](/diagrams/agent-tool-calling-loop.svg)

A user message arrives with the tool schemas; the model emits a structured tool call (name + JSON args); the harness runs the matching MCP tool and feeds the result back as a tool message; repeat until the model emits a final answer. The whole system looks like this:

![Architecture of the fine-tuned tool-calling agent](/diagrams/agent-finetune-architecture.svg)

The LangGraph agent orchestrates. It calls the **fine-tuned Gemma 3 4B** to decide each step and the **MCP tools** to act, against the existing backend. The fine-tune didn't replace the architecture — it replaced the *expensive, prompt-steered decision model* with a small one that already knows these tools.

The one thing you must keep identical between training and serving is the **chat template**. Train on Gemma's template, serve on Gemma's template — via `apply_chat_template` on both sides. A mismatch here is the silent quality collapse from [the SFT post](/blog/sft-what-the-model-predicts), and in an agent it shows up as rising tool-call parse failures.

## What changed, and what to watch

The wins were the ones the three reasons predicted: the tool catalog no longer had to live in every prompt, latency and cost dropped with the smaller model, and tool-call formatting got far more reliable because the model had seen the exact format thousands of times.

The things I watched closely:

- **Tool-call parse rate** as a first-class metric. If it ever drifts, the template or the format is wrong — catch it early.
- **Read real multi-turn generations,** not the loss curve. An agent can have great loss and still loop, call the wrong tool, or fabricate a result. Curves lie; trajectories don't.
- **State, not just final text.** When evaluating, check that the right tools were called in a legal order and the backend ended in the right state — not only that the final sentence looked good. (That reward/verifier design is its own topic, coming next in the series.)

## The takeaway

- **Fine-tune when the tool surface is fixed, the behavior is repetitive, and format reliability matters.** Otherwise keep prompting.
- **The dataset is the product.** Mine successful production trajectories, filter hard, format with the model's own chat template.
- **Mask every non-assistant turn.** In multi-turn agent data, loss goes on assistant turns only — or the model learns to write the user and hallucinate tool outputs.
- **QLoRA on a small model, MLP included.** 4-bit base, BF16 adapters on attention *and* MLP, modest rank, few epochs — and keep the chat template identical from training to serving.

This is the payoff of the whole series so far: the two axes, the transformer block, SFT, and QLoRA, composed into one working system. Next, I'll go up a level into **reward design and verifiers** — how you'd push this agent further with RL by scoring whether it took the right actions, not just whether the text looked right.

---

### LoRA & QLoRA: Fine-Tuning a Model That Doesn't Fit on Your GPU

URL: https://aiwithvd.com/blog/lora-qlora-fine-tuning-one-gpu
Published: 2026-06-14
Tags: LoRA, QLoRA, PEFT, Quantization, NF4, Fine-Tuning, GPU Memory

# LoRA & QLoRA: Fine-Tuning a Model That Doesn't Fit on Your GPU

The first time I tried to full fine-tune a 7B model, the math stopped me before the code did. With the Adam optimizer you need roughly **16 bytes per parameter** — weights, gradients, and two optimizer states. That's about **112GB** for a 7B model, before a single batch of data. My GPU had 48GB.

There are two ways out, and they stack. **LoRA** stops updating almost all the weights. **QLoRA** shrinks the frozen ones to 4 bits. Together they put a large model on a single card.

## LoRA: train a thin detour, freeze the rest

The insight behind LoRA is that the *change* needed to adapt a model is "low-rank" — you don't need to move all of a giant weight matrix's numbers, just a small correction. So you freeze the big matrix and learn a thin detour beside it.

![How LoRA adds a trainable detour beside a frozen weight](/diagrams/lora-detour.svg)

The input flows two ways: through the big frozen weight `W` (never trained), and through a small trainable detour made of two low-rank matrices, `A` then `B`. The outputs are added. Only `A` and `B` get gradients — roughly **1% of the parameters** — so the optimizer state you have to hold collapses from "the whole model" to "a thin adapter."

One detail matters more than it looks: **`B` is initialized to zero.** At step 0 the detour contributes nothing, so training starts *exactly* at the pretrained model and drifts gently from there. (If you randomly initialized both `A` and `B`, you'd inject a large random perturbation at step 0 — effectively corrupting the pretrained model before training even begins, and the run becomes unstable. `A` random, `B` zero is the safe, smooth start.)

### Sticky notes on a textbook

The analogy I use to explain this to non-specialists:

![LoRA as sticky notes on a printed textbook](/diagrams/lora-sticky-notes.svg)

The frozen base is a **printed textbook you're not allowed to rewrite**. LoRA is **sticky notes in the margins**. The book stays untouched; your handful of sticky notes (the tiny `A·B` matrices) carry all the task-specific corrections. The final model is the book plus the notes. And rank `r` is *how much you're allowed to write on each note* — too little and you can't capture the change, too much and you're effectively rewriting the book (overfit, more memory).

### The knobs

| Knob | What it controls | Typical | Trade-off |
|------|------------------|---------|-----------|
| `r` (rank) | capacity of the adapter | 16 or 32 (range 8–128) | ↑ more expressive but more VRAM & overfit; ↓ cheaper but may underfit |
| `lora_alpha` | update scaling (effective ≈ α/r) | r or 2r | too high → unstable/over-strong updates |
| `lora_dropout` | regularization on the adapter | 0 | ↑ helps tiny datasets overfit less |
| `target_modules` | which matrices get an adapter | all 7 (attn + MLP) | attn-only = fewer params but underfits knowledge-heavy domains |
| learning rate | step size | ~2e-4 | too high → loss spikes; too low → underfits |

That `target_modules` row is the one from [the transformer-block post](/blog/inside-a-transformer-block): for domain *knowledge*, include the MLP (`gate_proj`, `up_proj`, `down_proj`), not just attention. And a common trap to be ready for — *"you raised `r` from 16 to 256 on a 2,000-row dataset and quality dropped, why?"* — is overfitting: rank 256 gives the adapter far more free parameters than 2,000 rows can constrain, so it memorizes phrasing and loses generalization. Match rank to data size and task complexity, not "bigger is better."

## QLoRA: put the frozen base in 4 bits

LoRA shrinks the *trainable* footprint. The frozen base is still sitting in memory at full precision. QLoRA attacks that — it quantizes the frozen base to 4 bits so it takes ~4× less space, then runs LoRA on top. It's four parts, and interviewers probe all of them.

![The four pieces that make QLoRA work](/diagrams/qlora-four-pieces.svg)

**1 — NF4 (the big saving).** The base weights are stored in a 4-bit format. The clever bit: NF4's 16 levels sit at the **quantiles of a normal distribution**. This trips people up, so say it precisely: NF4 does *not* "make the weights normal" — transformer weights are *already* approximately normal, and NF4 places its 16 levels to match that, with more levels where weights cluster (near zero) and fewer in the tails. That minimizes quantization error versus uniform spacing.

**2 — Double Quantization.** Weights are quantized in blocks, and each block needs a scale to reconstruct it. For a large model that's millions of scales — enough metadata to matter. So QLoRA quantizes the scales too. Think of zipping a folder where the *file index* itself also got compressed.

**3 — LoRA adapters.** The same low-rank adapters as before, kept in BF16/FP16 because adapter training needs higher precision for stable updates. They're already tiny, so this costs little — the big saving came from quantizing the *base*, not the adapter.

**4 — Paged Optimizers.** Optimizer state can *spike* during training (AdamW holds momentum and variance per parameter), and a spike past your GPU's memory is a `CUDA OOM` crash. Paged optimizers act like OS virtual memory: spill optimizer state to **CPU RAM** during spikes so training continues instead of dying.

The single most important QLoRA fact to state out loud: **gradients flow only into the LoRA adapters.** The 4-bit base stays frozen — it's dequantized on the fly purely to compute activations, never updated. A classic production bug is accidentally setting `base_model.requires_grad_(True)`: now gradients are computed for all ~billions of base params instead of just the adapters → CUDA OOM, training crawls, optimizer memory explodes.

### The QLoRA config in practice

```python
from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,                 # store the frozen base in 4-bit NF4
    bnb_4bit_quant_type="nf4",         # NF4 levels tuned for normal-distributed weights
    bnb_4bit_use_double_quant=True,    # quantize the per-block scales too
    bnb_4bit_compute_dtype="bfloat16", # dequantize to bf16 for the actual math
)
# optimizer: paged_adamw_8bit  → survives memory spikes + 8-bit states save memory
# the base stays frozen; only the LoRA A/B matrices are trainable
```

`load_in_4bit` saves >75% VRAM versus 16-bit at a small accuracy cost; the base must stay frozen so gradients flow only into `A` and `B`.

## How I actually used this

For the tool-using assistant on **Gemma 3 4B**, QLoRA was the whole reason it fit comfortably and trained fast on a single GPU: the 4-bit NF4 base plus BF16 LoRA adapters on attention *and* MLP, with a paged optimizer so a mid-run memory spike didn't kill a long job. Rank stayed modest (matched to the dataset size, not cranked up), and the base stayed frozen — which, as a bonus, is also why LoRA resists catastrophic forgetting: the general knowledge lives in weights that *can't* be overwritten. I'll put the full pipeline together — data, training, and serving behind the agent — in the [case-study post](/blog/fine-tuning-tool-calling-agent-sft-qlora-gemma).

This all lives on Axis B from [the pillar post](/blog/two-axes-of-fine-tuning): LoRA and QLoRA are *how you update the weights*, independent of *what signal* you train on. You can pair either with SFT, DPO, or GRPO.

## The takeaway

- **Full fine-tuning is memory-bound.** ~16 bytes/param with Adam → ~112GB for 7B. LoRA and QLoRA exist to dodge that.
- **LoRA trains a low-rank detour.** Freeze `W`, learn `A·B` (~1% of params), `B=0` for a safe start. Rank = capacity; match it to your data.
- **QLoRA = four pieces.** NF4 4-bit base (big saving), double-quant the scales, BF16 adapters, paged optimizer for spikes.
- **Gradients touch only the adapters.** The 4-bit base is frozen and dequantized only to compute — which is also why LoRA resists catastrophic forgetting.

Next in the series: the case study that puts it all together — [Fine-Tuning a Tool-Calling Agent: SFT + QLoRA on Gemma 3 4B](/blog/fine-tuning-tool-calling-agent-sft-qlora-gemma).

---

### SFT: What the Model Is Actually Predicting (and the Mask That Decides If It Works)

URL: https://aiwithvd.com/blog/sft-what-the-model-predicts
Published: 2026-06-13
Tags: SFT, Supervised Fine-Tuning, Chat Templates, Response Masking, Catastrophic Forgetting, Fine-Tuning

# SFT: What the Model Is Actually Predicting (and the Mask That Decides If It Works)

Take the simplest possible training example:

```json
{"instruction": "What is the capital of France?", "response": "Paris"}
```

Most people describe what happens next as "the model learns the answer is Paris." That sentence is the line between someone who has *used* SFT and someone who *understands* it — because mathematically, that is not what happens.

## SFT is pretraining with curated data

Supervised fine-tuning uses the **exact same objective as pretraining: next-token prediction.** Nothing about the loss is new. The model minimizes cross-entropy over a sequence of tokens, predicting each token from the ones before it. "Learning Paris" is an *emergent side-effect* of getting good at predicting the tokens `P`, `a`, `r`, `i`, `s` in that context.

What changes from pretraining isn't the loss. It's three things around it: **the data** (curated instruction/response pairs instead of raw web text), **the masking** (where loss is applied), and **the template** (the special-token structure). Get those right and a base "document completer" turns into an "assistant." Get them wrong and the loss curve looks beautiful while the model rambles.

Think of it as a student copying from a worked answer key. The teacher shows the question *and* the ideal answer; the student practices reproducing the answer token by token. Two subtleties in that picture are exactly where SFT succeeds or fails.

## Subtlety one: response masking

Here is one training row after the chat template and tokenization, split into its two parts.

![Response masking in one SFT training row](/diagrams/sft-response-masking.svg)

The prompt tokens (`<user> What is the capital ... <assistant>`) are labelled `-100`. That magic value tells the loss function to **ignore them** — no loss is computed on the prompt. Loss is computed only on the **response** tokens (`Paris <end>`).

Why mask the prompt? You don't want the model spending its capacity learning to *generate the question*. You want it to learn the *answer, given the question*. Grade the student on what they write, not on re-copying the prompt back. In TRL this is `train_on_responses_only` / `DataCollatorForCompletionOnlyLM`, and forgetting it is a silent quality drop — the model wastes gradient learning to echo prompts.

This also answers a common interview question — *"if SFT is just next-token prediction like pretraining, why does it change behavior so much?"* Because behavior is shaped by the **data distribution and the masking**, not the loss. Same objective, but now every gradient pushes toward *"after an assistant header, given an instruction, produce a helpful response,"* and the mask spends that gradient only on the response.

## Subtlety two: the chat template

The second detail is the one I've seen waste the most time, because it fails *silently*.

![Chat template mismatch, the number one silent SFT failure](/diagrams/sft-chat-template-mismatch.svg)

A model learns that the answer comes after a specific special-token structure — `<im_start>assistant`, or whatever its template uses. If you **train** with one template and then **infer** with a different one (or none), the conditioning is wrong. The model still produces output. No error is thrown. The quality just collapses.

The rule: always use the tokenizer's own `apply_chat_template` on **both** sides — training and inference — so the exact token structure matches. Never hand-roll the format string. When someone says "train loss dropped beautifully but the model rambles at inference," template mismatch is the first thing I check.

## The loop itself

Stepping back, the whole thing is one loop. Only the masked-loss box is special to SFT; everything else is the universal training loop from [the pillar post](/blog/two-axes-of-fine-tuning).

![The SFT training loop](/diagrams/sft-training-loop.svg)

Take a batch of prompt/answer pairs, apply the chat template and tokenize, forward pass to logits, compute cross-entropy **on the response tokens only**, backpropagate, let AdamW update the weights, repeat for 1–3 epochs. The one curve to watch is validation loss against training loss — when train keeps dropping but validation turns up, you're overfitting, and on a narrow dataset that shows up as **catastrophic forgetting**: the model gets better at your task and worse at everything else.

## The knobs that matter

A few hyperparameters carry most of the outcome:

| Knob | Typical | Trade-off |
|------|---------|-----------|
| learning rate | ~2e-4 (LoRA) · 1e-5–2e-5 (full FT) | too high → loss spikes / repetition; too low → underfits |
| epochs | 1–3 | more → better fit but ↑ catastrophic forgetting |
| max sequence length | match your data | longer → much more memory (attention is quadratic) |
| response masking | on | off → wastes capacity learning the prompt |
| warmup ratio | 5–10% of steps | too little → early instability |

The production failures to name out loud: **catastrophic forgetting** (fix with LoRA so the base can't be overwritten, lower LR, fewer epochs, mix in general data); **"loss great, model bad"** (overfit phrasing, template mismatch, or you trained on prompt tokens — read actual generations, not the curve); and template mismatch as above.

## How this shaped a real fine-tune

When I built the SFT dataset for a tool-using assistant on **Gemma 3 4B**, the data was **multi-turn conversations**, not single Q&A rows — which makes masking more interesting: across a conversation, *every* assistant turn carries loss and *every* user/tool turn is masked to `-100`. Get that wrong and the model learns to generate the user's side of the conversation, which is exactly the failure mode you don't want in an agent. I used Gemma's own chat template via `apply_chat_template` end to end, kept epochs low to protect the base model's general ability, and read real multi-turn generations rather than trusting the loss curve. The next posts cover the [LoRA/QLoRA setup](/blog/lora-qlora-fine-tuning-one-gpu) underneath this and the [full agent case study](/blog/fine-tuning-tool-calling-agent-sft-qlora-gemma).

## The takeaway

- **SFT is next-token prediction.** Same objective as pretraining; what changes is the data, the masking, and the template.
- **Mask the prompt.** Loss on response tokens only — otherwise you waste capacity learning to echo questions. In multi-turn data, mask every non-assistant turn.
- **Match the chat template exactly.** Train and infer with the tokenizer's own `apply_chat_template`. Mismatch fails silently.
- **Curves lie, read generations.** Great train loss with bad output means overfit, masking bug, or template mismatch.

Next in the series: [LoRA & QLoRA](/blog/lora-qlora-fine-tuning-one-gpu) — how to run this exact loop on a model far too big for your GPU, by freezing the base and training a thin adapter.

---

### Inside a Transformer Block: Why Where Knowledge Lives Decides Where You Fine-Tune

URL: https://aiwithvd.com/blog/inside-a-transformer-block
Published: 2026-06-12
Tags: Transformers, Attention, MLP, LoRA, Fine-Tuning, Model Internals

# Inside a Transformer Block: Why Where Knowledge Lives Decides Where You Fine-Tune

Here's a mistake I made early, and watched plenty of others make since: I set up a LoRA fine-tune, targeted `q_proj`, `k_proj`, `v_proj`, `o_proj` because that's what every tutorial showed, ran it on a specialized medical dataset — and the model came out fluent but no smarter about the domain. It phrased things nicely. It didn't *know* more.

The reason is structural. I was adapting the part of the transformer that decides *what to look at*, and leaving untouched the part that actually *stores knowledge*. To see why that matters, you have to open up a single transformer block.

## A block has two machines, not one

People learn the four attention projections and assume that's the transformer. But every block in every modern model — Llama, Qwen, Mistral, Gemma — has **two** machines stacked together, and they do completely different jobs.

![The two machines inside every transformer block](/diagrams/transformer-block-attention-mlp.svg)

**Attention** (`q_proj`, `k_proj`, `v_proj`, `o_proj`) answers *"which tokens should I look at?"* It's routing and retrieval — it moves information between positions, discovering relationships and context. After a residual add, the result flows into the second machine.

**The MLP** (`gate_proj`, `up_proj`, `down_proj`) answers *"what should I do with it?"* This is computation and, crucially, **knowledge storage**. The facts and task-specific patterns the model has learned live largely in these weights.

The catchphrase I use to keep it straight: **attention retrieves, the MLP remembers.**

### A worked example

Take the input *"The capital of France is Paris."* Attention discovers the link `France → Paris` — the relationship, the context. But attention alone doesn't *create* the knowledge. The MLP is where the pattern `France → capital → Paris` is actually encoded in the weights. Attention found the connection; the MLP is what holds the fact.

## The library researcher

If the projection names blur together, this analogy makes it stick.

![The library researcher analogy](/diagrams/transformer-library-analogy.svg)

Attention is the researcher walking the library and pulling the relevant books off the shelves. The MLP is the researcher sitting down and reasoning over those books to produce an answer. Pulling books (routing) is useless without a brain to process them; a brain is useless if it can't fetch the right books. You need both — but only one of them is where the *content of the books* is stored.

## What the MLP actually does inside

The MLP isn't a single matrix. It expands, gates, then compresses — modern Llama-style models use a SwiGLU shape that combines `gate_proj` and `up_proj`.

![What the MLP does inside: expand, gate, multiply, compress](/diagrams/transformer-mlp-internals.svg)

The input goes to two projections in parallel. `up_proj` expands it into a much wider space (more room to compute); `gate_proj` decides which of those expanded features actually matter. The two are multiplied together, and `down_proj` compresses the result back down to the original size. That wide middle is where there's enough capacity to store and combine knowledge — which is exactly why it matters for fine-tuning.

## Why this decides your LoRA config

Now the practical payoff. Here are the two configs people actually argue about:

```python
# Config A — attention only (the common default)
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"]

# Config B — attention + MLP (knowledge-heavy domains)
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                  "gate_proj", "up_proj", "down_proj"]
```

A classic interview question is: *"Config A is attention only, Config B is attention plus MLP — which learns more medical knowledge?"* The answer is **Config B**, and now you can say *why*, not just guess:

> Attention layers mainly change how the model routes and relates tokens. The MLP layers are heavily involved in encoding and transforming knowledge. In a specialized domain the goal isn't only better attention patterns — it's learning new concepts, terminology, and relationships. Adapting `gate_proj`, `up_proj`, `down_proj` gives the model the capacity to store and use that domain knowledge.

| | What you adapt | Good for |
|--|----------------|----------|
| **Attention only** | routing / how tokens relate | style, format, light behavior shifts |
| **Attention + MLP** | routing **and** knowledge storage | new domain facts, terminology, deep adaptation |

So the question *"is Warfarin interacts with Aspirin more about attention or the MLP?"* has a clear answer: it's a new **fact** to store, not just a routing pattern — that's an **MLP** job. If your adapter only touches attention, the model has no room to encode it.

## How this played out in practice

When I fine-tuned **Gemma 3 4B** on a multi-turn conversational dataset for a tool-using assistant, this was a deliberate config decision, not a copy-paste. The assistant needed to learn domain behavior and vocabulary, not just talk more smoothly — so the LoRA adapters targeted the MLP projections alongside attention. The "attention-only" default would have produced exactly the fluent-but-shallow result I described at the top. I'll cover that full setup in the [LoRA & QLoRA post](/blog/lora-qlora-fine-tuning-one-gpu) and the [agent case study](/blog/fine-tuning-tool-calling-agent-sft-qlora-gemma) later in this series.

This sits one level down from [the two axes of fine-tuning](/blog/two-axes-of-fine-tuning): *which layers* your adapter touches is a detail inside Axis B (how you update the weights). Get the mental model of the block right, and the config stops being guesswork.

## The takeaway

- **A transformer block has two machines.** Attention (`q/k/v/o`) routes and retrieves; the MLP (`gate/up/down`) computes and stores knowledge.
- **Attention retrieves, the MLP remembers.** New domain *facts* live in the MLP, not in attention.
- **Target the MLP for knowledge-heavy domains.** Attention-only LoRA changes style; attention + MLP gives the model capacity to learn new concepts.
- **Config is a decision, not a default.** Match `target_modules` to whether you're shifting *behavior* or teaching *knowledge*.

Next in the series: [SFT — what the model is actually predicting](/blog/sft-what-the-model-predicts) — why supervised fine-tuning is the same objective as pretraining, and the masking detail that quietly decides whether it works.

---

### The Two Axes of Fine-Tuning: A Mental Model That Stops the Confusion

URL: https://aiwithvd.com/blog/two-axes-of-fine-tuning
Published: 2026-06-11
Tags: Fine-Tuning, LoRA, QLoRA, SFT, RLHF, GRPO, LLM Training, Mental Model

# The Two Axes of Fine-Tuning: A Mental Model That Stops the Confusion

Someone asked me last month: "LoRA or GRPO — which is better for our agent?"

It's a trick question, and the fact that it *sounds* reasonable is exactly the problem. LoRA and GRPO don't compete. Asking which is better is like asking "is a screwdriver better than a Phillips head pattern?" — one is a tool, the other is a thing the tool produces.

Every fine-tuning term you've heard — SFT, LoRA, QLoRA, DPO, RLHF, PPO, GRPO, RLVR — fits on **two independent axes**. Once you see the axes, the whole space collapses into something you can reason about in a meeting without hand-waving.

## The one idea holding all of it together

Here is the sentence I come back to every time:

> **Every fine-tuning method writes into the *same* transformer weights. They only differ in the *learning signal* they use.**

Same student, same brain, different teacher. SFT says "copy this answer." DPO says "answer A is better than B." PPO/RLHF says "the reward for that answer was high." GRPO and RLVR say "the verifier says you got it right."

The weights being updated are identical. What changes is the instruction you give the optimizer about *which direction is good*. Hold that, and the two axes fall out naturally.

![The two independent axes of fine-tuning](/diagrams/finetuning-two-axes.svg)

## Axis B — how you update the weights

This axis is about **parameter efficiency**: when the gradient arrives, which weights actually move?

- **Full fine-tune** — every weight is trainable. Maximum flexibility, maximum cost. For a 7B model with the Adam optimizer you need roughly 16 bytes per parameter (weights + gradients + optimizer states), so ~112GB of memory before you've even loaded a batch. You reach for this when you have a lot of data and a deep domain shift.
- **LoRA** — freeze the big weight matrix, and learn a thin *low-rank adapter* beside it. Only the adapter (~1% of parameters) gets gradients and optimizer states. This is the sensible default for most work.
- **QLoRA** — LoRA on top of a base that's been quantized to 4 bits. The frozen base shrinks ~4×, so a large model fits on a single GPU. This is what you pick when you're memory-bound.

Notice these are three points on *one* dial. They answer "how expensive is the update," not "what is the model learning."

## Axis A — what signal you train on

This axis is the **learning signal** — the teacher.

- **SFT (supervised fine-tuning)** — imitate labelled answers. The model copies curated (prompt, response) examples. This is where almost every project starts.
- **DPO (direct preference optimization)** — learn from preference pairs (this answer is better than that one), offline, with no separate reward model.
- **RLHF / PPO** — train a reward model on human preferences, then optimize the policy against it with reinforcement learning. Powerful, but it keeps four models in memory at once.
- **GRPO / RLVR** — reinforcement learning with the expensive parts removed: GRPO drops the critic and scores answers against the average of a group; RLVR replaces the learned reward model with an automatic *verifier* (does the code pass? is the math right?).

These are points on a *different* dial. They answer "what is the model learning," not "how expensive is the update."

## Why this matters: you pick one from each

The two axes are **independent**. You don't choose "LoRA or GRPO." You choose *one option from each axis* and combine them:

- SFT **+** QLoRA → a cheap instruction tune
- GRPO **+** LoRA → cheap reinforcement learning
- A typical agent recipe: **SFT → DPO → GRPO**, each stage usually running with LoRA or QLoRA underneath

So the honest answer to "LoRA or GRPO?" is: *"Those aren't alternatives. GRPO decides the objective; LoRA decides parameter efficiency. I'd typically run GRPO with LoRA."* That single reframing is the difference between sounding confused and sounding like you've actually shipped this.

## The universal training loop

Here's the part that makes the mental model click. Every method on both axes runs the **same loop**. Only one box changes.

![The universal training loop every fine-tuning method shares](/diagrams/finetuning-universal-loop.svg)

Batch the data, apply the chat template and tokenize, run a forward pass to get logits, **compute the learning signal**, backpropagate, let the optimizer step, repeat. Steps 1–3 and 5–7 are identical whether you're doing SFT or GRPO.

The *only* thing that differs is box 4 — the loss:

| Method | What box 4 computes | In plain words |
|--------|---------------------|----------------|
| SFT | cross-entropy vs the one correct answer | "copy this" |
| DPO | push chosen answer up, rejected down | "A beats B" |
| PPO / RLHF | maximise reward, critic estimates advantage | "reward was high" |
| GRPO / RLVR | score a group of answers, no critic | "verifier says correct" |

Axis B changes *which weights* box 5 sends gradients into (all of them, or just the LoRA adapters). Axis A changes *what box 4 says is good*. That's the entire space.

## How I use this before a project

When a new fine-tuning task lands, I make two decisions, in order:

1. **Axis A first — what's my teacher?** Do I have labelled demonstrations (SFT)? Preference pairs (DPO)? A way to *automatically verify* correctness like tests or a SQL result (RLVR with GRPO)? The signal I can actually get usually decides this for me.
2. **Axis B second — what's my budget?** One GPU and a big model → QLoRA. Room to spare and a default mindset → LoRA. Lots of data and a deep domain shift with real hardware → full fine-tune.

Two questions, and the method names stop being a fog. Everything else — response masking, rank, KL leashes, verifiers — is detail that lives *inside* one of these two choices.

In the rest of this series I go one level deeper into each piece, always tied back to real work: a [tool-calling agent fine-tuned with SFT + QLoRA on Gemma 3 4B](/blog/fine-tuning-tool-calling-agent-sft-qlora-gemma), and the reward design behind a multi-tool agent. If you've ever had to choose between prompting, RAG, and fine-tuning in the first place, that decision sits one level above this one — I covered it in [the decision framework post](/blog/prompt-rag-finetune-decision-framework).

## The takeaway

- **Two axes, not one list.** How you update the weights (full / LoRA / QLoRA) is independent of what signal you train on (SFT / DPO / RLHF / GRPO).
- **Same brain, different teacher.** Every method runs the identical training loop; only the loss in box 4 changes.
- **Pick one from each.** "LoRA + GRPO" is a normal recipe, not a contradiction.
- **Decide the signal before the budget.** What you can teach the model is usually more constrained than how cheaply you can teach it.

Next in the series: [Inside a Transformer Block](/blog/inside-a-transformer-block) — why *where* knowledge lives in the model decides *which* layers your adapter should touch.

---

### Using Mem0 for User Preference Memory and Context Switching in AI Agents

URL: https://aiwithvd.com/blog/mem0-user-preference-memory-context-switching
Published: 2026-06-09
Tags: Mem0, Agent Memory, User Preferences, Context Switching, LangGraph, Personalization, AI Agents

# Using Mem0 for User Preference Memory and Context Switching in AI Agents

Every time a user has to re-explain their preferences, your agent feels like it never learned anything.

"I told you last time I want bullet points, not paragraphs." "I said I'm a senior engineer — please stop explaining what a function is." "When I'm analysing data, I want tables. When I'm drafting a document, I want prose."

These are not edge cases. They are the default experience of working with a stateless AI agent. Each new session resets every preference the user has ever expressed. The agent has no memory of who this person is, how they work, or what they have already told it.

[Mem0](https://mem0.ai/) solves this at the infrastructure level — a memory layer that persists across sessions, searches semantically, and can be queried with metadata filters. But the interesting engineering problem is not *that* you can store preferences. It is *what* to store, *how* to tag it, and *when* to retrieve it — especially when a user switches between completely different types of tasks in the same conversation.

## What to Store as User Preferences

Not everything a user says should be stored as a preference. You want to store information that will *change how you respond in the future* — not just what was discussed.

Five categories are useful in practice:

**Communication style.** Does this user want concise answers or detailed explanations? Technical language or plain English? Do they push back when you hedge with "it depends"? These preferences cut across every topic — they shape every response regardless of context.

**Output format.** Bullet points or prose? Tables for comparisons? Code blocks for examples? Some users scan; some read. Some are in a situation where they'll paste your output into a document; others are skimming on a phone. Format preferences are often the first thing a user corrects and the easiest to store.

**Expertise level.** A senior practitioner in one domain and a complete beginner in another. Getting this wrong is costly — over-explain to an expert and they stop trusting you; under-explain to a beginner and they get lost. Store domain-specific expertise levels, not a single global "expertise" flag.

**Topic-specific context.** "When we discuss contracts, I always care about liability clauses first." "For this client specifically, budget is the constraint, not timeline." These are contextual preferences tied to a specific domain or task — they should be stored and retrieved when that context is active, not on every query.

**Decision history.** "I decided last quarter not to use approach X because of Y." Past decisions and their rationale are a form of preference: they tell the agent what paths have already been explored and rejected. Surfacing them prevents the agent from re-suggesting something the user already ruled out.

## Storing Preferences with Mem0

Mem0's storage API is intentionally simple. The interesting design decision is the metadata schema — specifically the `context_tag` field, which controls which preferences get retrieved for which situations.

### Explicit extraction

When a user directly states a preference, extract and store it immediately:

```python
from mem0 import MemoryClient

mem0 = MemoryClient(host="http://mem0-api:8000", api_key=os.getenv("MEM0_JWT_SECRET"))

async def store_preference(
    text: str,
    user_id: str,
    preference_type: str,
    context_tag: str = "global",
    confidence: float = 1.0,
) -> None:
    mem0.add(
        data=text,
        user_id=user_id,
        metadata={
            "preference_type": preference_type,
            "context_tag": context_tag,
            "confidence": confidence,
            "source": "explicit",
        },
    )
```

Example calls based on what a user says:

```python
# "Always give me bullet points"
await store_preference(
    text="User prefers bullet points over prose in all responses",
    user_id=user_id,
    preference_type="output_format",
    context_tag="global",
)

# "When I'm doing data analysis, I want tables for comparisons"
await store_preference(
    text="When context is data analysis, use tables for comparisons and numeric data",
    user_id=user_id,
    preference_type="output_format",
    context_tag="analysis",
)

# "I'm a senior Go engineer but new to ML"
await store_preference(
    text="Expert in Go and systems programming; beginner in ML and data science",
    user_id=user_id,
    preference_type="expertise_level",
    context_tag="global",
)
```

The `context_tag` distinguishes global preferences (apply everywhere) from context-specific ones (only apply in a particular mode). A `"global"` tag is retrieved in every context; a `"analysis"` tag is retrieved only when the agent detects the user is doing data analysis.

### Implicit extraction

Not all preferences are stated directly. A user who consistently rephrases long answers into shorter ones is expressing a preference. A user who always asks follow-up questions about a specific aspect is signalling what they care about.

Implicit extraction runs as a post-processing step after each response cycle:

```python
IMPLICIT_EXTRACTION_PROMPT = """Analyse this interaction. Did the user express or imply any preferences
about how they want to be communicated with? Consider: format preferences, detail level,
what they asked follow-up questions about, any corrections they made.

If you find a preference, return: {{"found": true, "preference": "...", "type": "...", "context_tag": "..."}}
If none: {{"found": false}}

User message: {user_message}
Agent response: {agent_response}
User follow-up: {followup}"""

async def extract_implicit_preference(interaction: dict) -> dict | None:
    response = await claude.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=200,
        messages=[{"role": "user", "content": IMPLICIT_EXTRACTION_PROMPT.format(**interaction)}],
    )
    result = json.loads(response.content[0].text)
    return result if result.get("found") else None
```

Use `claude-haiku` for this — it is fast and cheap for structured extraction on short interactions. Implicit preferences get a `"confidence": 0.6` versus `1.0` for explicit ones, so you can weight them differently at retrieval time.

## Retrieving Preferences and Injecting into Context

Before any substantive response, retrieve preferences relevant to the current context and inject them into the system prompt.

![User Preference Memory Lifecycle](/diagrams/mem0-preference-lifecycle.svg)

```python
async def retrieve_preferences(
    query: str,
    user_id: str,
    context_tag: str,
    limit: int = 8,
) -> list[dict]:
    """Retrieve both global and context-specific preferences."""
    global_prefs = mem0.search(
        query=query,
        user_id=user_id,
        filters={"context_tag": "global"},
        limit=limit // 2,
    )
    context_prefs = mem0.search(
        query=query,
        user_id=user_id,
        filters={"context_tag": context_tag},
        limit=limit // 2,
    )
    return global_prefs.get("results", []) + context_prefs.get("results", [])
```

Two searches: one for global preferences, one for context-specific. This ensures communication style preferences (global) are always included, while only loading context-relevant preferences for the current task.

### Prompt injection

Structure the system prompt so preferences are clearly separated from task instructions:

```python
def build_personalized_system_prompt(
    base_prompt: str,
    preferences: list[dict],
) -> str:
    if not preferences:
        return base_prompt

    pref_lines = "\n".join(f"- {p['memory']}" for p in preferences)
    preference_block = f"""
## User Preferences (apply to this response)
{pref_lines}

These preferences were learned from previous interactions with this user.
Apply them unless the user's current request explicitly overrides them.
"""
    return base_prompt + preference_block
```

The key instruction — "unless the user's current request explicitly overrides them" — prevents preferences from becoming constraints. A user who generally wants bullet points but asks "write me a paragraph about X" should get a paragraph, not bullets.

## Context Switching

Context switching is the moment a user moves from one type of task to another in the same session. The agent needs to detect this shift and load the appropriate preference set.

Common context transitions:
- Research → document drafting (from information gathering to writing)
- Data analysis → client communication (from technical to non-technical register)
- Planning → execution (from high-level to step-by-step)

### Detecting context

A lightweight LLM call classifies the intent before retrieval:

```python
CONTEXT_TAGS = ["analysis", "drafting", "research", "planning", "review", "onboarding"]

CONTEXT_DETECTION_PROMPT = """Classify this user message into one of these contexts: {tags}.
Return only the tag. If unclear, return the most likely one.

Message: {message}"""

async def detect_context(message: str) -> str:
    response = await claude.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=20,
        messages=[{"role": "user", "content": CONTEXT_DETECTION_PROMPT.format(
            tags=", ".join(CONTEXT_TAGS),
            message=message,
        )}],
    )
    tag = response.content[0].text.strip().lower()
    return tag if tag in CONTEXT_TAGS else "research"
```

### Handling the switch in LangGraph

The agent state carries the active `context_tag`. When it changes, the previous context's preference cache is flushed and new preferences are loaded:

```python
from langgraph.graph import StateGraph, END
from typing import TypedDict

class AgentState(TypedDict):
    query: str
    user_id: str
    context_tag: str
    previous_context_tag: str
    preferences: list[dict]
    response: str

async def detect_and_load_context(state: AgentState) -> AgentState:
    new_tag = await detect_context(state["query"])
    context_changed = new_tag != state.get("context_tag", "")

    if context_changed or not state.get("preferences"):
        preferences = await retrieve_preferences(
            query=state["query"],
            user_id=state["user_id"],
            context_tag=new_tag,
        )
    else:
        preferences = state["preferences"]  # reuse if same context

    return {
        **state,
        "previous_context_tag": state.get("context_tag", ""),
        "context_tag": new_tag,
        "preferences": preferences,
    }

async def generate_response(state: AgentState) -> AgentState:
    system_prompt = build_personalized_system_prompt(BASE_PROMPT, state["preferences"])
    # ... agent execution
    return {**state, "response": result}

async def update_preferences(state: AgentState) -> AgentState:
    implicit = await extract_implicit_preference({
        "user_message": state["query"],
        "agent_response": state["response"],
        "followup": "",
    })
    if implicit:
        await store_preference(
            text=implicit["preference"],
            user_id=state["user_id"],
            preference_type=implicit["type"],
            context_tag=state["context_tag"],
            confidence=0.6,
        )
    return state

# Build the graph
graph = StateGraph(AgentState)
graph.add_node("detect_context", detect_and_load_context)
graph.add_node("generate", generate_response)
graph.add_node("update_prefs", update_preferences)

graph.add_edge("detect_context", "generate")
graph.add_edge("generate", "update_prefs")
graph.add_edge("update_prefs", END)
graph.set_entry_point("detect_context")
```

The `context_changed` check in `detect_and_load_context` is important: reloading preferences on every turn is wasteful if the user is mid-task in the same context. Only reload when the tag changes.

## Advanced Patterns

### Preference conflict resolution

A user might give contradicting instructions across sessions: "always use tables" in one session, "never use tables" in another. Mem0's semantic search returns the most relevant matches — recency is implicitly factored in because newer preferences will often more closely match the current query.

For explicit conflicts on the same `preference_type`, keep only the most recent:

```python
async def store_preference_with_dedup(
    text: str,
    user_id: str,
    preference_type: str,
    context_tag: str,
) -> None:
    # Remove old preferences of the same type + context before storing new one
    existing = mem0.search(
        query=text,
        user_id=user_id,
        filters={
            "preference_type": preference_type,
            "context_tag": context_tag,
        },
        limit=3,
    )
    for old_pref in existing.get("results", []):
        mem0.delete(memory_id=old_pref["id"])

    await store_preference(text, user_id, preference_type, context_tag)
```

Use this for preferences that are genuinely mutually exclusive (output format, expertise level), not for additive ones (topic context additions that are meant to accumulate).

### Preference expiry

Some preferences expire. A user who was "new to the codebase" six months ago is not new anymore. A preference tied to a project that ended is noise.

Use a `created_at` timestamp in metadata and filter by age when retrieving:

```python
from datetime import datetime, timedelta

async def retrieve_preferences_with_ttl(
    query: str,
    user_id: str,
    context_tag: str,
    max_age_days: int = 90,
) -> list[dict]:
    cutoff = (datetime.utcnow() - timedelta(days=max_age_days)).isoformat()
    results = mem0.search(
        query=query,
        user_id=user_id,
        filters={
            "context_tag": context_tag,
            "created_at": {"gte": cutoff},
        },
        limit=8,
    )
    return results.get("results", [])
```

A 90-day TTL for domain context preferences and a 365-day TTL for communication style preferences is a reasonable starting point. Communication style rarely changes; domain expertise does.

## Production Checklist

- **Never store PII or secrets as preferences.** Preferences are loaded into prompts — anything stored there ends up in LLM context. Stick to behavioural patterns ("prefers concise answers") not identifying information.
- **Use `context_tag="global"` sparingly.** If everything is global, context switching has no effect. Global tags are for preferences that genuinely apply across all tasks — communication style, not topic-specific context.
- **Test context switching explicitly.** Write a test that runs several turns in one context, then sends a message in a new context. Verify the preference set changes and the old context's topic-specific prefs are not included.
- **Log which preferences were injected.** When a user's experience changes in a way they did not expect, the first question is "what preferences were active?" Make this observable — log the preference list used on each turn.
- **Combine explicit and implicit extraction.** Relying only on explicit statements misses most of what users actually prefer. Running implicit extraction after every turn adds one cheap Haiku call per response and captures the signal in corrections, follow-ups, and rewrites.

The preference memory pattern described here complements the verification layer from [the first post in this series](/blog/building-verification-layer-llm-outputs). Verification ensures what the agent says is true; preference memory ensures how it says it matches what the user actually needs. Together, they make the gap between a demo-quality agent and a production-quality one visible in the right place — the user's experience, not a benchmark.

---

### Building a 4-Tier Verification Layer for LLM Outputs in Regulated Domains

URL: https://aiwithvd.com/blog/building-verification-layer-llm-outputs
Published: 2026-06-02
Tags: LLM Verification, NLI Cross-Encoder, Hallucination Detection, LangGraph, Production AI, Trust Layer

# Building a 4-Tier Verification Layer for LLM Outputs in Regulated Domains

A client in a regulated industry asked me to build an AI research tool. The catch: every citation in the output had to be real. Not "probably real." Not "looks right." Real — traceable to a specific ruling ID in the knowledge base, verified against the source text, with zero tolerance for fabrication.

In most LLM applications, a hallucinated fact is a bad user experience. In regulated domains — finance, law, healthcare, tax — it is a liability event. A fabricated ruling number in a client-facing document can trigger professional indemnity claims, regulatory action, and loss of practitioner credentials.

I needed a verification layer where the user **never** sees a fabricated citation. Here is how I built it.

## Why Existing Approaches Fall Short

The obvious first attempt is embedding similarity: embed the LLM output and the source chunks, compute cosine similarity, flag anything below a threshold. I tried this. It catches maybe 72% of problems.

The failure mode is subtle. Cosine similarity measures whether two texts are *about the same topic*, not whether one *supports* the other. An LLM can generate a claim that is topically related to a source chunk but factually contradicts it — and similarity scoring will call it a match.

**Example:** The source says "Section 109D applies to loans made *after* 4 December 1997." The LLM outputs "Section 109D applies to all private company loans." Same topic, high similarity score, completely wrong scope.

The second attempt — running an LLM-as-judge on every claim — works better but costs $0.15+ per query and adds 8-10 seconds of latency. For a tool handling hundreds of queries per day, this is unsustainable.

I wrote about evaluation metrics in [Evaluating Generative AI in Production](/blog/evaluating-generative-ai-production). That post covers *measurement* — how to know if your system is accurate. This post is about *enforcement* — how to structurally prevent bad output from reaching the user.

## The 4-Tier Architecture

The insight that unlocked the design: different verification techniques have different cost-accuracy profiles. Cheap methods handle the easy cases; expensive methods handle only the hard cases. Stack them.

![4-Tier Verification Pipeline](/diagrams/verification-pipeline.svg)

### Tier 1: Claim Extraction (Claude)

Before you can verify claims, you need to identify them. Raw LLM output is prose — paragraphs mixing analysis, citations, qualifications, and conclusions. Verification requires atomic claims.

Claude decomposes the output into individually verifiable statements:

```python
EXTRACTION_PROMPT = """Extract every factual claim from the text below.
For each claim, return:
- claim_text: the atomic statement
- claim_type: quantitative | explicit | conditional
- cited_source: any ruling/section ID referenced

Return JSON array. Do not add claims not present in the text."""

async def extract_claims(output_text: str) -> list[dict]:
    response = await claude.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=2000,
        messages=[{"role": "user", "content": f"{EXTRACTION_PROMPT}\n\n{output_text}"}],
    )
    return json.loads(response.content[0].text)
```

This step costs ~$0.03 and takes ~2 seconds. A typical skill output produces 8-15 atomic claims.

### Tier 2: Parallel NLI Cross-Encoder (Local GPU)

This is the core of the pipeline. Instead of embedding similarity, I use a Natural Language Inference (NLI) cross-encoder — a model explicitly trained to classify whether a hypothesis is *entailed by*, *contradicted by*, or *neutral to* a premise.

The model is `cross-encoder/nli-deberta-v3-large`. It runs locally on a GPU. Zero API cost per call.

```python
from sentence_transformers import CrossEncoder

nli_model = CrossEncoder("cross-encoder/nli-deberta-v3-large", device="cuda")

def score_claims(claims: list[dict], chunks: list[str]) -> list[dict]:
    pairs = []
    claim_indices = []
    for i, claim in enumerate(claims):
        for chunk in chunks:
            pairs.append((chunk, claim["claim_text"]))
            claim_indices.append(i)

    scores = nli_model.predict(pairs, batch_size=32)
    # scores shape: (n_pairs, 3) for [contradiction, neutral, entailment]

    results = [{"max_entailment": 0.0} for _ in claims]
    for idx, score in zip(claim_indices, scores):
        entailment = float(score[2])
        if entailment > results[idx]["max_entailment"]:
            results[idx]["max_entailment"] = entailment
            results[idx]["contradiction"] = float(score[0])

    return results
```

Why NLI cross-encoder over embedding similarity:

| Approach | What it measures | Accuracy on entailment |
|----------|-----------------|----------------------|
| Cosine similarity | Topical relatedness | ~72% |
| NLI cross-encoder | Logical entailment/contradiction | ~92% |

The cross-encoder explicitly classifies contradictions. Embedding similarity cannot distinguish "supports" from "contradicts" when the topic is the same. This is the difference between a verification layer that catches 72% of problems and one that catches 92%.

Batch inference on a GPU processes 12 claims against 10 chunks (~120 pairs) in ~0.5 seconds. The entire tier costs $0.00.

### Tier 3: Claude Judge (Borderline Cases Only)

Claims score in three bands:

- **> 0.85 entailment**: Supported. Move on.
- **< 0.30 entailment**: Unsupported. Flag it.
- **0.30 – 0.85**: Borderline. These need human-level reasoning.

Only borderline claims go to the Claude judge. In practice, this is 15-25% of claims — the rest are resolved by the NLI model alone.

```python
JUDGE_PROMPT = """You are a verification judge. For each claim below,
determine if the source evidence supports it.

Respond with: SUPPORTED, PARTIAL, or UNSUPPORTED.
Include one sentence of reasoning.

Claims: {claims}
Evidence: {evidence}"""

async def judge_borderline(borderline_claims: list[dict], evidence: list[str]) -> list[dict]:
    response = await claude.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=1500,
        messages=[{"role": "user", "content": JUDGE_PROMPT.format(
            claims=json.dumps(borderline_claims),
            evidence="\n".join(evidence),
        )}],
    )
    return parse_judge_response(response.content[0].text)
```

Batching all borderline claims into a single LLM call keeps this at ~$0.02 and ~3 seconds.

### Tier 4: Hard-Fail Gate

The final tier is binary. If any claim references a citation ID that does not exist in the knowledge base, or if the verification map contains any "Fabricated" classification, the response is blocked.

This is not a soft warning. It is an HTTP 422 with an incident log entry. The user sees "Verification failed" and the system logs the full state — which claims failed, which sources were checked, what the NLI scores were.

```python
def apply_gate(verification_map: list[dict], kb_ruling_ids: set[str]) -> bool:
    for claim in verification_map:
        if claim.get("cited_source"):
            if claim["cited_source"] not in kb_ruling_ids:
                return False  # fabricated citation ID
        if claim["status"] == "fabricated":
            return False
    return True
```

The hard-fail gate is the structural guarantee. Every other tier can have edge cases and accuracy gaps. This tier is deterministic: if the ruling ID is not in the knowledge base, the response does not ship. Period.

## Integrating with LangGraph

The verification layer is not a utility function called from business logic. It is a **mandatory node** in the LangGraph state machine. There is no code path that bypasses it.

```python
from langgraph.graph import StateGraph, END

graph = StateGraph(AgentState)

graph.add_node("parse", parse_query)
graph.add_node("execute_skills", execute_skills)
graph.add_node("verify", run_verification_pipeline)
graph.add_node("compose", compose_result)
graph.add_node("hard_fail", log_incident_and_block)

graph.add_edge("parse", "execute_skills")
graph.add_edge("execute_skills", "verify")

graph.add_conditional_edges("verify", check_gate, {
    "pass": "compose",
    "fail": "hard_fail",
})

graph.add_edge("compose", END)
graph.add_edge("hard_fail", END)
```

Three properties make this trustworthy:

1. **Verification is a node, not a middleware.** It has its own state, its own error handling, and its own audit trail.
2. **No edge connects `execute_skills` to `compose` directly.** The graph structure makes bypassing verification impossible, not just unlikely.
3. **Hard-fail is a first-class node.** Incident logging happens in a dedicated state with its own persistence, not in a `try/catch` buried in business logic.

The state object carries the complete audit trail — retrieved chunks, LLM prompt, model version, NLI scores, judge reasoning, gate decision. Every query is fully reproducible.

## The Economics

The tiered approach is dramatically cheaper than running full LLM verification on every claim:

| Component | Accuracy | Cost / query | Latency |
|-----------|----------|-------------|---------|
| Claude claim extraction | ~95% | $0.03 | ~2s |
| Local NLI cross-encoder | ~92% NLI | $0.00 | ~0.5s |
| Claude judge (borderline) | ~96% | $0.02 | ~3s |
| **Total** | **~94% e2e** | **$0.05** | **~5.5s** |

Running Claude as judge on every claim (no NLI pre-filter) would cost $0.15+ per query. The NLI tier eliminates 75-85% of claims from the expensive path, saving $0.10 per query at scale.

At 1,000 queries/day: $50/day with the pipeline vs $150/day without. Over a month, the NLI model pays for its GPU instance many times over.

The local NLI model also eliminates an API dependency. If the Claude API has a latency spike, verification still runs — the NLI tier operates independently on local hardware.

## Production Checklist

If you are building verification for a regulated-domain LLM system:

- **Start with the gate, not the model.** The hard-fail on fabricated citation IDs is the cheapest, most reliable check. Implement it first.
- **Use NLI, not similarity.** If your system makes factual claims against source documents, a cross-encoder trained for entailment is the right tool. Embedding similarity is for search, not verification.
- **Batch the expensive tier.** Sending borderline claims to the LLM judge one at a time multiplies cost and latency. Batch them.
- **Make verification structural.** If it is possible to bypass verification via a code path, someone will. Use a state machine where the graph topology enforces the flow.
- **Log everything.** The verification map — claims, scores, sources, gate decision — is your audit trail. In regulated domains, reproducibility is not optional.

The verification layer I described here runs in production on a platform serving regulated professionals. The hard-fail gate has fired exactly twice in the first month — both on edge cases where the LLM referenced a superseded ruling that had been removed from the knowledge base. Both were caught before reaching any user.

Next in this series: [Self-Hosted Mem0 for Persistent Agent Memory](/blog/self-hosted-mem0-persistent-agent-memory) — how to store verified research findings so your agent builds institutional knowledge over time, instead of re-researching the same topics every session.

---

### The Rise of the Autonomous Company: Hermes, OpenClaw, and the Paperclip Revolution

URL: https://aiwithvd.com/blog/the-rise-of-the-autonomous-company-hermes-openclaw-and-the-paperclip-revolution
Published: 2026-05-18
Tags: AI Agents, Autonomous Companies, OpenClaw, Hermes, Paperclip

# The Rise of the Autonomous Company: Hermes, OpenClaw, and the Paperclip Revolution

If you had told me in 2024 that by May 2026 I would be managing a company where the CEO, CTO, and the entire engineering staff were AI agents, I might have checked you for a hallucination. Yet here we are. This morning, my autonomous pipeline completed its daily research, drafted this post, generated its own cover art, and enqueued it for my final review—all while I was drinking my first cup of coffee.

The "Autonomous Company" isn't just a buzzword anymore; it's a production reality enabled by a fierce rivalry between three dominant frameworks: **OpenClaw**, **Hermes Agent**, and the rapidly ascending **Paperclip AI**. Each has taken a radically different path to solving the "agent problem," and choosing between them defines your entire engineering culture.

## The Problem: From "Chatbots" to "Colleagues"

The early days of agents (remember AutoGPT?) were plagued by "infinite loops of nothingness." They could search the web, but they couldn't *learn* from their mistakes, and they certainly couldn't work together in a structured way. As we moved into 2025, the industry realized that universal connectivity was table stakes; the real challenge was **organizational orchestration** and **self-improvement**.

We needed agents that didn't just perform tasks, but filled *roles*. We needed an org chart for silicon.

## The Framework Landscape (May 2026)

The following table breaks down how the "Big Three" stack up in the current ecosystem:

| Feature | OpenClaw | Hermes Agent | Paperclip AI |
| :--- | :--- | :--- | :--- |
| **Primary Strength** | Universal connectivity (44k+ skills) | Self-improving memory | Multi-agent orchestration |
| **Developer** | OpenClaw Foundation | Nous Research | Paperclip AI |
| **Philosophy** | The "Universal Employee" | The "Self-Taught Expert" | The "Operating System" |
| **Core Primitive** | Skills & Tools | Reflective Loops | Heartbeats & Org Charts |
| **Market Share** | 42% (Volume) | 35% (Growth) | 20% (Enterprise) |

### 1. Hermes Agent: The Self-Improver
Hermes (by Nous Research) recently claimed the #1 spot on OpenRouter, processing 224 billion tokens daily. Its "secret sauce" is the **Reflective Phase**. After every task, Hermes analyzes its own execution log, identifies inefficiencies, and autonomously writes new `SKILL.md` files to optimize its future self.

### 2. OpenClaw: The Legacy Giant
OpenClaw remains the most-starred repo in history. While it has faced security challenges (the "ClawHavoc" incidents), its **ClawHub** ecosystem is unparalleled. If you need an agent to talk to a legacy COBOL mainframe via a Telegram bot, OpenClaw has a skill for that.

### 3. Paperclip AI: The Manager
Paperclip is the framework I'm using for this very blog. It doesn't try to be the smartest "individual" agent. Instead, it acts as the **Control Plane**. It defines roles (CEO, CTO, Researcher) and manages the "Heartbeats"—scheduled windows where agents wake up, do work, and hand off to others.

## Implementation: Building a Multi-Agent Pipeline

To give you a sense of how this looks in practice, here is a snippet of a **Paperclip-style orchestration** where a CEO agent delegates a research task to a sub-agent.

```python
from paperclip import Agent, Company, Heartbeat

# Define the Silicon Org Chart
company = Company(name="AgentSys Engineering")

ceo = company.hire_role(
    role="CEO",
    instructions="./agents/ceo/AGENTS.md",
    budget_limit_usd=100.0
)

cto = ceo.delegate_hire(
    role="CTO",
    instructions="./agents/cto/AGENTS.md"
)

# Trigger a synchronous workflow
@ceo.on_heartbeat
def daily_strategy(context):
    research_task = ceo.create_issue(
        title="Daily AI Ecosystem Research",
        assignee=cto,
        priority="high"
    )
    # The CEO now enters 'in_review' status automatically 
    # until the Research task is 'done'
    return research_task.wait_for_completion()
```

The power here isn't in the Python code—it's in the **durable state**. If the server reboots mid-task, Paperclip's control plane ensures the agent resumes exactly where it left off during the next Heartbeat.

## Key Takeaways
1. **Org Charts > Individual Agents**: Orchestration frameworks like Paperclip are winning because they mimic human business structures, making them easier to manage and audit.
2. **Self-Improvement is Table Stakes**: Hermes has proven that agents must autonomously update their own `SKILL.md` files to stay relevant in a fast-moving market.
3. **The SKILL.md Standard**: We've finally reached a "lingua franca." Skills written for OpenClaw are now 90% portable to Hermes or Paperclip, ending the era of vendor lock-in.
4. **Budgeting as a Feature**: Modern frameworks include built-in cost controls. An agent won't just "run away" with your API credits; it hits a hard ceiling and pauses for review.

## Code & Resources
- **Full source code:** [github.com/aiwithvd/autonomous-pipeline](https://github.com/aiwithvd/autonomous-pipeline)
- **Framework Docs:** [paperclip.ing/docs](https://paperclip.ing/docs)

---

**Related:**
- [OpenClaw: A Self-Hosted AI Assistant with Ollama, Telegram & Discord](/blog/openclaw-self-hosted-ai-assistant-guide)
- [Getting Started with LLM Agents in Production](/blog/getting-started-with-llm-agents)
- [Memory Management for AI Agents: Context Window Optimization](/blog/ai-agent-memory-management)

**Questions?** [Email me](mailto:contact@aiwithvd.com) or connect on [LinkedIn](https://www.linkedin.com/in/vishvdeep/).

---

### Building a Self-Hosted Voice AI Assistant with LiveKit Agents

URL: https://aiwithvd.com/blog/building-voice-ai-assistant-livekit
Published: 2026-05-13
Tags: LiveKit, Voice AI, FastAPI, Whisper, Ollama, WebRTC, Next.js, Self-Hosted, Edge-TTS, Real-Time Audio

# Building a Self-Hosted Voice AI Assistant with LiveKit Agents

Voice assistants are everywhere — Siri, Alexa, Google Assistant. But they all share the same fundamental trade-off: your audio leaves your device, gets processed on someone else's servers, and your conversation history lives on infrastructure you don't control.

I wanted something different: a voice AI that runs entirely on my own machine, with real-time conversation, no cloud dependencies, and the freedom to swap any component of the pipeline. So I built [Voice AI Demo](https://github.com/aiwithvd/voiceai) — a fully self-hosted, open-source conversational voice AI assistant powered by LiveKit Agents.

In this post I'll walk through the architecture, the four-stage voice pipeline, how LiveKit orchestrates WebRTC media, and the production patterns that make it reliable.

## The Problem

Building a voice AI assistant locally means solving five hard problems:

1. **Real-time audio transport** — You can't use HTTP for streaming audio. You need WebRTC with low-latency media channels
2. **Voice Activity Detection** — When does the user start speaking? When do they stop? Naive approaches break on background noise
3. **Speech-to-Text** — Transcribing audio locally requires a model that's both accurate and fast enough for real-time use
4. **Language Understanding** — The LLM needs to respond naturally and fast. Cloud APIs add latency and privacy concerns
5. **Text-to-Speech** — Synthesizing natural-sounding audio locally without sounding robotic

LiveKit Agents solves problems 1 and 5 elegantly. The rest is about choosing the right models and wiring them together resiliently.

## Architecture Overview

![Voice AI Demo Architecture](/diagrams/voiceai-architecture.svg)

At a glance:

```
Browser → LiveKit (WebRTC) → Agent Worker → VAD → STT → LLM → TTS
                  ↕                         ↕
            LiveKit Server              Ollama (local)
```

**Core components:**

| Layer | Technology | Role |
|-------|------------|------|
| **Frontend** | Next.js 15, React 19, Agents UI | Voice interface with mic controls, audio visualizer, chat transcript |
| **Agent Server** | Python 3.12, FastAPI, LiveKit Agents SDK | JWT token generation, health check, pipeline orchestration |
| **Media** | LiveKit Server (Docker/Go) | WebRTC SFU, room management, job dispatch |
| **VAD** | Silero VAD | Speech segment and utterance boundary detection |
| **STT** | faster-whisper (large-v3-turbo) | Local speech-to-text transcription |
| **LLM** | Ollama (llama3.2:3b) | Local language model via OpenAI-compatible API |
| **TTS** | Edge-TTS | Microsoft Edge TTS engine for speech synthesis |

## How LiveKit Agents Works

LiveKit Agents is the backbone of this system. It provides:

- **WebRTC SFU** — A Selective Forwarding Unit that routes media between browsers and agent workers
- **PipelineAgent** — A high-level abstraction that processes audio through config stages (VAD → STT → LLM → TTS)
- **Job dispatch** — When a user connects, LiveKit automatically dispatches a job to an available agent worker
- **JWT authentication** — Secure token-based room access with configurable permissions

The data flow is elegant:

1. User opens `http://localhost:3000` in a browser
2. Frontend calls `GET /token` on FastAPI → receives a signed LiveKit JWT
3. Browser connects to LiveKit Server via WebRTC using the JWT
4. LiveKit dispatches a job to the Agent Worker (background thread)
5. Agent processes audio through the voice pipeline
6. Response audio streams back through LiveKit → browser plays it in real-time

## The 4-Stage Voice Pipeline

### Stage 1: VAD (Voice Activity Detection)

Silero VAD detects when speech starts and ends. This is critical for two reasons:

- **Utterance detection** — Know when the user has finished speaking so the pipeline can start processing
- **Noise filtering** — Silence and background noise are discarded before they reach the STT model

Silero is pre-trained, tiny (~1.7MB), and runs efficiently on CPU. It operates on 30ms audio frames and returns a probability score between 0 and 1 for each frame.

### Stage 2: STT (Speech-to-Text)

faster-whisper is a reimplementation of OpenAI's Whisper model using CTranslate2. It's 4x faster than the original while maintaining accuracy.

The pipeline uses `large-v3-turbo` — a distilled model that balances accuracy and speed. On Apple Silicon, Metal GPU acceleration brings inference time well under real-time:

```
Audio duration: 5 seconds
Processing time: ~1.2 seconds (Metal GPU)
Real-time factor: 0.24x
```

The first run downloads the model (~3GB). Subsequent runs load from cache and are instant.

### Stage 3: LLM (Language Model)

Ollama serves Llama 3.2 3B locally via an OpenAI-compatible API. The agent sends transcribed text along with a system prompt, and Ollama streams tokens back.

The model is small enough to run on a MacBook with good performance:

```
Prompt processing: ~30 tokens/s
Token generation: ~25 tokens/s
End-to-end response: 1-3 seconds for typical queries
```

The magic is that you can swap to any OpenAI-compatible provider at runtime. Just change `.env`:

```env
LLM_PROVIDER=ollama  # or openai_compatible
LLM_MODEL=llama3.2:3b  # or llama-3.3-70b-versatile (Groq)
LLM_BASE_URL=http://localhost:11434/v1  # or https://api.groq.com/openai/v1
```

### Stage 4: TTS (Text-to-Speech)

Edge-TTS uses Microsoft's Edge browser TTS engine under the hood. It produces natural-sounding speech with multiple voice options (`en-US-AriaNeural` is the default).

The TTS runs locally, streams audio as it generates, and feeds it back through LiveKit's audio track to the browser. The whole round-trip — from user speaking to hearing the response — completes in under 2 seconds on Apple Silicon.

## Safe Wrappers: Error Resilience

Each pipeline stage has a `Safe` wrapper that catches failures and provides graceful fallbacks:

```python
class SafeSTT:
    async def transcribe(self, audio: AudioFrame) -> str:
        try:
            return await self.stt.transcribe(audio)
        except Exception as e:
            logger.error(f"STT failed: {e}")
            return ""  # Empty transcript — LLM handles gracefully
```

This means a single model failure doesn't crash the entire conversation. If Whisper fails, the user gets a "I didn't catch that" response. If Ollama fails, TTS gets a fallback message. The conversation keeps going.

## Frontend: Agents UI

The Next.js frontend uses LiveKit's [Agents UI](https://github.com/livekit/agents-js) — a shadcn-based React component library:

- **Audio visualizer** — Real-time waveform display of microphone input
- **Chat transcript** — Scrollable conversation history with timestamps
- **Mic controls** — Mute/unmute, push-to-talk, and connection status
- **Dark theme** — Optimized for the voice-first interface

The frontend is deliberately minimal. The complexity lives in the agent pipeline, not the UI.

## Provider Switching

The most powerful feature: every component of the pipeline is swappable at runtime. Here are three example configurations:

**Fully local (default):**
```
STT_PROVIDER=whisper
LLM_PROVIDER=ollama
TTS_PROVIDER=edge-tts
```
Zero API keys required. Everything runs on your machine.

**Cloud-powered (faster responses):**
```env
STT_PROVIDER=openai_compatible
STT_MODEL=whisper-1
STT_BASE_URL=https://api.openai.com/v1
# Add OPENAI_API_KEY to .env
```
OpenAI's Whisper API is faster than local for very long audio. Good trade-off if you have an API key.

**Hybrid (best of both):**
```env
LLM_PROVIDER=openai_compatible
LLM_MODEL=llama-3.3-70b-versatile
LLM_BASE_URL=https://api.groq.com/v1
# Add GROQ_API_KEY to .env
```
Free Groq inference for the LLM, local Whisper for STT, local Edge-TTS for speech. Fast AND free.

## Quick Start

Getting the full stack running takes about 5 minutes:

```bash
# 1. Start LiveKit Server (Docker)
cp .env.example .env
docker compose up -d

# 2. Pull LLM model
ollama pull llama3.2:3b

# 3. Start Agent Service
cd agent
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python main.py

# 4. Start Frontend
cd frontend
npm install
npm run dev
```

Open `http://localhost:3000` and click "Start audio". You're talking to a fully local voice AI.

## Key Takeaways

1. **LiveKit Agents is the right abstraction** — It handles WebRTC complexity, media routing, job dispatch, and pipeline orchestration so you focus on the AI models
2. **Local models are production-viable** — Whisper large-v3-turbo + Llama 3.2 3B + Edge-TTS on Apple Silicon delivers sub-2s latency with zero API costs
3. **Provider swappability is essential** — The ability to switch STT, LLM, or TTS at runtime (not rebuild time) makes the system adaptable to any environment
4. **Safe wrappers prevent cascade failures** — A single model crash should never take down the conversation. Graceful degradation keeps the UX intact
5. **Docker for the hard parts** — LiveKit Server runs in Docker. The agent and frontend run natively. This separation keeps development fast and deployment flexible

## Code & Resources

- **Full source code:** [github.com/aiwithvd/voiceai](https://github.com/aiwithvd/voiceai)
- **Docker setup:** One-command LiveKit server with `docker compose up -d`
- **Testing:** 15 agent tests + 4 frontend tests + `--test-mode` for CI
- **Models:** Default pipeline uses zero cloud models; Ollama requires one `ollama pull`

---

*Want to build production voice AI systems? [Let's connect](mailto:contact@aiwithvd.com). I advise teams on real-time AI architecture, voice pipeline design, and self-hosted deployment strategies.*

---

### Prompt Engineering vs RAG vs Fine-Tuning: The Decision Framework I Actually Use

URL: https://aiwithvd.com/blog/prompt-rag-finetune-decision-framework
Published: 2026-04-25
Tags: Prompt Engineering, RAG, Fine-Tuning, LLM Strategy, ROI Analysis, Decision Framework

# Prompt Engineering vs RAG vs Fine-Tuning: The Decision Framework I Actually Use

The product manager asked for "an AI feature." Vague, exciting, and terrifying all at once.

I've sat in that meeting too many times. The team wants to ship fast, the PM wants magic, and you're staring at three paths: **prompt engineering** (cheap, fast), **RAG** (adds complexity), or **fine-tuning** (expensive, slow).

Choose wrong and you waste $50K and three months. This is the decision framework I've refined across the [Document Extraction Pipeline](/projects/document-extraction), [DeepAgent](/projects/deepagent), and a half-dozen production systems.

## The Decision Matrix

Start with this 2×2 matrix. It narrows your options in 30 seconds.

![Decision Matrix](/diagrams/decision-matrix.svg)

|  | Static Knowledge | Dynamic Knowledge |
|--|------------------|-------------------|
| **Generic Output** | Prompt Engineering | RAG |
| **Specific Output** | Fine-Tuning | RAG + Fine-Tuning |

**Definitions:**
- **Static Knowledge**: Fixed facts (product docs, legal clauses)
- **Dynamic Knowledge**: Frequently updated (stock prices, user data)
- **Generic Output**: Standard tone, common formats
- **Specific Output**: Brand voice, specialized formatting, unique style

## Phase 1: Prompt Engineering (Always Start Here)

**Cost:** $0.001–0.02 per 1K tokens
**Time:** Hours to days
**Break-even:** Immediate

You should always start here. It's the cheapest way to establish a baseline and understand your problem.

### What You Can Achieve with Prompting

```python
# Example: Structured extraction with just prompting
EXTRACTION_PROMPT = """Extract invoice information from the text below.

## Text
{text}

## Instructions
1. Identify the invoice number, date, total amount, and vendor
2. Return ONLY valid JSON in this exact format:
{{
    "invoice_number": "...",
    "date": "YYYY-MM-DD",
    "total_amount": 0.00,
    "vendor": "..."
}}

3. Use null for missing fields
4. Ensure date is ISO format (YYYY-MM-DD)
5. Amount should be a number, not string

## Output"""

response = openai_client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": EXTRACTION_PROMPT.format(text=invoice_text)}],
    response_format={"type": "json_object"}  # Enforce JSON output
)
```

**Advanced prompt engineering techniques:**

1. **Chain-of-thought**: Ask the model to reason step-by-step
2. **Few-shot examples**: Include 2-3 examples in the prompt
3. **Structured outputs**: Use JSON schema or function calling
4. **System prompts**: Set behavior in the system message
5. **Prompt chaining**: Break complex tasks into multiple calls

### When to Move On

You know you've hit the prompt engineering ceiling when:

- [ ] Accuracy plateaus (not improving with better prompts)
- [ ] Latency is too high (prompt is 5K+ tokens)
- [ ] Context window overflow (prompt + context > 100K tokens)
- [ ] Consistency issues (outputs vary significantly)
- [ ] Cost per request exceeds budget

**From the Document Extraction Pipeline:**
- Started with basic prompting: 72% extraction accuracy
- Advanced prompting (few-shot, schemas): 85% accuracy
- Ceiling: Couldn't reach 95% without external knowledge
- **Decision:** Move to RAG at month 2

## Phase 2: RAG (Dynamic Knowledge)

**Cost:** ~20% latency increase + vector DB costs (~$0.10/1M vectors)
**Time:** 1–2 weeks
**Break-even:** When you need proprietary or frequently updated data

RAG (Retrieval-Augmented Generation) adds a knowledge base that the LLM can query.

### When RAG is the Right Choice

✅ **Use RAG when:**
- Knowledge changes frequently (product docs, policies)
- You have proprietary data not in training corpus
- You need source attribution ("according to doc X...")
- You want to reduce hallucinations with grounded context
- You need to control information access (user-specific docs)

❌ **Don't use RAG when:**
- Knowledge is static and fits in context window
- You need specific tone/style (use fine-tuning)
- You're doing code generation (use fine-tuned code models)
- Retrieval adds unacceptable latency (>200ms)

### RAG Architecture

```python
from sentence_transformers import SentenceTransformer
import pinecone

class RAGSystem:
    def __init__(self):
        self.embedding_model = SentenceTransformer('all-MiniLM-L6-v2')
        self.vector_db = pinecone.Index("document-embeddings")
    
    def query(self, question: str, top_k: int = 5) -> str:
        """Answer question using RAG."""
        
        # 1. Embed the question
        query_embedding = self.embedding_model.encode(question)
        
        # 2. Retrieve relevant chunks
        results = self.vector_db.query(
            vector=query_embedding.tolist(),
            top_k=top_k,
            include_metadata=True
        )
        
        # 3. Format context
        contexts = [match.metadata["text"] for match in results.matches]
        context_str = "\n\n---\n\n".join(contexts)
        
        # 4. Generate with context
        prompt = f"""Answer the question using the provided context.
If the answer isn't in the context, say "I don't have that information."

## Context
{context_str}

## Question
{question}

## Answer"""
        
        response = openai_client.chat.completions.create(
            model="gpt-4",
            messages=[{"role": "user", "content": prompt}]
        )
        
        return response.choices[0].message.content
```

### Chunking Strategy

The #1 mistake in RAG: wrong chunk size.

```python
class DocumentChunker:
    """Chunk documents for optimal retrieval."""
    
    def __init__(
        self,
        chunk_size: int = 500,      # Tokens per chunk
        overlap: int = 100           # Overlap between chunks
    ):
        self.chunk_size = chunk_size
        self.overlap = overlap
    
    def chunk_text(self, text: str) -> List[dict]:
        """Create overlapping chunks."""
        
        # Simple word-based chunking (use tiktoken for token-based)
        words = text.split()
        chunks = []
        
        start = 0
        while start < len(words):
            end = min(start + self.chunk_size, len(words))
            chunk_words = words[start:end]
            
            chunks.append({
                "text": " ".join(chunk_words),
                "start_idx": start,
                "end_idx": end,
                "metadata": {
                    "chunk_index": len(chunks),
                    "total_chunks": None  # Set later
                }
            })
            
            start += self.chunk_size - self.overlap
        
        # Update total
        for chunk in chunks:
            chunk["metadata"]["total_chunks"] = len(chunks)
        
        return chunks

# Optimal chunk sizes by use case:
CHUNK_CONFIGS = {
    "general_qa": {"size": 500, "overlap": 100},
    "legal_documents": {"size": 1000, "overlap": 200},
    "code": {"size": 300, "overlap": 50},
    "technical_docs": {"size": 400, "overlap": 80}
}
```

### When RAG Wasn't Enough

**Document Extraction Pipeline case study:**

- RAG improved accuracy to 89%
- But specialized invoice formats needed specific extraction logic
- Prompt + RAG reached 92%—still below 95% SLA
- **Decision:** Fine-tuned schema-specific extractors at month 4

## Phase 3: Fine-Tuning (The Nuclear Option)

**Cost:** $500–5,000 training + ongoing inference
**Time:** 2–4 weeks
**Break-even:** When you need specific tone/format or want to distil a big model

Fine-tuning adapts a base model to your specific task. It's powerful but expensive.

### When Fine-Tuning is the Right Choice

✅ **Use fine-tuning when:**
- RAG + prompting can't achieve required accuracy
- You need specific tone, style, or format consistency
- You want to reduce latency (distil GPT-4 → smaller model)
- You want to reduce cost (fine-tuned 7B model vs GPT-4)
- You have 1,000+ high-quality training examples

❌ **Don't fine-tune when:**
- You have <500 training examples (use few-shot prompting)
- Knowledge is dynamic (use RAG)
- You need general capabilities (use base model)
- Budget is constrained (training + serving costs)

### Fine-Tuning Example

```python
# 1. Prepare training data
# training_data.jsonl
{"messages": [
    {"role": "system", "content": "You extract invoice data from text."},
    {"role": "user", "content": "Invoice #12345 dated 2024-01-15..."},
    {"role": "assistant", "content": '{"invoice_number": "12345", "date": "2024-01-15", "total": 1500.00}'}
]}

# 2. Upload and train (OpenAI example)
import openai

# Upload training file
with open("training_data.jsonl", "rb") as f:
    file = openai.files.create(file=f, purpose="fine-tune")

# Create fine-tuning job
job = openai.fine_tuning.jobs.create(
    training_file=file.id,
    model="gpt-3.5-turbo",
    suffix="invoice-extractor"
)

# 3. Use fine-tuned model
response = openai_client.chat.completions.create(
    model="ft:gpt-3.5-turbo:my-org:invoice-extractor:12345",  # Fine-tuned model
    messages=[{"role": "user", "content": invoice_text}]
)
```

### Cost Comparison: Fine-Tuned vs GPT-4

| Metric | GPT-4 | Fine-Tuned 7B Model |
|--------|-------|---------------------|
| Training cost | $0 | $2,000–5,000 |
| Inference cost/1K tokens | $0.03 | $0.002 |
| Latency | 2–5s | 0.5–1s |
| Accuracy (invoice extraction) | 92% | 96% |
| Break-even volume | N/A | ~200K requests |

**From Document Extraction Pipeline:**
- Fine-tuned on 5,000 labeled invoices
- Training cost: $3,200
- Inference cost reduction: 93% vs GPT-4
- Accuracy improvement: 92% → 96%
- Break-even: Month 3 at current volume

### Distillation: Big → Small

Use GPT-4 to generate training data, then fine-tune a small model:

```python
class DistillationPipeline:
    """Distil GPT-4 into smaller model."""
    
    def generate_training_data(
        self,
        inputs: List[str],
        teacher_model: str = "gpt-4"
    ) -> List[dict]:
        """Generate high-quality training data from teacher model."""
        
        training_examples = []
        
        for input_text in inputs:
            # Get teacher response
            teacher_response = openai_client.chat.completions.create(
                model=teacher_model,
                messages=[{"role": "user", "content": input_text}],
                temperature=0.0  # Deterministic
            )
            
            training_examples.append({
                "messages": [
                    {"role": "user", "content": input_text},
                    {"role": "assistant", "content": teacher_response.choices[0].message.content}
                ]
            })
        
        return training_examples
    
    def fine_tune_student(
        self,
        training_data: List[dict],
        student_base: str = "meta-llama/Llama-2-7b"
    ):
        """Fine-tune student model on teacher-generated data."""
        
        # Use HuggingFace or custom training pipeline
        from transformers import AutoModelForCausalLM, TrainingArguments
        
        model = AutoModelForCausalLM.from_pretrained(student_base)
        
        training_args = TrainingArguments(
            output_dir="./distilled_model",
            num_train_epochs=3,
            per_device_train_batch_size=4,
            learning_rate=2e-5,
            warmup_steps=100
        )
        
        # Train...
        trainer = Trainer(
            model=model,
            args=training_args,
            train_dataset=training_data
        )
        
        trainer.train()
```

## The 10-Question Decision Framework

Use this flowchart to decide:

![Decision Flowchart](/diagrams/decision-flowchart.svg)

**The 10 questions:**

1. **Is your knowledge static or dynamic?**
   - Static → Consider fine-tuning
   - Dynamic → Use RAG

2. **Do you need specific tone/format/style?**
   - Yes → Fine-tuning may be needed
   - No → Start with prompting

3. **How many training examples do you have?**
   - <100 → Prompt engineering only
   - 100–500 → Few-shot prompting
   - 500–1000 → RAG + prompting
   - >1000 → Can consider fine-tuning

4. **What accuracy do you need?**
   - <85% → Prompt engineering
   - 85–92% → RAG + prompting
   - >92% → Fine-tuning or hybrid

5. **Is latency critical?**
   - Yes (<500ms) → Fine-tuned small model
   - No → GPT-4 + RAG

6. **Is cost per request critical?**
   - Yes (<$0.01) → Fine-tuned model
   - No → Use best available model

7. **Do you need source attribution?**
   - Yes → RAG
   - No → Any approach works

8. **How often does knowledge change?**
   - Daily/weekly → RAG
   - Monthly/yearly → Fine-tuning viable
   - Never → Prompt engineering

9. **Do you have engineering resources for infrastructure?**
   - Yes → RAG (vector DB)
   - Limited → Prompt engineering or fine-tune via API

10. **What's your timeline?**
    - Days → Prompt engineering
    - Weeks → RAG
    - Months → Fine-tuning

## Hybrid Approaches (What Actually Works)

The best systems combine all three:

### Pattern 1: Fine-Tuned Base + RAG Context

```python
class HybridSystem:
    """Fine-tuned model with RAG context injection."""
    
    def __init__(self):
        self.fine_tuned_model = "ft:gpt-3.5-turbo:custom:123"
        self.rag = RAGSystem()
    
    def generate(self, query: str) -> str:
        # 1. Retrieve relevant context
        contexts = self.rag.retrieve(query, top_k=3)
        
        # 2. Format prompt with context
        prompt = f"""Answer using your training AND the following context:
        
## Context
{contexts}

## Query
{query}

## Answer"""
        
        # 3. Use fine-tuned model (knows task + format)
        response = openai_client.chat.completions.create(
            model=self.fine_tuned_model,
            messages=[{"role": "user", "content": prompt}]
        )
        
        return response.choices[0].message.content
```

**When to use:**
- Need specific format/style (fine-tuning)
- Knowledge is dynamic (RAG)
- DeepAgent uses this pattern for skill-specific responses

### Pattern 2: Prompt Templates + Vector Retrieval

```python
class TemplateRAGSystem:
    """Prompt templates with dynamic context injection."""
    
    def __init__(self):
        self.templates = {
            "support": "You are a helpful support agent...",
            "sales": "You are a sales consultant...",
            "technical": "You are a technical expert..."
        }
        self.rag = RAGSystem()
    
    def generate(self, query: str, persona: str) -> str:
        # 1. Select template
        system_prompt = self.templates.get(persona, self.templates["support"])
        
        # 2. Retrieve persona-specific context
        contexts = self.rag.retrieve(
            query, 
            top_k=5,
            filter={"category": persona}
        )
        
        # 3. Generate with template + context
        messages = [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": f"Context: {contexts}\n\nQuestion: {query}"}
        ]
        
        return openai_client.chat.completions.create(
            model="gpt-4",
            messages=messages
        )
```

**When to use:**
- Multiple personas/use cases
- Shared knowledge base
- Need consistent tone per persona

### Pattern 3: Skills-Based (DeepAgent Pattern)

```python
class SkillsBasedSystem:
    """DeepAgent-style skill routing."""
    
    def __init__(self):
        self.skills = {
            "web_search": WebSearchSkill(),
            "calculator": CalculatorSkill(),
            "document_qa": DocumentQASkill(rag_system),
            "creative_writing": CreativeWritingSkill(fine_tuned_model)
        }
    
    def route_and_execute(self, query: str) -> str:
        # 1. Classify intent
        intent = self.classify_intent(query)
        
        # 2. Route to appropriate skill
        skill = self.skills.get(intent)
        
        if not skill:
            # Fallback to general prompt
            return self.general_prompt(query)
        
        # 3. Execute with skill-specific approach
        return skill.execute(query)

class DocumentQASkill:
    """Uses RAG for document Q&A."""
    
    def __init__(self, rag_system):
        self.rag = rag_system
    
    def execute(self, query: str) -> str:
        contexts = self.rag.retrieve(query)
        # ... generate with context

class CreativeWritingSkill:
    """Uses fine-tuned model for creative tasks."""
    
    def __init__(self, fine_tuned_model):
        self.model = fine_tuned_model
    
    def execute(self, query: str) -> str:
        # Use fine-tuned model
        return generate_with_model(self.model, query)
```

**When to use:**
- Multiple distinct tasks
- Different approaches optimal for each
- Complex agent systems

## ROI Analysis by Approach

### Cost Per Request

| Approach | Input Tokens | Output Tokens | Cost/Request | Monthly (10K req) |
|----------|--------------|---------------|--------------|-------------------|
| Basic Prompt | 500 | 300 | $0.033 | $330 |
| Advanced Prompt | 1500 | 300 | $0.063 | $630 |
| RAG (avg) | 2000 | 300 | $0.078 | $780 |
| Fine-tuned (7B) | 500 | 300 | $0.003 | $30 |
| Hybrid (FT+RAG) | 2000 | 300 | $0.063 | $630 |

### Accuracy vs Cost Trade-off

![Cost vs Accuracy Trade-off](/diagrams/cost-latency-tradeoff.svg)

**Sweet spots:**
- **Budget constrained:** Fine-tuned 7B model (high accuracy, low cost)
- **Accuracy critical:** GPT-4 + RAG (best results, higher cost)
- **Speed critical:** Fine-tuned small model (<1s latency)

## Real Case Studies

### Case 1: Document Extraction Pipeline

**Evolution:**
1. **Month 1:** Basic prompting → 72% accuracy
2. **Month 2:** Advanced prompting (few-shot, schemas) → 85% accuracy
3. **Month 3:** RAG (retrieve similar extractions) → 89% accuracy
4. **Month 4:** Fine-tuned schema extractors → 96% accuracy

**Final architecture:**
- Fine-tuned model for extraction logic
- RAG for similar-document context
- Total cost: $0.005/request (vs $0.03 for GPT-4)

### Case 2: DeepAgent

**Evolution:**
1. **Month 1:** System prompts only → Functional but generic
2. **Month 2:** RAG for skill documentation → Better context
3. **Month 3:** Fine-tuned reasoning model → Better reasoning chains

**Final architecture:**
- Skills-based routing
- RAG for dynamic knowledge
- Fine-tuned models for specific skills
- SSE streaming for real-time responses

### Case 3: Customer Support Chatbot (Consulting Project)

**Decision:**
- Started with RAG (product docs change frequently)
- Added fine-tuning for tone consistency
- Kept GPT-4 for complex escalations

**Results:**
- 87% resolution rate without human
- $0.08/request average cost
- 3-month payback period

## The "Don't" List

- ❌ **Don't** fine-tune for static knowledge (use RAG or prompt)
- ❌ **Don't** use RAG for code generation (use fine-tuned code models)
- ❌ **Don't** start with fine-tuning (expensive experiment)
- ❌ **Don't** over-engineer early (start simple, measure, iterate)
- ❌ **Don't** ignore latency (RAG adds 100-200ms)
- ❌ **Don't** skip evaluation (know when to upgrade)

## Production Decision Checklist

Before choosing your approach:

- [ ] Measured baseline with prompt engineering
- [ ] Defined accuracy requirements (target number)
- [ ] Calculated cost budget per request
- [ ] Assessed latency requirements
- [ ] Estimated training data availability
- [ ] Evaluated infrastructure resources
- [ ] Considered knowledge update frequency
- [ ] Planned evaluation framework
- [ ] Documented rollback strategy

## Summary: The Framework

| Phase | Approach | Time | Cost | When to Move On |
|-------|----------|------|------|-----------------|
| 1 | Prompt Engineering | Days | Low | Accuracy plateau, high latency |
| 2 | RAG | 1–2 weeks | Medium | Still need higher accuracy or style control |
| 3 | Fine-Tuning | 2–4 weeks | High | Maximize accuracy, minimize latency/cost |

**My rule:** Start with prompting, add RAG at month 2 if needed, fine-tune at month 6 if still necessary.

## Questions to Ask Yourself

1. **What's my accuracy target?** (Quantify it)
2. **What's my cost per request budget?** (Cents matter at scale)
3. **How often does my knowledge change?** (Daily = RAG, Yearly = Fine-tuning)
4. **Do I have 1000+ training examples?** (Required for fine-tuning)
5. **Is latency critical?** (<500ms = Fine-tuned small model)
6. **Do I need source attribution?** (Yes = RAG)

**Still unsure?** [Email me](mailto:contact@aiwithvd.com) your specific scenario—I'll help you decide.

---

**Related:**
- [Production LLM System Architecture](/blog/production-llm-system-architecture)
- [Resilient LLM API Patterns](/blog/resilient-llm-api-patterns)
- [Memory Management for AI Agents](/blog/ai-agent-memory-management)
- [Evaluating Generative AI](/blog/evaluating-generative-ai-production)
- [Document Extraction Pipeline](/projects/document-extraction)
- [DeepAgent Project](/projects/deepagent)

## Related Resources

If you're building production LLM systems, these resources complement the decision framework above and cover the implementation details that sit beneath each technique.

**Prompt Engineering**
- [OpenAI Prompt Engineering Guide](https://platform.openai.com/docs/guides/prompt-engineering) — Structured best practices from the provider that defined the field.
- [Anthropic's Claude Prompt Library](https://docs.anthropic.com/en/prompt-library) — Curated patterns for reasoning, extraction, and classification tasks.

**Retrieval-Augmented Generation**
- [LangChain RAG Tutorial](https://python.langchain.com/docs/tutorials/rag/) — End-to-end walkthrough with vector stores, embedding models, and chunking strategies.
- [LlamaIndex Documentation](https://docs.llamaindex.ai/) — Advanced RAG patterns including recursive retrieval and agentic query engines.

**Fine-Tuning**
- [OpenAI Fine-Tuning Guide](https://platform.openai.com/docs/guides/fine-tuning) — Data preparation, hyperparameter selection, and evaluation protocols.
- [Hugging Face TRL Library](https://huggingface.co/docs/trl/index) — Open-source fine-tuning with PPO, DPO, and ORPO methods.

**Evaluation & Cost Tracking**
- [Langfuse](https://langfuse.com/) — Open-source LLM observability with tracing, scoring, and cost attribution.
- [Weights & Biases](https://wandb.ai/) — Experiment tracking for fine-tuning runs and prompt versioning.

**Questions?** [Email me](mailto:contact@aiwithvd.com) or connect on [LinkedIn](https://www.linkedin.com/in/vishvdeep/).

---

### Evaluating Generative AI in Production: Metrics Beyond 'Correct' and 'Incorrect'

URL: https://aiwithvd.com/blog/evaluating-generative-ai-production
Published: 2026-04-24
Tags: LLM Evaluation, LLM-as-Judge, Generative AI Metrics, RAG Evaluation, A/B Testing, Production AI

# Evaluating Generative AI in Production: Metrics Beyond "Correct" and "Incorrect"

I shipped the feature. The demo looked great. Users started complaining.

The LLM was generating responses that *sounded* good but were subtly wrong—wrong dates, fabricated statistics, confidently incorrect answers. And I had no systematic way to catch it. I was manually reviewing 50 random outputs and hoping for the best.

Traditional ML evaluation doesn't work for generative AI. ROUGE and BLEU scores measure text overlap, not truth. Accuracy requires ground truth, which doesn't exist for creative tasks. In this post, I'll show you the four-tier evaluation framework I use to ship reliable generative AI.

## The Evaluation Pyramid

Not all evaluation is equal. Start cheap and deterministic, then add expensive judgment layers.

![Evaluation Pyramid](/diagrams/evaluation-pyramid.svg)

**The four tiers:**

1. **Deterministic checks** (cheap, instant, always run)
2. **Heuristic metrics** (ROUGE, BERTScore—limited value)
3. **LLM-as-judge** (expensive, nuanced, sample-based)
4. **Human evaluation** (gold standard, slow, expensive)

In the [Document Extraction Pipeline](/projects/document-extraction), I use all four tiers: JSON schema validation (deterministic), semantic similarity (heuristic), GPT-4 grading (LLM-as-judge), and weekly human audits.

## Tier 1: Deterministic Checks That Actually Matter

Before you call any LLM API, verify the output is structurally valid. These are your first line of defense.

### JSON Schema Validation

For structured outputs (extraction, classification, configuration), enforce schemas with Pydantic:

```python
from pydantic import BaseModel, Field, validator
from typing import List, Optional
import json

class InvoiceExtraction(BaseModel):
    """Schema for invoice data extraction."""
    invoice_number: str = Field(..., min_length=1, max_length=50)
    date: str = Field(..., pattern=r"^\d{4}-\d{2}-\d{2}$")
    total_amount: float = Field(..., gt=0)
    vendor_name: str = Field(..., min_length=1)
    line_items: List[dict] = Field(..., min_items=1)
    
    @validator('date')
    def validate_date_format(cls, v):
        from datetime import datetime
        try:
            datetime.strptime(v, '%Y-%m-%d')
            return v
        except ValueError:
            raise ValueError('Invalid date format')
    
    @validator('total_amount')
    def validate_reasonable_amount(cls, v):
        if v > 1000000:
            raise ValueError('Amount seems unreasonably high')
        return v

def validate_llm_output(raw_output: str) -> tuple[bool, Optional[InvoiceExtraction]]:
    """Validate and parse LLM output."""
    try:
        # Parse JSON
        data = json.loads(raw_output)
        
        # Validate against schema
        validated = InvoiceExtraction(**data)
        
        return True, validated
        
    except json.JSONDecodeError as e:
        print(f"Invalid JSON: {e}")
        return False, None
    except Exception as e:
        print(f"Schema validation failed: {e}")
        return False, None

# Usage in extraction pipeline
def extract_invoice(document_text: str) -> InvoiceExtraction:
    prompt = f"""Extract invoice data from this text as JSON:
    
{document_text}

Return valid JSON matching this schema:
- invoice_number: string
- date: YYYY-MM-DD format
- total_amount: positive number
- vendor_name: string
- line_items: array of objects"""
    
    response = llm_client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}]
    )
    
    raw_output = response.choices[0].message.content
    
    # Clean up markdown code blocks if present
    raw_output = raw_output.replace("```json", "").replace("```", "").strip()
    
    is_valid, parsed = validate_llm_output(raw_output)
    
    if not is_valid:
        # Retry with stronger prompt or escalate
        raise ExtractionValidationError("Failed to extract valid invoice data")
    
    return parsed
```

**Results from production:**
- 8% of LLM outputs failed schema validation initially
- After prompt engineering: 2% failure rate
- Automatic retry logic catches 95% of remaining failures

### Format Compliance Checks

Beyond JSON, check output meets business rules:

```python
class OutputValidator:
    """Deterministic checks for LLM outputs."""
    
    @staticmethod
    def check_length(output: str, min_len: int = 10, max_len: int = 2000) -> bool:
        """Check response is within length bounds."""
        return min_len <= len(output) <= max_len
    
    @staticmethod
    def check_forbidden_words(output: str, forbidden: List[str]) -> bool:
        """Check response doesn't contain prohibited terms."""
        output_lower = output.lower()
        return not any(word.lower() in output_lower for word in forbidden)
    
    @staticmethod
    def check_required_sections(output: str, required: List[str]) -> bool:
        """Check all required sections are present."""
        output_lower = output.lower()
        return all(section.lower() in output_lower for section in required)
    
    @staticmethod
    def check_json_validity(output: str) -> bool:
        """Check if output is valid JSON."""
        try:
            json.loads(output)
            return True
        except json.JSONDecodeError:
            return False
    
    @staticmethod
    def check_language(output: str, allowed_languages: List[str]) -> bool:
        """Check output is in allowed language."""
        # Simplified check - use langdetect library in production
        return True  # Placeholder

# Usage
def validate_support_response(response: str) -> Dict[str, any]:
    """Validate customer support response."""
    checks = {
        "length_ok": OutputValidator.check_length(response, 50, 1000),
        "no_profanity": OutputValidator.check_forbidden_words(
            response, ["stupid", "idiot", "dumb"]
        ),
        "has_greeting": OutputValidator.check_required_sections(
            response, ["Hello", "Hi"]
        ),
        "has_closing": OutputValidator.check_required_sections(
            response, ["Best", "Regards", "Thanks"]
        ),
    }
    
    return {
        "all_passed": all(checks.values()),
        "checks": checks,
        "failed_checks": [k for k, v in checks.items() if not v]
    }
```

### Latency and Performance Checks

```python
import time
from dataclasses import dataclass

@dataclass
class LLMResponse:
    content: str
    latency_ms: float
    tokens_input: int
    tokens_output: int
    model: str

def call_llm_with_validation(prompt: str) -> tuple[bool, Optional[LLMResponse]]:
    """Call LLM with full validation."""
    start = time.time()
    
    response = llm_client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=500
    )
    
    latency_ms = (time.time() - start) * 1000
    
    result = LLMResponse(
        content=response.choices[0].message.content,
        latency_ms=latency_ms,
        tokens_input=response.usage.prompt_tokens,
        tokens_output=response.usage.completion_tokens,
        model="gpt-4"
    )
    
    # Performance checks
    if latency_ms > 10000:  # 10 seconds
        print(f"Warning: High latency {latency_ms}ms")
        return False, None
    
    if result.tokens_output > 450:  # Near limit, probably truncated
        print("Warning: Response may be truncated")
    
    return True, result
```

## Tier 2: Heuristic Metrics (Limited Value)

ROUGE, BLEU, BERTScore measure text similarity. They're okay for summarization, useless for creative tasks.

```python
from rouge import Rouge
from bert_score import score

def calculate_rouge(reference: str, hypothesis: str) -> dict:
    """Calculate ROUGE scores."""
    rouge = Rouge()
    scores = rouge.get_scores(hypothesis, reference)[0]
    return {
        "rouge-1": scores["rouge-1"]["f"],
        "rouge-2": scores["rouge-2"]["f"],
        "rouge-l": scores["rouge-l"]["f"]
    }

def calculate_bertscore(references: List[str], candidates: List[str]):
    """Calculate BERTScore (semantic similarity)."""
    P, R, F1 = score(candidates, references, lang="en")
    return {
        "precision": P.mean().item(),
        "recall": R.mean().item(),
        "f1": F1.mean().item()
    }

# My take: These are trailing indicators. If your ROUGE score drops, 
# something's already broken. Don't rely on them for quality gates.
```

## Tier 3: LLM-as-Judge (The Workhorse)

Use a capable model (GPT-4, Claude 3.5) to evaluate outputs against strict rubrics.

### Building the Judge Prompt

```python
JUDGE_PROMPT = """You are an expert evaluator of AI-generated responses. 
Evaluate the following response based on the criteria below.

## Response to Evaluate
{response}

## Original Query
{query}

## Context (if any)
{context}

## Evaluation Criteria

Score each dimension 1-5:

1. **Accuracy** (1-5): Is the information factually correct?
   - 5: Completely accurate, no errors
   - 4: Mostly accurate, minor errors
   - 3: Some accurate info, some errors
   - 2: Mostly incorrect
   - 1: Completely wrong or fabricated

2. **Relevance** (1-5): Does it directly address the query?
   - 5: Perfectly addresses the query
   - 4: Mostly relevant with minor tangents
   - 3: Partially relevant
   - 2: Barely relevant
   - 1: Completely off-topic

3. **Completeness** (1-5): Does it cover all aspects of the query?
   - 5: Comprehensive coverage
   - 4: Good coverage, minor gaps
   - 3: Partial coverage
   - 2: Major gaps
   - 1: Incomplete

4. **Tone** (1-5): Is the tone appropriate?
   - 5: Perfect tone (professional, helpful)
   - 4: Good tone, minor issues
   - 3: Acceptable tone
   - 2: Tone issues
   - 1: Inappropriate tone

5. **Safety** (1-5): Is the response safe and ethical?
   - 5: Completely safe
   - 4: Minor concerns
   - 3: Some concerns
   - 2: Significant issues
   - 1: Harmful or unsafe

## Instructions
1. Provide scores for each dimension
2. Explain your reasoning for each score
3. Identify specific issues if any
4. Suggest improvements if score < 4

## Output Format
Return JSON only:
{{
    "accuracy": {{"score": 4, "reasoning": "..."}},
    "relevance": {{"score": 5, "reasoning": "..."}},
    "completeness": {{"score": 3, "reasoning": "..."}},
    "tone": {{"score": 5, "reasoning": "..."}},
    "safety": {{"score": 5, "reasoning": "..."}},
    "overall": 4.4,
    "issues": ["issue1", "issue2"],
    "suggestions": ["suggestion1"]
}}"""

class LLMJudge:
    def __init__(self, model: str = "gpt-4"):
        self.model = model
    
    def evaluate(
        self,
        response: str,
        query: str,
        context: str = ""
    ) -> dict:
        """Evaluate response using LLM-as-judge."""
        
        prompt = JUDGE_PROMPT.format(
            response=response,
            query=query,
            context=context
        )
        
        evaluation = llm_client.chat.completions.create(
            model=self.model,
            messages=[{"role": "user", "content": prompt}],
            temperature=0.0,  # Deterministic
            response_format={"type": "json_object"}
        )
        
        result = json.loads(
            evaluation.choices[0].message.content
        )
        
        # Add pass/fail threshold
        result["passed"] = result["overall"] >= 4.0
        
        return result
```

### Reducing Judge Bias

One judge is biased. Use multiple judges and consensus:

```python
class ConsensusJudge:
    def __init__(self, judges: List[str] = None):
        self.judges = judges or ["gpt-4", "claude-3-5-sonnet"]
    
    def evaluate_consensus(
        self,
        response: str,
        query: str
    ) -> dict:
        """Get consensus from multiple judges."""
        
        evaluations = []
        for judge_model in self.judges:
            judge = LLMJudge(model=judge_model)
            eval_result = judge.evaluate(response, query)
            evaluations.append(eval_result)
        
        # Calculate consensus scores
        consensus = {
            "accuracy": self._consensus_score(
                [e["accuracy"]["score"] for e in evaluations]
            ),
            "relevance": self._consensus_score(
                [e["relevance"]["score"] for e in evaluations]
            ),
            "overall": np.mean([e["overall"] for e in evaluations]),
            "agreement": self._calculate_agreement(evaluations),
            "individual_evaluations": evaluations
        }
        
        consensus["passed"] = consensus["overall"] >= 4.0
        
        return consensus
    
    def _consensus_score(self, scores: List[int]) -> dict:
        """Calculate consensus metrics for a dimension."""
        return {
            "mean": np.mean(scores),
            "std": np.std(scores),
            "min": min(scores),
            "max": max(scores),
            "agreement": "high" if np.std(scores) < 0.5 else "low"
        }
```

### Cost Optimization

LLM-as-judge is expensive. Optimize:

```python
class SamplingEvaluator:
    """Evaluate sample of outputs rather than all."""
    
    def __init__(self, sample_rate: float = 0.1):
        self.sample_rate = sample_rate
    
    def should_evaluate(self, request_id: str) -> bool:
        """Deterministic sampling based on request ID."""
        import hashlib
        hash_val = int(hashlib.md5(request_id.encode()).hexdigest(), 16)
        return (hash_val % 1000) / 1000 < self.sample_rate
    
    def evaluate_batch(
        self,
        responses: List[dict],
        judge: LLMJudge
    ) -> dict:
        """Evaluate sampled batch."""
        
        to_evaluate = [
            r for r in responses 
            if self.should_evaluate(r["request_id"])
        ]
        
        results = []
        for resp in to_evaluate:
            result = judge.evaluate(
                resp["output"],
                resp["input"]
            )
            results.append(result)
        
        # Extrapolate to full population
        pass_rate = sum(1 for r in results if r["passed"]) / len(results)
        
        return {
            "sampled_count": len(to_evaluate),
            "total_count": len(responses),
            "pass_rate": pass_rate,
            "confidence_interval": self._calculate_ci(pass_rate, len(to_evaluate)),
            "estimated_passed": int(pass_rate * len(responses))
        }
```

**Cost comparison:**
- Evaluating 100% of outputs: ~$500/month for 10K requests
- Evaluating 10% sample: ~$50/month
- 90% cost savings with statistically valid confidence intervals

## RAG-Specific Evaluation

For retrieval-augmented generation, you need additional metrics:

### Faithfulness

Does the generated answer actually use the retrieved context?

```python
def calculate_faithfulness(answer: str, contexts: List[str]) -> float:
    """Check if answer is grounded in retrieved contexts."""
    
    # Extract claims from answer
    claims = extract_claims(answer)  # Use NER or LLM
    
    grounded_claims = 0
    for claim in claims:
        # Check if claim appears in any context
        for context in contexts:
            if semantic_similarity(claim, context) > 0.8:
                grounded_claims += 1
                break
    
    return grounded_claims / len(claims) if claims else 0

def extract_claims(text: str) -> List[str]:
    """Extract factual claims from text."""
    # Simplified: use LLM to extract claims
    prompt = f"Extract all factual claims from this text as a JSON list:\n\n{text}"
    response = llm_client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"}
    )
    return json.loads(response.choices[0].message.content)["claims"]
```

### Answer Relevance

Is the answer relevant to the question, not just the retrieved context?

```python
def calculate_answer_relevance(
    question: str,
    answer: str,
    embedding_model
) -> float:
    """Calculate semantic similarity between question and answer."""
    
    q_embedding = embedding_model.encode(question)
    a_embedding = embedding_model.encode(answer)
    
    return cosine_similarity(q_embedding, a_embedding)
```

### Context Retrieval Accuracy

Did we retrieve the right chunks?

```python
def evaluate_retrieval(
    query: str,
    retrieved_chunks: List[str],
    ground_truth_chunks: List[str]  # Labeled dataset
) -> dict:
    """Evaluate retrieval quality."""
    
    # Precision: % of retrieved that are relevant
    relevant_retrieved = sum(
        1 for chunk in retrieved_chunks
        if chunk in ground_truth_chunks
    )
    precision = relevant_retrieved / len(retrieved_chunks)
    
    # Recall: % of relevant that were retrieved
    recall = relevant_retrieved / len(ground_truth_chunks)
    
    # F1 score
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
    
    return {
        "precision": precision,
        "recall": recall,
        "f1": f1,
        "retrieved_count": len(retrieved_chunks),
        "relevant_count": len(ground_truth_chunks)
    }
```

## Tier 4: A/B Testing in Production

The ultimate evaluation: which version do users prefer?

### Shadow Mode

Test new model without affecting users:

```python
class ShadowModeEvaluator:
    """Run new model in shadow mode."""
    
    def __init__(self, production_model: str, candidate_model: str):
        self.prod_model = production_model
        self.candidate_model = candidate_model
        self.sample_rate = 0.1
    
    async def generate_with_shadow(
        self,
        prompt: str,
        request_id: str
    ) -> dict:
        """Generate with production model, shadow test candidate."""
        
        # Always call production model (return this to user)
        prod_response = await self._call_model(self.prod_model, prompt)
        
        # Sample for shadow evaluation
        if self._should_shadow(request_id):
            try:
                # Call candidate model (don't block response)
                candidate_response = await asyncio.wait_for(
                    self._call_model(self.candidate_model, prompt),
                    timeout=30.0
                )
                
                # Log both responses for comparison
                await self._log_comparison(
                    request_id,
                    prompt,
                    prod_response,
                    candidate_response
                )
                
            except asyncio.TimeoutError:
                print(f"Shadow call timed out for {request_id}")
        
        return prod_response
```

### Gradual Rollout

```python
class GradualRollout:
    """Gradually shift traffic to new model."""
    
    def __init__(self):
        self.rollout_percentage = 0  # Start at 0%
    
    def select_model(self, user_id: str) -> str:
        """Route user to model based on rollout percentage."""
        
        # Deterministic routing
        hash_val = int(hashlib.md5(user_id.encode()).hexdigest(), 16)
        user_bucket = hash_val % 100
        
        if user_bucket < self.rollout_percentage:
            return "new_model"
        else:
            return "production_model"
    
    def evaluate_rollout(self, days: int = 7) -> dict:
        """Evaluate metrics during rollout."""
        
        metrics = {
            "new_model": self._get_metrics("new_model", days),
            "production_model": self._get_metrics("production_model", days)
        }
        
        # Statistical significance test
        from scipy import stats
        
        t_stat, p_value = stats.ttest_ind(
            metrics["new_model"]["user_satisfaction"],
            metrics["production_model"]["user_satisfaction"]
        )
        
        return {
            "new_model_better": metrics["new_model"]["mean"] > metrics["production_model"]["mean"],
            "statistically_significant": p_value < 0.05,
            "p_value": p_value,
            "recommended_action": "increase_rollout" if (p_value < 0.05 and metrics["new_model"]["mean"] > metrics["production_model"]["mean"]) else "hold"
        }
```

### User Feedback Loops

```python
class FeedbackCollector:
    """Collect explicit user feedback (thumbs up/down)."""
    
    def record_feedback(
        self,
        request_id: str,
        user_id: str,
        feedback: str,  # "positive" or "negative"
        comment: str = ""
    ):
        """Store user feedback."""
        
        feedback_record = {
            "request_id": request_id,
            "user_id": user_id,
            "feedback": feedback,
            "comment": comment,
            "timestamp": datetime.utcnow(),
            "model_version": self._get_model_for_request(request_id)
        }
        
        # Store in database
        self.db.feedback.insert_one(feedback_record)
        
        # Real-time alert if negative feedback spike
        self._check_feedback_spike()
    
    def calculate_satisfaction_rate(
        self,
        model_version: str,
        days: int = 7
    ) -> float:
        """Calculate satisfaction rate for model version."""
        
        feedbacks = self.db.feedback.find({
            "model_version": model_version,
            "timestamp": {"$gte": datetime.utcnow() - timedelta(days=days)}
        })
        
        total = feedbacks.count()
        positive = sum(1 for f in feedbacks if f["feedback"] == "positive")
        
        return positive / total if total > 0 else 0
```

## Building Your Evaluation Framework

### Offline Evaluation Dataset

Before deploying, build a labeled dataset:

```python
# examples/evaluation_dataset.jsonl
{
    "id": "eval_001",
    "input": "What are the system requirements?",
    "context": "System Requirements: 8GB RAM, 4 CPU cores, Python 3.9+",
    "expected_output": "You need 8GB RAM, 4 CPU cores, and Python 3.9 or higher.",
    "evaluation_criteria": {
        "min_length": 20,
        "must_contain": ["8GB", "4 CPU", "Python 3.9"]
    }
}
```

### CI/CD Integration

Block deployment on regression:

```python
# tests/test_llm_regression.py
import pytest

@pytest.mark.evaluation
async def test_no_regression():
    """Ensure new model doesn't regress on evaluation set."""
    
    # Load evaluation dataset
    eval_dataset = load_eval_dataset()
    
    # Run current model
    current_results = []
    for example in eval_dataset:
        result = await current_model.generate(example["input"])
        current_results.append({
            "example_id": example["id"],
            "output": result,
            "passed": evaluate_output(result, example["evaluation_criteria"])
        })
    
    current_pass_rate = sum(1 for r in current_results if r["passed"]) / len(current_results)
    
    # Compare to baseline
    baseline_pass_rate = 0.85  # From previous run
    
    assert current_pass_rate >= baseline_pass_rate - 0.02, \
        f"Regression detected: {current_pass_rate:.2%} vs baseline {baseline_pass_rate:.2%}"
```

### Monitoring Dashboard

Track these metrics:

```python
# Prometheus metrics
EVALUATION_PASS_RATE = Gauge(
    'llm_eval_pass_rate', 
    'Pass rate by evaluation tier',
    ['tier']
)

JUDGE_SCORE = Histogram(
    'llm_judge_score',
    'LLM-as-judge scores',
    ['dimension']
)

RAG_FAITHFULNESS = Gauge(
    'rag_faithfulness',
    'RAG faithfulness score'
)

USER_SATISFACTION = Gauge(
    'user_satisfaction_rate',
    'User thumbs up rate'
)
```

## What Good Looks Like

**Real numbers from production systems:**

| Metric | Target | Good | Excellent |
|--------|--------|------|-----------|
| Deterministic pass rate | > 95% | 97% | 99% |
| LLM-as-judge pass rate | > 80% | 85% | 90% |
| RAG faithfulness | > 70% | 80% | 90% |
| User satisfaction | > 75% | 85% | 90% |
| A/B test significance | < 0.05 | 0.01 | 0.001 |

**Cost per evaluation:**
- Deterministic: $0.0001 (running code)
- LLM-as-judge: $0.005 per evaluation
- Human evaluation: $0.50-$2.00 per sample

## The "Don't" List

- ❌ **Don't** rely solely on ROUGE/BERTScore for generative tasks
- ❌ **Don't** use single-judge evaluation (bias is real)
- ❌ **Don't** evaluate 100% of traffic (wasteful)
- ❌ **Don't** skip deterministic checks (they're free!)
- ❌ **Don't** ignore user feedback (ground truth)

## Production Checklist

Before trusting your evaluation:

- [ ] Deterministic checks for all structured outputs
- [ ] LLM-as-judge rubric defined and tested
- [ ] Sampling strategy implemented (10-20%)
- [ ] RAG metrics if using retrieval (faithfulness, relevance)
- [ ] A/B testing framework ready
- [ ] User feedback collection active
- [ ] CI/CD evaluation gates configured
- [ ] Dashboard with pass rates by tier
- [ ] Alerting on evaluation failure spikes

## Next Steps

Evaluation tells you if your system works. But how do you decide which technique to use in the first place? In the [final post](/blog/prompt-rag-finetune-decision-framework), I'll share the decision framework I use to choose between prompt engineering, RAG, and fine-tuning—complete with ROI analysis.

**Code examples:** [Evaluation framework GitHub](https://github.com/aiwithvd/evaluation-framework)

---

**Related:**
- [Memory Management for AI Agents](/blog/ai-agent-memory-management)
- [Prompt vs RAG vs Fine-Tuning Decision Framework](/blog/prompt-rag-finetune-decision-framework)
- [Document Extraction Pipeline](/projects/document-extraction)

**Questions?** [Email me](mailto:contact@aiwithvd.com) or connect on [LinkedIn](https://www.linkedin.com/in/vishvdeep/).

---

### Memory Management for AI Agents: Context Window Optimization Without Token Bankruptcy

URL: https://aiwithvd.com/blog/ai-agent-memory-management
Published: 2026-04-23
Tags: AI Agent Memory, Context Window, LangGraph, Vectorized Memory, Token Optimization, Multi-Turn Conversations

# Memory Management for AI Agents: Context Window Optimization Without Token Bankruptcy

I got the invoice at the end of March: **$2,847 for LLM API calls.** For a side project. The culprit? I was appending every conversation turn to the context window. After 50 multi-turn sessions, I was sending 50,000 tokens per request.

You cannot just append everything to the prompt forever. Context windows have limits (128k for GPT-4) and costs scale linearly with token count. In this post, I'll show you three memory management strategies that cut my costs by 70% while improving response quality.

## Understanding the Problem

**The math:**
- GPT-4: ~$0.03 per 1K input tokens, $0.06 per 1K output tokens
- 4 characters ≈ 1 token (rough estimate)
- A 10-turn conversation history: ~3,000 tokens
- Cost per request: 3,000 × $0.03/1K = $0.09
- At 1,000 requests/day: **$90/day = $2,700/month**

**The constraint:**
- GPT-4 context window: 128k tokens
- Claude 3: 200k tokens
- But costs don't care about limits—you pay for every token

In the [DeepAgent project](/projects/deepagent), I implemented multi-turn conversation support with LangGraph. Without memory management, sessions with 20+ turns would cost $0.50+ per message.

## Memory Strategy #1: Sliding Window

Keep only the last N messages. Simple, predictable, effective for short sessions.

![Sliding Window Memory](/diagrams/sliding-window-memory.svg)

**Implementation:**

```python
from typing import List, Dict
from dataclasses import dataclass

@dataclass
class Message:
    role: str  # "user" or "assistant"
    content: str
    timestamp: float

class SlidingWindowMemory:
    def __init__(self, max_messages: int = 10):
        self.max_messages = max_messages
        self.messages: List[Message] = []
    
    def add_message(self, role: str, content: str):
        """Add message and maintain window size."""
        import time
        
        message = Message(role=role, content=content, timestamp=time.time())
        self.messages.append(message)
        
        # Remove oldest if over limit
        if len(self.messages) > self.max_messages:
            self.messages = self.messages[-self.max_messages:]
    
    def get_context(self) -> List[Dict[str, str]]:
        """Return messages formatted for LLM API."""
        return [
            {"role": msg.role, "content": msg.content}
            for msg in self.messages
        ]
    
    def estimate_tokens(self) -> int:
        """Rough token estimation."""
        total_chars = sum(len(msg.content) for msg in self.messages)
        return total_chars // 4  # ~4 chars per token

# Usage with LangGraph
from langgraph.graph import StateGraph

class AgentState:
    def __init__(self):
        self.memory = SlidingWindowMemory(max_messages=10)
        self.session_id: str = ""

def chat_node(state: AgentState, user_input: str):
    # Add user message
    state.memory.add_message("user", user_input)
    
    # Get sliding window context
    messages = state.memory.get_context()
    
    # Call LLM with limited context
    response = llm_client.chat.completions.create(
        model="gpt-4",
        messages=messages
    )
    
    # Store assistant response
    state.memory.add_message("assistant", response.choices[0].message.content)
    
    return response
```

**Trade-offs:**
- ✅ Simple to implement
- ✅ Predictable token costs
- ✅ Low latency (no retrieval overhead)
- ❌ Loses older context
- ❌ Can't reference earlier conversation facts
- ❌ Breaks long-running sessions (>20 turns)

**When to use:**
- Short sessions (<10 turns)
- Real-time latency requirements
- Stateless conversation patterns
- Cost-sensitive applications

## Memory Strategy #2: Summarization

Once context reaches a threshold, summarize older messages and replace them with the summary.

![Summarization Memory](/diagrams/summarization-memory.svg)

**Implementation:**

```python
class SummarizationMemory:
    def __init__(
        self,
        max_messages: int = 10,
        token_threshold: int = 3000,
        summary_model: str = "gpt-3.5-turbo"  # Cheaper for summaries
    ):
        self.max_messages = max_messages
        self.token_threshold = token_threshold
        self.summary_model = summary_model
        self.messages: List[Message] = []
        self.summary: str = ""
    
    def add_message(self, role: str, content: str):
        import time
        message = Message(role=role, content=content, timestamp=time.time())
        self.messages.append(message)
        
        # Check if we need to summarize
        if self.estimate_tokens() > self.token_threshold:
            self._summarize_older_messages()
    
    def _summarize_older_messages(self):
        """Summarize oldest half of messages."""
        if len(self.messages) < 4:
            return
        
        # Take oldest 50% for summarization
        to_summarize = self.messages[:len(self.messages)//2]
        to_keep = self.messages[len(self.messages)//2:]
        
        # Create summary prompt
        conversation_text = "\n".join([
            f"{msg.role}: {msg.content}" 
            for msg in to_summarize
        ])
        
        summary_prompt = f"""Summarize this conversation concisely, preserving key facts and context:

{conversation_text}

Summary:"""
        
        # Generate summary with cheaper model
        summary_response = llm_client.chat.completions.create(
            model=self.summary_model,
            messages=[{"role": "user", "content": summary_prompt}],
            max_tokens=200
        )
        
        new_summary = summary_response.choices[0].message.content
        
        # Combine with existing summary
        if self.summary:
            self.summary = f"Previous context: {self.summary}\nRecent: {new_summary}"
        else:
            self.summary = new_summary
        
        # Replace summarized messages
        self.messages = to_keep
    
    def get_context(self) -> List[Dict[str, str]]:
        """Return summary + recent messages."""
        context = []
        
        if self.summary:
            context.append({
                "role": "system",
                "content": f"Previous conversation summary: {self.summary}"
            })
        
        context.extend([
            {"role": msg.role, "content": msg.content}
            for msg in self.messages
        ])
        
        return context
    
    def estimate_tokens(self) -> int:
        summary_tokens = len(self.summary) // 4 if self.summary else 0
        message_tokens = sum(len(msg.content) // 4 for msg in self.messages)
        return summary_tokens + message_tokens
```

**Cost analysis:**
- Summarization call: ~200 tokens in, 150 tokens out
- Cost per summary: $0.006 + $0.009 = **$0.015**
- Savings: Reduces 4,000 tokens to 2,000 tokens (50%)
- Break-even: One summarization saves ~$0.06, pays for itself in 4 turns

**Trade-offs:**
- ✅ Retains key facts from long conversations
- ✅ Reduces token costs significantly
- ✅ Works for medium-length sessions
- ❌ Adds latency (background LLM call)
- ❌ Summary can lose nuance
- ❌ Additional cost for summary generation

**When to use:**
- Medium sessions (10-30 turns)
- Need to preserve key facts
- Can tolerate ~500ms latency hit
- Long-form conversations (interviews, consultations)

## Memory Strategy #3: Vectorized Memory (The Smart Approach)

Store conversation embeddings in a vector database. Retrieve only semantically relevant past messages for each turn.

![Vectorized Memory](/diagrams/vectorized-memory.svg)

**Architecture:**

```python
from sentence_transformers import SentenceTransformer
import numpy as np
from typing import List, Dict, Tuple

class VectorizedMemory:
    def __init__(
        self,
        embedding_model: str = "all-MiniLM-L6-v2",
        vector_db_client=None,  # Pinecone, Milvus, or Chroma
        top_k: int = 5
    ):
        self.embedding_model = SentenceTransformer(embedding_model)
        self.vector_db = vector_db_client
        self.top_k = top_k
        self.recent_messages: List[Message] = []  # Keep last 3 for recency
        self.max_recent = 3
    
    def add_message(self, role: str, content: str, session_id: str):
        """Store message in vector DB with embedding."""
        import time
        
        message = Message(role=role, content=content, timestamp=time.time())
        
        # Generate embedding
        embedding = self.embedding_model.encode(content)
        
        # Store in vector DB
        message_id = f"{session_id}_{int(time.time() * 1000)}"
        self.vector_db.upsert(
            vectors=[{
                "id": message_id,
                "values": embedding.tolist(),
                "metadata": {
                    "role": role,
                    "content": content,
                    "timestamp": message.timestamp,
                    "session_id": session_id
                }
            }]
        )
        
        # Maintain recent messages
        self.recent_messages.append(message)
        if len(self.recent_messages) > self.max_recent:
            self.recent_messages = self.recent_messages[-self.max_recent:]
    
    def get_relevant_context(
        self,
        current_query: str,
        session_id: str
    ) -> List[Dict[str, str]]:
        """Retrieve semantically relevant past messages."""
        
        # Get query embedding
        query_embedding = self.embedding_model.encode(current_query)
        
        # Search vector DB
        results = self.vector_db.query(
            vector=query_embedding.tolist(),
            top_k=self.top_k,
            filter={"session_id": session_id}  # Only this session
        )
        
        # Format results
        relevant_messages = []
        for match in results.matches:
            relevant_messages.append({
                "role": match.metadata["role"],
                "content": match.metadata["content"]
            })
        
        # Combine with recent messages (recency bias)
        recent_context = [
            {"role": msg.role, "content": msg.content}
            for msg in self.recent_messages
        ]
        
        # Deduplicate while preserving order
        seen = set()
        combined = []
        for msg in recent_context + relevant_messages:
            key = (msg["role"], msg["content"])
            if key not in seen:
                seen.add(key)
                combined.append(msg)
        
        return combined
    
    def estimate_tokens(self, context: List[Dict[str, str]]) -> int:
        """Estimate tokens for context."""
        total_chars = sum(len(msg["content"]) for msg in context)
        return total_chars // 4

# Usage with LangGraph
class LangGraphVectorMemory:
    """Vectorized memory for LangGraph state management."""
    
    def __init__(self, session_id: str):
        self.session_id = session_id
        self.memory = VectorizedMemory()
        self.short_term = []  # Last 2 turns for immediate context
    
    def add_turn(self, user_input: str, assistant_response: str):
        """Add conversation turn to memory."""
        self.memory.add_message("user", user_input, self.session_id)
        self.memory.add_message("assistant", assistant_response, self.session_id)
        
        # Update short-term memory
        self.short_term.append({"role": "user", "content": user_input})
        self.short_term.append({"role": "assistant", "content": assistant_response})
        if len(self.short_term) > 4:  # Keep last 2 exchanges
            self.short_term = self.short_term[-4:]
    
    def get_context_for_query(self, query: str) -> List[Dict[str, str]]:
        """Get context combining short-term + relevant historical."""
        # Get semantically relevant historical messages
        relevant = self.memory.get_relevant_context(query, self.session_id)
        
        # Combine with short-term (recency is always included)
        context = self.short_term + relevant
        
        return context
```

**Cost analysis:**
- Vector DB: ~$0.10/1M vectors (Pinecone) or free (Chroma local)
- Embedding model: Free (local) or $0.0001/1K tokens (OpenAI)
- Typical retrieval: Top 5 messages × 200 tokens = 1,000 tokens
- **Savings: 60-70% vs full conversation history**

**Trade-offs:**
- ✅ Most token-efficient for long sessions
- ✅ Retrieves relevant context, ignores noise
- ✅ Scales to 100+ turn sessions
- ❌ Adds complexity (vector DB infrastructure)
- ❌ Retrieval latency (~50-100ms)
- ❌ Requires embedding model

**When to use:**
- Long sessions (20+ turns)
- Need to reference specific past facts
- Domain-specific conversations (support, consultations)
- When you have vector DB infrastructure

## The Hybrid Approach: What Actually Works in Production

After testing all three in production, here's the winning combination:

![Hybrid Memory Architecture](/diagrams/hybrid-memory-architecture.svg)

```python
class HybridMemoryManager:
    """
    Production memory strategy:
    - Always keep last 3 turns (immediate context)
    - Summarize turns 4-10 (medium-term context)
    - Vector retrieval for turns 11+ (long-term context)
    """
    
    def __init__(self, session_id: str):
        self.session_id = session_id
        
        # Short-term: Sliding window (always included)
        self.short_term = SlidingWindowMemory(max_messages=6)  # 3 turns
        
        # Medium-term: Summarization
        self.summary = ""
        self.messages_since_summary = 0
        
        # Long-term: Vectorized storage
        self.vector_memory = VectorizedMemory()
    
    def add_message(self, role: str, content: str):
        import time
        
        # Always add to short-term
        self.short_term.add_message(role, content)
        
        # Add to vector DB for long-term retrieval
        self.vector_memory.add_message(role, content, self.session_id)
        
        self.messages_since_summary += 1
        
        # Trigger summarization every 4 messages (2 turns)
        if self.messages_since_summary >= 4 and len(self.short_term.messages) >= 6:
            self._update_summary()
    
    def _update_summary(self):
        """Summarize older messages."""
        # Get messages to summarize (oldest 2 turns)
        to_summarize = self.short_term.messages[:4]
        
        conversation = "\n".join([
            f"{msg.role}: {msg.content}" for msg in to_summarize
        ])
        
        summary_prompt = f"Summarize key facts from this conversation:\n\n{conversation}"
        
        # Use cheaper model for summary
        response = llm_client.chat.completions.create(
            model="gpt-3.5-turbo",
            messages=[{"role": "user", "content": summary_prompt}],
            max_tokens=150
        )
        
        new_summary = response.choices[0].message.content
        
        if self.summary:
            self.summary = f"{self.summary} | {new_summary}"
        else:
            self.summary = new_summary
        
        self.messages_since_summary = 0
    
    def get_context(self, current_query: str = None) -> List[Dict[str, str]]:
        """Build context from all three memory layers."""
        context = []
        
        # Layer 1: Summary (if exists)
        if self.summary:
            context.append({
                "role": "system",
                "content": f"Previous conversation context: {self.summary}"
            })
        
        # Layer 2: Short-term (recent 3 turns)
        context.extend(self.short_term.get_context())
        
        # Layer 3: Vector retrieval (if query provided)
        if current_query:
            relevant = self.vector_memory.get_relevant_context(
                current_query, self.session_id
            )
            
            # Filter out messages already in short-term
            short_term_contents = {msg.content for msg in self.short_term.messages}
            for msg in relevant:
                if msg["content"] not in short_term_contents:
                    context.append(msg)
        
        return context
    
    def estimate_cost_savings(self) -> Dict[str, float]:
        """Calculate cost vs naive approach."""
        # Assume 50-turn conversation
        naive_tokens = 50 * 300  # 300 tokens per message avg
        
        # Hybrid approach
        summary_tokens = len(self.summary) // 4 if self.summary else 0
        short_term_tokens = self.short_term.estimate_tokens()
        vector_retrieval_tokens = 5 * 200  # Top 5 × 200 tokens
        
        total_tokens = summary_tokens + short_term_tokens + vector_retrieval_tokens
        
        savings = naive_tokens - total_tokens
        cost_savings = (savings / 1000) * 0.03  # $0.03 per 1K tokens
        
        return {
            "naive_tokens": naive_tokens,
            "hybrid_tokens": total_tokens,
            "savings_tokens": savings,
            "cost_savings_per_request": cost_savings,
            "percent_reduction": (savings / naive_tokens) * 100
        }
```

## Real Results from DeepAgent

**Before (no memory management):**
- 20-turn session: ~6,000 tokens per request
- Cost per session: $0.18
- At 500 sessions/day: **$90/day = $2,700/month**

**After (hybrid approach):**
- 20-turn session: ~2,500 tokens per request
- Cost per session: $0.075
- At 500 sessions/day: **$37.50/day = $1,125/month**
- **Savings: $1,575/month (58% reduction)**

**Latency impact:**
- Vector retrieval: +80ms
- Summary generation: +400ms (every 4 messages)
- Net impact: +~100ms average (acceptable for chat)

## Decision Framework: Which Strategy to Use?

| Criteria | Sliding Window | Summarization | Vectorized | Hybrid |
|----------|---------------|---------------|------------|--------|
| Session length | < 10 turns | 10-30 turns | 20+ turns | Any length |
| Latency requirement | < 200ms | < 500ms | < 300ms | < 400ms |
| Cost sensitivity | High | Medium | High | High |
| Infrastructure | Minimal | Minimal | Vector DB | Vector DB |
| Context quality | Low | Medium | High | Highest |

**My recommendation:**
- Start with **Sliding Window** (simple, effective)
- Add **Summarization** at month 2 (when sessions get longer)
- Migrate to **Hybrid** at month 6 (when you have vector DB infrastructure)

## Implementation with LangGraph

LangGraph's built-in memory support makes this easier:

```python
from langgraph.graph import StateGraph
from langgraph.checkpoint.memory import MemorySaver
from langchain_core.messages import HumanMessage, AIMessage

# Configure memory
memory = MemorySaver()

# Build graph with checkpointing
workflow = StateGraph(AgentState)
workflow.add_node("agent", agent_node)
workflow.add_node("tools", tool_node)

# Add memory checkpointing
app = workflow.compile(checkpointer=memory)

# Each thread gets isolated memory
thread_id = "user_123_session_456"
config = {"configurable": {"thread_id": thread_id}}

# Run with automatic memory management
result = app.invoke(
    {"messages": [HumanMessage(content="Hello")]},
    config=config
)

# Memory persists across invocations
result2 = app.invoke(
    {"messages": [HumanMessage(content="What did I ask earlier?")]},
    config=config  # Same thread_id = same memory
)
```

## The "Don't" List

Learn from my mistakes:

- ❌ **Don't** append every message forever (bankruptcy)
- ❌ **Don't** use sliding window for long-form consultations (lose critical context)
- ❌ **Don't** summarize with GPT-4 (use GPT-3.5 or local models)
- ❌ **Don't** store embeddings without metadata filtering (retrieval quality degrades)
- ❌ **Don't** forget to clear memory between sessions (privacy, cost)

## Production Checklist

Before shipping multi-turn agents:

- [ ] Memory strategy selected based on use case
- [ ] Token usage monitored per session
- [ ] Cost alerts set (> $0.10 per session)
- [ ] Session memory cleared on logout/timeout
- [ ] Vector DB indexed with session_id filter
- [ ] Fallback to sliding window if vector DB fails
- [ ] Embedding model cached (don't reload per request)

## Next Steps

Memory management keeps costs down. But how do you know if your agent is actually good? In the [next post](/blog/evaluating-generative-ai-production), I'll cover evaluation frameworks—LLM-as-judge, deterministic checks, and A/B testing for generative AI.

**Code examples:** [DeepAgent GitHub](https://github.com/aiwithvd/langchain_deepagent)

---

**Related:**
- [Resilient LLM API Patterns](/blog/resilient-llm-api-patterns)
- [Evaluating Generative AI in Production](/blog/evaluating-generative-ai-production)
- [DeepAgent Project](/projects/deepagent)

**Questions?** [Email me](mailto:contact@aiwithvd.com) or connect on [LinkedIn](https://www.linkedin.com/in/vishvdeep/).

---

### From Document Processing to LLM Resilience: Patterns That Scale

URL: https://aiwithvd.com/blog/resilient-llm-api-patterns
Published: 2026-04-22
Tags: LLM Resilience, Circuit Breaker, Exponential Backoff, Multi-Provider Fallback, Redis Rate Limiting, Production AI

# From Document Processing to LLM Resilience: Patterns That Scale

The [Document Extraction Pipeline](/projects/document-extraction) taught me a lot about building resilient async systems. FastAPI + Celery + Redis handled OCR and LLM extraction at scale, with retries, job queues, and error handling. But when I started building real-time LLM APIs for the [DeepAgent project](/projects/deepagent), I realized document processing resilience wasn't enough.

The difference? **In document processing, you're batch-oriented and can retry. In LLM APIs, you're real-time and can't afford to fail.**

This post shows how to evolve your async patterns from "it eventually works" to "it never breaks user experience."

## Case Study Foundation: Document Extraction Pipeline

Let's recap the resilience patterns already working in production:

```python
# From Document Extraction Pipeline
def extract_document_task(self, document_id: str, schema: str):
    """Celery task with built-in retry."""
    try:
        document = fetch_document(document_id)
        ocr_result = run_mineru_ocr(document)
        
        # LLM extraction with provider retry
        for attempt in range(3):
            try:
                result = call_llm_provider(ocr_result, schema)
                return result
            except LLMProviderError as e:
                if attempt == 2:
                    raise
                time.sleep(2 ** attempt)  # Exponential backoff
                
    except Exception as exc:
        # Celery handles retry with countdown
        raise self.retry(exc=exc, countdown=60)
```

**What this handles:**
- ✅ Transient OCR failures
- ✅ LLM provider timeouts (3 attempts)
- ✅ Exponential backoff between retries
- ✅ Job persistence via Redis

**What's missing for pure LLM APIs:**
- ❌ Circuit breakers (keeps hitting failing provider)
- ❌ Multi-provider fallback (single point of failure)
- ❌ Real-time rate limiting (429 errors)
- ❌ Latency-aware routing

## Extending to LLM-Specific Resilience

### The Problem: 99.5% Uptime Isn't Enough

Cloud infrastructure promises 99.9% uptime. LLM providers run at 99-99.5%. That's 6-14x worse reliability—and when your entire product depends on LLM calls, that 0.5% downtime kills user trust.

**Real scenario from DeepAgent:**
- 2 PM: OpenAI API throws 503 errors
- 2:01 PM: Users see 30-second timeouts
- 2:05 PM: Queue backs up, Redis memory spikes
- 2:10 PM: Cascade failure, entire system down

Circuit breakers and multi-provider fallbacks prevent this.

## The Resilience Stack

![Circuit Breaker States](/diagrams/circuit-breaker-states.svg)

**Four layers:**
1. **Exponential backoff + jitter** - Handle transient failures
2. **Circuit breaker** - Stop hitting failing providers
3. **Dual rate limiting** - Client-side + provider limits
4. **Multi-provider fallback** - Route around outages

## Implementing Exponential Backoff

Celery has built-in retry, but for real-time LLM calls, you need more control:

```python
import random
import time
from functools import wraps
from typing import Callable, TypeVar

T = TypeVar('T')

def exponential_backoff(
    max_retries: int = 3,
    base_delay: float = 1.0,
    max_delay: float = 60.0,
    exceptions: tuple = (Exception,)
):
    """Decorator for exponential backoff with jitter."""
    def decorator(func: Callable[..., T]) -> Callable[..., T]:
        @wraps(func)
        def wrapper(*args, **kwargs) -> T:
            for attempt in range(max_retries + 1):
                try:
                    return func(*args, **kwargs)
                except exceptions as e:
                    if attempt == max_retries:
                        raise
                    
                    # Exponential backoff + full jitter
                    delay = min(base_delay * (2 ** attempt), max_delay)
                    jitter = random.uniform(0, delay)
                    
                    # Log for monitoring
                    print(f"Attempt {attempt + 1} failed: {e}. Retrying in {jitter:.2f}s...")
                    time.sleep(jitter)
            
            raise RuntimeError("Unreachable")
        return wrapper
    return decorator

# Usage for LLM calls
@exponential_backoff(
    max_retries=3,
    base_delay=1.0,
    exceptions=(LLMProviderError, TimeoutError)
)
def call_openai_with_retry(prompt: str) -> str:
    return openai_client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}]
    )
```

**Handling different error codes:**

```python
def classify_error(error: Exception) -> str:
    """Classify errors for appropriate handling."""
    if isinstance(error, RateLimitError):  # 429
        return "rate_limited"  # Longer backoff
    elif isinstance(error, ServiceUnavailableError):  # 503
        return "transient"  # Standard retry
    elif isinstance(error, AuthenticationError):  # 401
        return "fatal"  # Don't retry
    elif isinstance(error, BadRequestError):  # 400
        return "client_error"  # Don't retry
    else:
        return "unknown"  # Conservative retry
```

## Circuit Breaker Pattern for LLMs

When an LLM provider is down, stop trying. The circuit breaker has three states:

![Circuit Breaker Flow](/diagrams/circuit-breaker-flow.svg)

**Implementation:**

```python
import time
from enum import Enum
from dataclasses import dataclass
from typing import Optional

class CircuitState(Enum):
    CLOSED = "closed"      # Normal operation
    OPEN = "open"          # Failing, reject fast
    HALF_OPEN = "half_open"  # Testing recovery

@dataclass
class CircuitBreaker:
    failure_threshold: int = 5
    recovery_timeout: float = 60.0
    half_open_max_calls: int = 3
    
    def __post_init__(self):
        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.last_failure_time: Optional[float] = None
        self.half_open_calls = 0
    
    def call(self, func, *args, **kwargs):
        """Execute function with circuit breaker protection."""
        
        if self.state == CircuitState.OPEN:
            if time.time() - self.last_failure_time > self.recovery_timeout:
                self.state = CircuitState.HALF_OPEN
                self.half_open_calls = 0
            else:
                raise CircuitBreakerOpenError("Circuit breaker is OPEN")
        
        try:
            result = func(*args, **kwargs)
            self._on_success()
            return result
        except Exception as e:
            self._on_failure()
            raise
    
    def _on_success(self):
        if self.state == CircuitState.HALF_OPEN:
            self.half_open_calls += 1
            if self.half_open_calls >= self.half_open_max_calls:
                self.state = CircuitState.CLOSED
                self.failure_count = 0
        else:
            self.failure_count = 0
    
    def _on_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()
        
        if self.state == CircuitState.HALF_OPEN:
            self.state = CircuitState.OPEN
        elif self.failure_count >= self.failure_threshold:
            self.state = CircuitState.OPEN

# FastAPI middleware integration
circuit_breakers = {
    "openai": CircuitBreaker(failure_threshold=5, recovery_timeout=60),
    "anthropic": CircuitBreaker(failure_threshold=5, recovery_timeout=60),
    "ollama": CircuitBreaker(failure_threshold=3, recovery_timeout=30),
}

def call_with_circuit_breaker(provider: str, func, *args, **kwargs):
    breaker = circuit_breakers.get(provider)
    if not breaker:
        return func(*args, **kwargs)
    return breaker.call(func, *args, **kwargs)
```

## Multi-Provider Fallback Strategy

When OpenAI fails, try Claude. When Claude fails, use local Ollama.

![Multi-Provider Fallback](/diagrams/fallback-routing.svg)

**Implementation:**

```python
from dataclasses import dataclass
from typing import List, Optional, Callable
import time

@dataclass
class ProviderConfig:
    name: str
    client: Callable
    priority: int  # Lower = higher priority
    cost_per_1k: float
    max_latency: float  # Max acceptable latency

class MultiProviderRouter:
    def __init__(self, providers: List[ProviderConfig]):
        self.providers = sorted(providers, key=lambda p: p.priority)
        self.latency_tracker = {}
    
    async def generate_with_fallback(self, prompt: str) -> str:
        """Try providers in priority order with circuit breaker awareness."""
        
        for provider in self.providers:
            # Skip if circuit breaker is open
            breaker = circuit_breakers.get(provider.name)
            if breaker and breaker.state == CircuitState.OPEN:
                continue
            
            # Check latency history
            avg_latency = self.latency_tracker.get(provider.name, 0)
            if avg_latency > provider.max_latency:
                continue
            
            start = time.time()
            try:
                result = await call_with_circuit_breaker(
                    provider.name,
                    provider.client,
                    prompt
                )
                
                # Track latency
                self.latency_tracker[provider.name] = time.time() - start
                
                return {
                    "result": result,
                    "provider": provider.name,
                    "latency": self.latency_tracker[provider.name]
                }
                
            except Exception as e:
                print(f"Provider {provider.name} failed: {e}")
                continue
        
        raise AllProvidersFailedError("All LLM providers unavailable")

# Configuration
providers = [
    ProviderConfig("openai", openai_client, priority=1, cost_per_1k=0.03, max_latency=5.0),
    ProviderConfig("anthropic", anthropic_client, priority=2, cost_per_1k=0.03, max_latency=5.0),
    ProviderConfig("ollama", ollama_client, priority=3, cost_per_1k=0, max_latency=10.0),
]

router = MultiProviderRouter(providers)
```

**Cost implications:**
- Fallback to Claude: Same cost tier (~$0.03/1K tokens)
- Fallback to Ollama: $0 but slower (local GPU)
- Always monitor: Set alerts if >10% traffic hitting fallback

## Token Bucket Rate Limiting

Prevent 429 errors by enforcing client-side limits.

![Token Bucket](/diagrams/token-bucket.svg)

**Redis-based implementation (leveraging existing Redis from Document Pipeline):**

```python
import time
from dataclasses import dataclass

@dataclass
class RateLimitConfig:
    requests_per_minute: int = 60
    burst_size: int = 10  # Allow burst of 10

class TokenBucketRateLimiter:
    def __init__(self, redis_client, config: RateLimitConfig):
        self.redis = redis_client
        self.config = config
    
    async def is_allowed(self, key: str) -> bool:
        """Check if request is allowed under rate limit."""
        now = time.time()
        bucket_key = f"rate_limit:{key}"
        
        # Lua script for atomic operation
        lua_script = """
        local key = KEYS[1]
        local now = tonumber(ARGV[1])
        local window = tonumber(ARGV[2])
        local limit = tonumber(ARGV[3])
        local burst = tonumber(ARGV[4])
        
        -- Get current tokens and last update
        local data = redis.call('HMGET', key, 'tokens', 'last_update')
        local tokens = tonumber(data[1]) or burst
        local last_update = tonumber(data[2]) or now
        
        -- Add tokens based on time passed
        local time_passed = now - last_update
        local tokens_to_add = time_passed * (limit / window)
        tokens = math.min(burst, tokens + tokens_to_add)
        
        -- Check if request allowed
        if tokens >= 1 then
            tokens = tokens - 1
            redis.call('HMSET', key, 'tokens', tokens, 'last_update', now)
            redis.call('EXPIRE', key, window)
            return 1
        else
            redis.call('HSET', key, 'last_update', now)
            redis.call('EXPIRE', key, window)
            return 0
        end
        """
        
        allowed = await self.redis.eval(
            lua_script,
            1,  # num keys
            bucket_key,
            now,
            60,  # window in seconds
            self.config.requests_per_minute,
            self.config.burst_size
        )
        
        return bool(allowed)

# Per-user rate limiting in FastAPI
@app.post("/generate")
async def generate(
    request: Request,
    body: GenerateRequest,
    limiter: TokenBucketRateLimiter = Depends(get_limiter)
):
    user_id = get_user_id_from_token(request)
    
    if not await limiter.is_allowed(f"user:{user_id}"):
        raise HTTPException(
            status_code=429,
            detail="Rate limit exceeded. Try again in 60 seconds."
        )
    
    # Process request...
```

## Production Checklist

Before shipping resilient LLM APIs:

- [ ] Exponential backoff on all LLM calls
- [ ] Circuit breaker per provider (5 errors → 60s cooldown)
- [ ] Multi-provider fallback (minimum 2 providers)
- [ ] Client-side rate limiting (token bucket)
- [ ] Latency-aware routing (track p50/p99 per provider)
- [ ] Dead letter queue for failed requests
- [ ] Monitoring: circuit breaker state, fallback rate, latency
- [ ] Alerts: >5% fallback traffic, circuit breaker open >5min

## Lessons from Document Extraction Pipeline Scaling

**What translated directly:**
- Celery + Redis for job queuing
- Retry with exponential backoff
- Async/await patterns
- Error classification

**What needed adaptation:**
- Circuit breakers (not needed for batch processing)
- Multi-provider (document pipeline used single LLM provider)
- Real-time rate limiting (batch has natural backpressure)
- Streaming responses (documents are processed whole)

**Key insight:** Start with Document Extraction Pipeline's patterns, add circuit breakers at month 3, multi-provider fallback at month 6 when you have SLA commitments.

## Monitoring Dashboard

Track these metrics in production:

```python
# Prometheus metrics
CIRCUIT_BREAKER_STATE = Gauge('circuit_breaker_state', 'Circuit state', ['provider'])
FALLBACK_RATE = Counter('fallback_total', 'Fallback events', ['from_provider', 'to_provider'])
LATENCY_BY_PROVIDER = Histogram('llm_latency_seconds', 'Latency by provider', ['provider'])
RATE_LIMIT_HITS = Counter('rate_limit_hits_total', 'Rate limit hits', ['user_type'])
```

**Alert thresholds:**
- Circuit breaker open > 5 minutes
- Fallback rate > 10% of traffic
- P99 latency > 10 seconds
- Error rate > 1%

## Next Steps

This resilience layer sits on top of the architecture from the [previous post](/blog/production-llm-system-architecture). In the [next post](/blog/ai-agent-memory-management), I'll cover memory management—how to handle context windows in multi-turn conversations without blowing up token costs.

**Code examples:** [GitHub repository](https://github.com/aiwithvd/langchain_deepagent) (DeepAgent implementation)

---

**Related:**
- [Production LLM System Architecture](/blog/production-llm-system-architecture)
- [Memory Management for AI Agents](/blog/ai-agent-memory-management)
- [DeepAgent Project](/projects/deepagent)
- [Document Extraction Pipeline](/projects/document-extraction)

**Questions?** [Email me](mailto:contact@aiwithvd.com) or connect on [LinkedIn](https://www.linkedin.com/in/vishvdeep/).

---

### Production-Grade LLM System Architecture: From Notebook to 10k RPM

URL: https://aiwithvd.com/blog/production-llm-system-architecture
Published: 2026-04-21
Tags: LLM System Design, FastAPI, Celery, Redis, Production AI, RAG, Architecture

# Production-Grade LLM System Architecture: From Notebook to 10k RPM

The prototype worked beautifully in your Jupyter notebook. A single API call to OpenAI, a clever prompt, and impressive results. But now the product team wants to ship it to 10,000 users, and you're staring at a `TimeoutError` at 3 AM while your single-threaded Flask app gasps under the load.

I've been there. The gap between "it works" and "it scales" is where most AI projects die. This post is the architecture guide I wish I had when moving from prototype to production.

## The Decoupled Architecture Pattern

The single biggest mistake I see: keeping the client waiting during LLM calls. LLM APIs take 2-30 seconds. Holding HTTP connections open that long creates a cascading failure nightmare.

Here's the architecture that actually works at scale:

![LLM System Architecture](/diagrams/llm-system-architecture.svg)

**The flow:**
1. **API Gateway** (FastAPI/Nginx) receives the request, validates auth, returns a job ID immediately
2. **Async Queue** (Celery/Kafka/SQS) persists the task durably
3. **Worker Pool** picks up tasks and calls the LLM
4. **LLM Provider Layer** handles external APIs or self-hosted models
5. **Result Storage** (PostgreSQL/Redis) stores completions
6. **Client polls** or receives webhook notifications

This decoupling is non-negotiable for production. I've implemented this pattern in the [Document Extraction Pipeline](/projects/document-extraction), where FastAPI enqueues OCR and LLM extraction jobs to Celery workers, returning job IDs immediately while processing happens asynchronously.

## The Three Biggest Bottlenecks

### Bottleneck #1: I/O Bound (External API Latency)

You're waiting on OpenAI/Anthropic/Google. Their p99 latency can spike to 30+ seconds during peak hours.

**Solutions:**
- **Async/await everywhere**: Never block the event loop
- **Connection pooling**: Reuse HTTP connections with `httpx.AsyncClient`
- **Request timeouts**: 30s default, but make it configurable
- **Parallelization**: Fan out to multiple providers if latency-critical

### Bottleneck #2: GPU Memory (Self-Hosted OOM)

Running Llama 3 70B locally? Welcome to CUDA Out Of Memory errors. Each concurrent request loads model weights into VRAM.

**Solutions:**
- **vLLM inference server**: PagedAttention for 10x throughput
- **Batching**: Accumulate requests, batch-process
- **Model quantization**: GPTQ/AWQ for 4-bit inference
- **Request queuing**: Limit concurrent GPU requests
- **Multi-GPU**: Tensor parallelism across GPUs

### Bottleneck #3: Time To First Token (TTFT)

Users hate staring at a blank screen. For streaming UIs, TTFT > 500ms feels broken.

**Solutions:**
- **Streaming responses**: SSE (Server-Sent Events) for real-time tokens
- **Smaller models for first draft**: Use GPT-3.5 for initial response, GPT-4 for refinement
- **Caching** (see below)
- **Pre-warmed connections**: Keep connections to LLM providers hot

In the [DeepAgent project](/projects/deepagent), I implemented SSE streaming with LangGraph so users see the agent's reasoning in real-time rather than waiting 15 seconds for a complete response.

## Semantic Caching: The 80% Optimization

The fastest LLM call is the one you don't make. Implement semantic caching with Redis:

```python
import redis
import hashlib
import json
from sentence_transformers import SentenceTransformer

class SemanticCache:
    def __init__(self, redis_client: redis.Redis, model: SentenceTransformer):
        self.redis = redis_client
        self.model = model
        self.similarity_threshold = 0.95
    
    def get_cache_key(self, query: str, params: dict) -> str:
        """Create embedding-based cache key."""
        embedding = self.model.encode(query)
        # Quantize to reduce key size
        embedding_bytes = embedding.astype('float16').tobytes()
        param_hash = hashlib.md5(json.dumps(params, sort_keys=True).encode()).hexdigest()
        return f"llm_cache:{embedding_bytes.hex()[:32]}:{param_hash}"
    
    async def get(self, query: str, params: dict) -> str | None:
        """Retrieve from cache if similarity > threshold."""
        key = self.get_cache_key(query, params)
        cached = await self.redis.get(key)
        
        if cached:
            # Verify semantic similarity
            cached_embedding = await self.redis.get(f"{key}:embedding")
            query_embedding = self.model.encode(query)
            
            if self.cosine_similarity(query_embedding, cached_embedding) > self.similarity_threshold:
                return cached.decode()
        
        return None
    
    async def set(self, query: str, params: dict, result: str, ttl: int = 3600):
        """Cache result with embedding for similarity checks.""]
        key = self.get_cache_key(query, params)
        embedding = self.model.encode(query)
        
        pipe = self.redis.pipeline()
        pipe.setex(key, ttl, result)
        pipe.setex(f"{key}:embedding", ttl, embedding.tobytes())
        await pipe.execute()
```

**Results from production:** 40-60% cache hit rate on customer support queries, reducing costs by half.

## Vector Database Integration (RAG)

When your LLM needs access to proprietary, dynamic, or recent information, you need Retrieval-Augmented Generation (RAG).

**Architecture:**

```
User Query → Embedding Model → Vector DB (Pinecone/Milvus) → Top-K Chunks → Prompt Augmentation → LLM
```

**Production considerations:**
- **Chunking strategy**: 500-1000 tokens with 100-token overlap
- **Metadata filtering**: Filter by user, date, document type
- **Hybrid search**: Vector similarity + keyword matching (BM25)
- **Re-ranking**: Cross-encoder for final relevance sorting

In the Document Extraction Pipeline, I use RAG to retrieve similar past extractions when processing new documents, improving accuracy by 15% through contextual learning.

## Code Example: FastAPI + Celery + Redis

Here's a production-ready skeleton:

```python
from fastapi import FastAPI, HTTPException, BackgroundTasks
from celery import Celery
from pydantic import BaseModel
import redis
import json

app = FastAPI()
celery_app = Celery('llm_tasks', broker='redis://localhost:6379/0')
redis_client = redis.Redis(host='localhost', port=6379, db=1)

class LLMRequest(BaseModel):
    prompt: str
    model: str = "gpt-4"
    max_tokens: int = 500
    temperature: float = 0.7

@celery_app.task(bind=True, max_retries=3)
def process_llm_request(self, request_data: dict):
    """Celery task for async LLM processing."""
    try:
        # Check semantic cache first
        cache_key = f"llm:{hash(request_data['prompt'])}"
        cached = redis_client.get(cache_key)
        if cached:
            return json.loads(cached)
        
        # Call LLM provider
        response = call_llm_provider(request_data)
        
        # Cache result (1 hour TTL)
        redis_client.setex(cache_key, 3600, json.dumps(response))
        
        return response
        
    except Exception as exc:
        # Exponential backoff retry
        raise self.retry(exc=exc, countdown=2 ** self.request.retries)

@app.post("/generate")
async def generate(request: LLMRequest):
    """Enqueue LLM request, return job ID immediately."""
    task = process_llm_request.delay(request.model_dump())
    return {"job_id": task.id, "status": "queued"}

@app.get("/result/{job_id}")
async def get_result(job_id: str):
    """Poll for results."""
    task = celery_app.AsyncResult(job_id)
    
    if task.state == 'PENDING':
        return {"job_id": job_id, "status": "processing"}
    elif task.state == 'SUCCESS':
        return {"job_id": job_id, "status": "completed", "result": task.result}
    else:
        raise HTTPException(status_code=500, detail="Task failed")
```

## Real-World Metrics: What Good Looks Like

From the Document Extraction Pipeline at scale:

| Metric | Target | Actual |
|--------|--------|--------|
| P50 Latency | < 5s | 3.2s |
| P99 Latency | < 15s | 8.7s |
| Throughput | 100 req/min | 450 req/min |
| Error Rate | < 1% | 0.3% |
| Cache Hit Rate | 40% | 58% |
| GPU Utilization | 70-85% | 78% |

## The Production Checklist

Before you ship:

- [ ] Async architecture with job queuing
- [ ] Semantic caching implemented
- [ ] Connection pooling configured
- [ ] Retry logic with exponential backoff
- [ ] Timeouts on all external calls
- [ ] Circuit breaker for LLM provider failures
- [ ] Monitoring: latency percentiles, error rates, queue depth
- [ ] Rate limiting per user/IP
- [ ] Graceful degradation (cached responses on failure)

## Next Steps

This architecture gets you to 10k RPM. For the next 100k, you'll need:
- **Horizontal scaling**: Kubernetes HPA based on queue depth
- **Multi-region**: Deploy workers close to LLM providers
- **Smart routing**: Route to lowest-latency provider dynamically

In the [next post](/blog/resilient-llm-api-patterns), I'll cover resilience patterns—circuit breakers, fallbacks, and rate limiting—that keep this architecture stable when things go wrong.

---

**Related:**
- [Document Extraction Pipeline Project](/projects/document-extraction)
- [Resilient LLM API Patterns](/blog/resilient-llm-api-patterns)
- [Memory Management for AI Agents](/blog/ai-agent-memory-management)

**Questions?** [Email me](mailto:contact@aiwithvd.com) or connect on [LinkedIn](https://www.linkedin.com/in/vishvdeep/).

---

### OpenClaw: A Self-Hosted AI Assistant with Ollama, Telegram & Discord

URL: https://aiwithvd.com/blog/openclaw-self-hosted-ai-assistant-guide
Published: 2026-04-13
Tags: OpenClaw, Ollama, AI Agents, Self-Hosted, Telegram, Discord, Skills, CRM, Leads, Customer Support, MCP

# OpenClaw: A Self-Hosted AI Assistant with Ollama, Telegram & Discord

Cloud-hosted AI assistants are convenient, but they come with trade-offs you feel quickly once you start wiring real workflows: your data leaves your machine, your conversation history lives on someone else's disk, your rate limits and pricing change overnight, and you can't reach the assistant from a shell on your home network without yet another integration.

[OpenClaw](https://github.com/openclaw/openclaw) (formerly Moltbot / Clawdbot — yes, the lobster mascot 🦞 stays) takes the opposite approach: it's an open-source, self-hosted personal assistant that runs on your own machine, talks to whichever LLM you point it at (cloud or local), and lets you reach it from Telegram, Discord, WhatsApp, Slack or iMessage — because the brain lives next to your data, not behind someone else's auth layer.

In this post I'll walk through the full stack: what OpenClaw actually is, how to install it, how to wire it to Anthropic / OpenAI / Google or a local Ollama model, how to turn on Telegram and Discord for remote control, and how to extend it with community skills from [skills.sh](https://skills.sh).

## What is OpenClaw

OpenClaw is a single long-running daemon that connects four things:

- **A gateway** — a small WebSocket hub (default `ws://127.0.0.1:18789`) that every other component talks to.
- **Channels** — chat surfaces you already use: Telegram, Discord, WhatsApp, Slack, iMessage. Any message sent to a channel becomes a request to the gateway, and any reply is streamed back to the channel.
- **Skills** — structured capability packages (a folder with a `SKILL.md` instruction file + optional TypeScript/Markdown). Skills extend what the assistant can actually *do* — run shell commands, hit internal APIs, drive a browser, summarise your notes, etc.
- **LLMs** — Anthropic Claude, OpenAI, Google Gemini, or anything OpenAI-API-compatible running locally (Ollama, LM Studio, vLLM…).

The important design choice: every chat surface talks to the *same* gateway, which routes to the *same* LLM with access to the *same* skills and memory. So you can start a task from Telegram, finish it from Discord, and the assistant keeps context in one place.

## Architecture

![OpenClaw architecture — Gateway (WS:18789) routes between channels (Telegram, Discord, WhatsApp, Slack, iMessage), LLM providers (Anthropic, OpenAI, Google, Ollama, LM Studio), skills (bundled, workspace, ClawHub), and persistent memory](/diagrams/openclaw-architecture.svg)

The gateway is the hub. Everything else is a plugin that connects to it. That's what makes it straightforward to add or remove a channel without touching the LLM wiring, or swap Claude for a local Llama model without touching the Discord bot.

## Install

You need a recent Node. Node 24 is the sweet spot, but anything ≥ 22.16 works.

```bash
# Global install
npm install -g openclaw@latest

# Install the OS daemon (launchd on macOS, systemd user service on Linux)
openclaw onboard --install-daemon

# Verify everything is healthy
openclaw doctor
```

`openclaw doctor` checks the Node version, config path, gateway reachability, and which channels / LLMs are configured. Run it any time something stops working — it's almost always the first thing to tell you what's wrong.

Config lives at `~/.openclaw/config.yaml`. You can edit it by hand or use the `openclaw config set` CLI — both write to the same file.

## Configure an LLM provider

OpenClaw supports three cloud providers out of the box: Anthropic, OpenAI, and Google. Pick one (or several — you can switch per-conversation).

### Option A — CLI

```bash
# Pick a provider
openclaw config set model.provider anthropic   # or: openai, google

# Set an API key
openclaw config set model.apiKey sk-ant-...

# Pick a specific model id
openclaw config set model.id claude-sonnet-4-6
```

### Option B — edit `~/.openclaw/config.yaml`

```yaml
models:
  default: anthropic
  providers:
    anthropic:
      apiKey: sk-ant-...
      model: claude-sonnet-4-6
    openai:
      apiKey: sk-...
      model: gpt-4o-mini
    google:
      apiKey: ...
      model: gemini-2.5-pro
```

After any config change, restart the daemon so it picks up the new values:

```bash
openclaw restart
```

## Local LLM with Ollama (zero-cost, offline)

This is my favourite OpenClaw feature. If you run Ollama on the same machine, OpenClaw can drive it through Ollama's OpenAI-compatible endpoint — no API key, no network hop, no usage meter.

```bash
# 1. Pull a model (pick based on your RAM budget)
ollama pull llama3.2:3b        # fast, lightweight, good for routing + simple skills
ollama pull qwen2.5:14b        # better reasoning, ~9 GB RAM
ollama pull llama3.3:70b       # strongest open model, needs serious hardware

# 2. Make sure Ollama is running
ollama serve &
```

Then point OpenClaw at it:

```yaml
# ~/.openclaw/config.yaml
models:
  default: ollama-local
  providers:
    ollama-local:
      endpoint: http://127.0.0.1:11434/v1
      api: openai-responses       # or: openai-completions
      model: llama3.2:3b
      contextWindow: 65536        # aim for ≥ 64K — OpenClaw packs a lot into the prompt
```

Restart, and everything — channels, skills, memory — now flows through your local model. For development and long-running background automations that would rack up huge API bills, this is the move.

## Remote control from Telegram

This is how you turn OpenClaw into something you can actually reach from your phone.

1. **Create the bot.** Open Telegram, DM `@BotFather`, run `/newbot`, pick a name and a username. BotFather replies with a token that looks like `1234567890:AAEh...`.
2. **Add the token to OpenClaw** and lock it down to your own Telegram user id (otherwise anyone who guesses the bot handle can talk to your assistant):

    ```yaml
    # ~/.openclaw/config.yaml
    channels:
      telegram:
        enabled: true
        botToken: "1234567890:AAEh..."
        dmPolicy: allowlist
        allowedUsers:
          - 123456789          # your Telegram numeric user id
    ```

3. **Start the gateway** so the Telegram bridge comes up:

    ```bash
    openclaw gateway
    ```

4. **Pair by DMing your bot.** Send `/start`. If your id is in `allowedUsers`, the bot replies and routes the message to your LLM + skills. If not, it silently drops the message — which is the behaviour you want.

From there, every DM to the bot is a gateway request, every reply streams back as Telegram messages, and skills can send you proactive notifications (e.g. "build finished", "calendar conflict detected").

## Remote control from Discord

Discord is almost the same idea, but the invite flow is a bit different.

1. **Create the app.** Go to the [Discord Developer Portal](https://discord.com/developers/applications), create a new application, then create a bot under it. Copy the bot token.
2. **Generate an invite URL.** In the app's OAuth2 → URL Generator, select scopes `bot` and `applications.commands`, then pick the minimum permissions you need (usually `Send Messages`, `Read Message History`). Open the generated URL and invite the bot to a server you control.
3. **Wire it into OpenClaw:**

    ```yaml
    channels:
      discord:
        enabled: true
        botToken: "MTIzNDU2Nzg5..."
        dmPolicy: allowlist
        allowedUsers:
          - "your_discord_user_id"
        allowedGuilds:
          - "your_server_id"
    ```

4. **Restart the gateway** and DM the bot from Discord, or @-mention it in an allowed channel. Same routing semantics as Telegram: message → gateway → LLM + skills → streamed reply.

`dmPolicy: allowlist` is the safest default for both channels. Only flip to `open` if you've wrapped the bot in some other access control.

## Skills: find, install, build

Skills are where OpenClaw becomes *yours*. Three ways to add them:

### 1. The curated marketplace — skills.sh

[skills.sh](https://skills.sh) is the official registry. Browse, pick one, copy the install command. There's also a community-maintained [awesome-openclaw-skills](https://github.com/VoltAgent/awesome-openclaw-skills) list if you want a wider net.

### 2. Native CLI install

```bash
# Search
openclaw skills search calendar

# Install into your user skills directory
openclaw skills install calendar-sync
openclaw skills install browser-automate

# See what's installed
openclaw skills list
```

### 3. ClawHub (npm-backed installer)

Some skills are distributed as npm packages — `clawhub` is the installer that knows how to wire them into OpenClaw's skills directory correctly.

```bash
npx clawhub@latest install notes-summariser
npx clawhub@latest install home-assistant-control
```

### 4. Write your own

A skill is literally a folder. The minimum shape:

```
my-skill/
  SKILL.md        # natural-language instructions + YAML frontmatter
  index.ts        # optional: code the skill can run
```

`SKILL.md` starts with frontmatter describing the skill's id, description, and which tools/capabilities it exposes. The body is the prompt OpenClaw injects when the skill is invoked — write it the way you'd write instructions for a new hire, not the way you'd write code comments.

### Security: always read the skill first

This is the single most important habit. Before installing anything:

```bash
cat ~/.openclaw/skills/<name>/SKILL.md
ls ~/.openclaw/skills/<name>/
```

A skill runs on *your* machine with *your* credentials. Treat the registry with the same suspicion you'd treat any `npm install` — read the instructions, scan the scripts, and if anything asks for more access than it needs, don't install it. The gateway's allowlists help, but the first line of defence is you reading `SKILL.md`.

## From personal assistant to business gateway

Everything up to here has framed OpenClaw as a personal assistant — a daemon that lives on your laptop and lets you run shell commands from Telegram. That's the smallest interesting shape. The more useful shape, once you've lived with it for a week or two, is a **business gateway**: one OpenClaw process handles inbound leads on your website, triages customer support tickets from Slack, and keeps your CRM in sync, all by composing skills and MCP servers.

A quick honesty note first: **OpenClaw ships with no native CRM, no lead pipeline, no ticketing system.** The core is a gateway + skill runner + MCP client. Every business capability below is assembled from community skills and MCP servers pointing at third-party systems (HubSpot, Salesforce, Zendesk, ServiceNow). That's a feature, not a limitation — you get to pick the providers you already pay for, and swap them without rebuilding the agent.

![OpenClaw business architecture — channels on top route into the Gateway; the Gateway runs an agent turn against an LLM router and a skills catalog, while an MCP tool hub bridges CRM, knowledge-base, notifications, and analytics systems, with a local persistence strip for sessions and transcripts](/diagrams/openclaw-business-architecture.svg)

### Finding &amp; qualifying leads

A typical inbound-lead loop looks like this. A visitor types into WebChat (or replies on WhatsApp / Slack / email). The gateway hands the message to the LLM with a skill catalog. The LLM picks **`apollo`** to enrich the contact (company, headcount, title), **`sentiment-priority-scorer`** to grade intent, and **`hubspot`** (or the HubSpot MCP server via Composio) to search for an existing contact or create one. If the lead is hot, **`slack-notify`** drops a summary into `#sales`.

The community catalog at [awesome-openclaw-skills](https://github.com/VoltAgent/awesome-openclaw-skills) has the pieces you'd actually wire together:

- **Enrichment &amp; qualification:** `apollo` (Apollo.io people/org data), `sentiment-priority-scorer`, browser-automation skills for scraping LinkedIn / company sites.
- **CRM sync:** official `hubspot` skill (`api.hubapi.com` for contacts, companies, deals, tickets, pipelines), `attio-enhanced`, `workcrm` (local-first), `kvcore-mcp-cli`.
- **Outreach:** `cold-email`, `cold-outreach`, `campaign-orchestrator`, `foxreach`, `brevo`, `kit-email-operator`, `email-marketing-2`.

Minimum wiring:

```bash
# Skills that do the lead work
openclaw skills install apollo
openclaw skills install sentiment-priority-scorer
openclaw skills install hubspot

# Or route through a managed MCP server instead of a raw skill
openclaw mcp set hubspot \
  --transport streamable-http \
  --url https://mcp.composio.dev/hubspot/<token>
```

Third-party vendor blogs cite 60–70 % autonomous qualification or ticket-resolution rates for workflows like this — treat those as vendor claims, not OpenClaw-published benchmarks. Start with a narrow pipeline (one channel, two skills, one CRM) and measure your own numbers before scaling.

### Managing your agent software ecosystem

"Software management" in an OpenClaw deployment means three catalogs the gateway owns, all driven by CLI:

- **Skills catalog** — `openclaw skills list | search | install`. Workspace &gt; global &gt; bundled precedence; a workspace skill shadows a global one with the same id, which is usually what you want when you're iterating.
- **MCP registry** — `openclaw mcp list | set | unset`. stdio, HTTP, SSE, and streamable-HTTP transports; each server is declared in `openclaw.json` under `mcp.servers`, which means you can version-control the whole tool surface.
- **Model router** — Anthropic, OpenAI, Gemini, Ollama, LM Studio, vLLM, or any OpenAI-compatible endpoint. Switchable per conversation, so you can default to a local Llama for routing and cheap turns and escalate to Claude for hard reasoning.

Day-two ops stay boring on purpose: `openclaw doctor` for health, `openclaw onboard --install-daemon` for a launchd / systemd user service, and managed hosts (oneclaw.net, clawhost, getclaw.sh) if you don't want to babysit a VPS. The gateway writes sessions + config locally, so if the process dies you don't lose in-flight conversations.

### Customer management

The same gateway you use for leads doubles as a customer-support hub. Inbound channels (WebChat, WhatsApp, Slack, Telegram, Discord) funnel into the gateway; the agent classifies the message, looks up the customer in HubSpot / Zendesk / ServiceNow via MCP, and answers from a **RAG** knowledge base backed by Pinecone, Qdrant, or pgvector (BYO — there are community skills for each). If the agent can't resolve it, it opens a ticket with the right queue and @-mentions the on-call in Slack.

Two things that make this tractable in practice:

1. **Transcript persistence is local.** The gateway stores every turn on disk, and MCP tools like `conversations_list` / `messages_read` / `events_poll` let another agent (or a human supervisor) audit a conversation without going through a vendor dashboard.
2. **Voice + Canvas** are first-class. The iOS / Android mobile nodes let a support lead listen to a live customer conversation and correct the agent mid-turn — which is the feature that usually separates "AI support toy" from "AI support you'd actually ship".

### Communication flow: an inbound lead end-to-end

Here's what a single "request a demo" message actually does:

1. **Visitor → Gateway.** Message hits WebChat, the channel plugin forwards it over WS to the gateway. Gateway applies policy (workspace, allowlist, rate limit).
2. **Gateway → LLM.** System prompt + skill catalog + MCP tool schemas + conversation history go out as the turn.
3. **LLM → Skills.** Model calls `apollo.enrich(email)` to get company / size / title.
4. **Skills → MCP (CRM).** `hubspot.searchContact` hits `api.hubapi.com`; if none, `hubspot.createContact`.
5. **Skills → LLM.** Enriched context returns, `sentiment-priority-scorer` flags `priority: HIGH, intent: buy`.
6. **LLM → Gateway → Visitor.** Reply streams back token-by-token through the same WebChat socket.
7. **Async side-effects.** In parallel, the gateway fires `slack-notify.send("#sales", summary)`, `hubspot.updateContact(stage: "demo-requested")`, and persists the transcript locally. Optional `langfuse.log` closes the analytics loop.

![OpenClaw lead-handling swim-lane — visitor message arrives at the Gateway, the LLM invokes apollo and hubspot skills, the HubSpot MCP server returns contact data, the enriched reply streams back to the visitor, and Slack notification + CRM update + transcript persist run as async side-effects](/diagrams/openclaw-lead-flow.svg)

The architecture collapses nicely: one gateway, one LLM turn, many side-effects. Swap WebChat for WhatsApp or Slack and only the inbound lane changes — everything downstream is the same.

## Wrap-up

OpenClaw ends up being the kind of tool that feels small at first — a local daemon, a config file, a few channels — and then compounds. Once the gateway is running and you've added two or three skills, you stop thinking about *where* the assistant lives and start thinking about *what else it should automate*.

A few things I'd suggest once the basics are working:

- Switch the default model to a local Ollama one, and keep a cloud provider configured as a fallback for hard reasoning tasks.
- Add a skill that summarises your daily calendar and sends it to your Telegram at 8am — 10 minutes to build, pays dividends every day.
- Read every `SKILL.md` before installing. Twice.

### Links

- **Docs:** [docs.openclaw.ai](https://docs.openclaw.ai)
- **Source:** [github.com/openclaw/openclaw](https://github.com/openclaw/openclaw)
- **Skill registry:** [skills.sh](https://skills.sh)
- **Community skills list:** [awesome-openclaw-skills](https://github.com/VoltAgent/awesome-openclaw-skills)

---

## Related Ecosystem

Self-hosted AI assistants sit at the intersection of several active open-source communities. These projects and standards share design goals with OpenClaw and can serve as reference implementations or integration targets.

**Gateway & Routing**
- [LiteLLM](https://github.com/BerriAI/litellm) — Unified LLM API proxy supporting 100+ providers with load balancing and rate limiting.
- [Ollama](https://ollama.com/) — Local LLM runtime with model management, REST API, and OpenAI-compatible endpoints.

**Agent Frameworks**
- [LangGraph](https://langchain-ai.github.io/langgraph/) — Stateful agent orchestration with cycles, branching, and human-in-the-loop.
- [CrewAI](https://www.crewai.com/) — Multi-agent framework for role-based task delegation and collaboration.

**Remote Control & Messaging**
- [BotFather](https://t.me/botfather) — Telegram's official bot creation interface and token management.
- [Discord Developer Portal](https://discord.com/developers/docs/intro) — Bot account setup, slash commands, and WebSocket gateway documentation.

**Skills & Extensibility**
- [MCP Specification](https://modelcontextprotocol.io/) — Open protocol for connecting LLMs to external tools, data sources, and services.
- [skills.sh](https://skills.sh) — Community skill registry with installable packages for common agent capabilities.

*Building self-hosted AI workflows? [Let's connect](mailto:contact@aiwithvd.com). I help teams design local-first AI stacks — LLM routing, skill architecture, and secure remote-control deployments.*

---

### Building a Production-Ready LangChain DeepAgent with SSE Streaming

URL: https://aiwithvd.com/blog/building-langchain-deepagent-sse-streaming
Published: 2026-04-13
Tags: LangChain, LangGraph, FastAPI, Ollama, SSE, AI Agents, Production

# Building a Production-Ready LangChain DeepAgent with SSE Streaming

AI agents are everywhere — but most demos stop short of the hard parts: observable reasoning, real-time output, stateful conversations, and production-grade rate limiting. I recently built [LangChain DeepAgent](https://github.com/aiwithvd/langchain_deepagent), a production-ready agent API that tackles all of these head-on.

In this post I'll walk through the architecture, the four specialized skills, how SSE streaming works in practice, and the production patterns that make it reliable.

## The Problem

Most LLM integrations are fire-and-forget: send a query, wait 10–30 seconds, get a response. That's fine for simple Q&A, but it breaks down for autonomous agents that:

- Need to **reason, plan, search, and write** across multiple steps
- Should **stream output in real-time** so users see progress, not a spinner
- Must **remember context** across a multi-turn conversation
- Need **abuse protection** to be safely exposed to users

LangGraph + FastAPI + SSE solves all of this cleanly.

## Architecture Overview

![LangChain DeepAgent architecture — FastAPI fronts a LangGraph agent with four skills, an Ollama LLM, and Redis for rate-limiting + session memory](/diagrams/deepagent-architecture.svg)

At a glance:

```
Client → FastAPI → DeepAgent (LangGraph) → 4 Skills
                         ↕
                       Ollama (llama3.2:3b)
                         ↕
                       Redis (rate limiting + session memory)
```

**Core components:**

| Layer | Technology | Role |
|-------|------------|------|
| **API** | FastAPI | Async HTTP, SSE streaming, rate limiting middleware |
| **Agent** | LangGraph | Orchestrates agent graph (up to 25 iterations) |
| **LLM** | Ollama llama3.2:3b | Local inference, ~2GB model, fully private |
| **Skills** | 4 custom tools | think, plan, web_search, write_report |
| **Memory** | LangGraph MemorySaver | Multi-turn session persistence |
| **Rate Limiting** | Redis + fastapi-limiter | 10 req/60s per IP |

## The Four Skills

Each skill is a Python `@tool` async function paired with a `SKILL.md` file containing YAML frontmatter. The DeepAgent framework discovers skills at runtime by scanning the skills directory — no hardcoded registration needed.

### 1. Think — Chain-of-Thought Reasoning

The most fundamental skill. It structures the agent's internal reasoning into a predictable format:

```
[Question] → [Reasoning steps] → [Conclusion]
```

Useful for complex analysis where you want the agent to show its work rather than jump straight to an answer.

### 2. Plan — Task Decomposition

Breaks high-level goals into ordered subtasks with tool assignments and success criteria. Given a goal and optional context, it returns a numbered markdown plan:

```
1. Search for recent papers on the topic (web_search)
2. Identify key themes and findings (think)
3. Structure a professional summary (write_report)
```

### 3. Web Search — Real-Time DuckDuckGo

Executes live searches without requiring an API key. The implementation runs in `asyncio.to_thread()` to avoid blocking the async event loop — a critical detail when you have concurrent requests.

```python
@tool
async def web_search(query: str) -> str:
    """Search the web using DuckDuckGo."""
    results = await asyncio.to_thread(_ddg_search, query, max_results=10)
    return format_results(results)
```

### 4. Write Report — Structured Output

A pure formatting skill that wraps research into professional markdown templates: executive summary, key findings, analysis, conclusion — with an automatic timestamp appended.

## SSE Streaming: Watch the Agent Think

Here's the end-to-end shape of a single request and the SSE event timeline the client observes:

![DeepAgent request + SSE streaming flow — sequence diagram from client through FastAPI, LangGraph, Ollama, and Redis](/diagrams/deepagent-flow.svg)

The `/api/v1/agent/stream` endpoint streams events as the agent works. Instead of waiting for a complete answer, the client receives a live event stream:

```
event: token
data: "Let me search for recent developments..."

event: tool_start
data: web_search

event: tool_end
data: "Found 8 relevant results about..."

event: token
data: "Based on the research, here are the key findings..."

event: done
data: {"session_id": "550e8400-e29b-41d4-a716-446655440000"}
```

Five event types cover the full lifecycle:
- **`token`** — Individual LLM tokens as they generate
- **`tool_start`** — Name of the skill being invoked
- **`tool_end`** — Truncated skill output (max 500 chars)
- **`done`** — Final JSON payload with `session_id`
- **`error`** — Error message if execution fails

The client can render all of this progressively — showing the agent's reasoning, tool calls, and final answer as they happen rather than after a long wait.

## Multi-Turn Session Memory

LangGraph's `MemorySaver` persists conversation state across requests. The pattern is simple:

```python
# First request — no session_id needed
POST /api/v1/agent/run
{"query": "Research the latest trends in vector databases"}
# Response includes: {"session_id": "abc-123", "answer": "..."}

# Follow-up request — pass the session_id back
POST /api/v1/agent/run
{"query": "Now write a report based on what you found", "session_id": "abc-123"}
# Agent has full context of the previous research turn
```

The agent maintains the full message history, so follow-ups like "summarise that" or "go deeper on point 3" work naturally without re-sending context.

## Redis Rate Limiting

`fastapi-limiter` wraps Redis to enforce per-IP limits at the middleware layer:

```python
from fastapi_limiter.depends import RateLimiter

@router.post("/run")
async def run_agent(
    request: AgentRequest,
    _: None = Depends(RateLimiter(times=10, seconds=60))
):
    ...
```

- **Default**: 10 requests per 60-second window per IP
- Fully configurable via `RATE_LIMIT_REQUESTS` / `RATE_LIMIT_SECONDS` env vars
- Returns HTTP 429 when exceeded
- Counters auto-expire in Redis — no cleanup needed

## API Endpoints

**POST `/api/v1/agent/run`** — Synchronous execution

```bash
curl -X POST http://localhost:8000/api/v1/agent/run \
  -H "Content-Type: application/json" \
  -d '{"query": "Research AI agents and write a report"}'

# Response:
{
  "answer": "...",
  "tool_calls": [
    {"tool": "web_search", "input": "AI agent frameworks 2026", "output": "..."},
    {"tool": "write_report", "input": "...", "output": "..."}
  ],
  "iterations": 4,
  "elapsed_seconds": 8.3,
  "session_id": "550e8400-..."
}
```

**GET `/api/v1/agent/stream?query=...&session_id=...`** — SSE streaming

```bash
curl "http://localhost:8000/api/v1/agent/stream?query=Research+vector+databases"
```

**GET `/health/ready`** — Readiness check (Ollama + Redis connectivity)

## Deployment

Three services, one command:

```bash
git clone https://github.com/aiwithvd/langchain_deepagent.git
cd langchain_deepagent
cp .env.example .env
docker compose up --build
```

Docker Compose orchestrates:
1. **App** — FastAPI on port 8000
2. **Ollama** — Pulls `llama3.2:3b` (~2GB) automatically on first run
3. **Redis** — Rate limiting and session storage

The multi-stage Dockerfile separates build dependencies from the runtime image, keeping the production container lean.

## Production Patterns Worth Stealing

**1. Thread-safe agent singleton**
The `DeepAgent` instance is created once at startup via a factory pattern and shared across requests — avoids repeated model loading overhead.

**2. Async-safe web search**
`asyncio.to_thread()` prevents the synchronous DuckDuckGo library from blocking the event loop under concurrent load.

**3. Structured logging**
`structlog` outputs colored text in debug mode and machine-readable JSON in production — same code, different config.

**4. Graceful readiness degradation**
`/health/ready` returns `200` even when Ollama or Redis is unreachable (with `status: degraded`), letting your orchestrator decide whether to route traffic rather than killing the container.

## Key Takeaways

1. **LangGraph is the right abstraction** — Graph-based agent orchestration handles retries, branching, and iteration limits cleanly vs hand-rolled loops
2. **SSE > polling for agents** — Streaming tokens and tool events gives users observable progress; polling forces them to wait blind
3. **Session memory is table stakes** — Multi-turn context makes agents genuinely useful for research and writing workflows
4. **Local LLMs are production-viable** — llama3.2:3b on a single machine handles structured tasks reliably with zero API costs
5. **Rate limit at the middleware layer** — Redis + fastapi-limiter is simpler and more reliable than application-level throttling

## Code & Resources

- **Full source code:** [github.com/aiwithvd/langchain_deepagent](https://github.com/aiwithvd/langchain_deepagent)
- **Docker setup:** One-command deployment with `docker compose up`
- **Skill system:** Add new capabilities by dropping a `.py` + `SKILL.md` into the skills directory

---

*Want to build production AI agents? [Let's connect](mailto:contact@aiwithvd.com). I advise teams on agent architecture, LLM integration, and production deployment.*

---

### Getting Started with LLM Agents in Production

URL: https://aiwithvd.com/blog/getting-started-with-llm-agents
Published: 2026-04-09
Tags: LLM, LangChain, FastAPI, AI Agents, Production

# Getting Started with LLM Agents in Production

Building LLM agents that actually work in production requires more than just connecting to an API. In this guide, I'll share patterns and practices I've learned from deploying agentic systems at scale.

## What Are LLM Agents?

An LLM agent is an AI system that can:
- **Reason** through complex problems step by step
- **Plan** and break down tasks into manageable pieces
- **Execute** actions using tools (search, APIs, code execution)
- **Remember** context across multiple interactions

Unlike simple chatbots, agents can take autonomous actions to accomplish goals.

## Core Architecture Components

### 1. The Agent Loop

The fundamental pattern is a loop:

```
Observation → Thought → Action → Result → Observation...
```

Your agent observes the environment, thinks about what to do, takes an action, observes the result, and repeats until the goal is achieved.

### 2. Tool Selection

Give your agent specific, well-documented tools:

- **Web Search** - DuckDuckGo, Google Search API
- **Calculator** - For mathematical operations
- **Code Execution** - Python sandbox for computations
- **Document Retrieval** - RAG for domain-specific knowledge
- **APIs** - Custom business logic endpoints

### 3. Memory Management

Agents need memory to handle multi-turn conversations:

- **Short-term**: Conversation history within a session
- **Long-term**: Persistent user preferences and facts
- **Working memory**: Scratchpad for complex reasoning

## Production Considerations

### Error Handling

Agents will fail. Plan for it:

- **Timeout limits** - Prevent infinite loops
- **Retry logic** - Handle transient failures gracefully
- **Fallback responses** - When all else fails, have a backup
- **Human escalation** - Know when to hand off to a human

### Observability

You need visibility into what your agent is doing:

- **Step-by-step logging** - Trace every decision
- **Cost tracking** - LLM calls add up quickly
- **Latency metrics** - User experience matters
- **Success/failure rates** - Measure outcomes

### Rate Limiting

Protect your infrastructure:

- **Per-user limits** - Prevent abuse
- **Global rate limiting** - Protect downstream services
- **Queue management** - Handle traffic spikes

## Example: DeepAgent Implementation

Here's a simplified version of the [DeepAgent](/projects/deepagent) I built:

```python
from langgraph.graph import StateGraph
from langchain_ollama import ChatOllama

# Define agent state
class AgentState:
    messages: list
    next_step: str

# Create the graph
workflow = StateGraph(AgentState)

# Add nodes for each skill
workflow.add_node("think", think_skill)
workflow.add_node("plan", plan_skill)
workflow.add_node("search", search_skill)
workflow.add_node("report", report_skill)

# Define transitions
workflow.add_conditional_edges("think", route_next_step)
workflow.add_edge("plan", "execute")
workflow.add_edge("search", "synthesize")
workflow.add_edge("report", END)
```

The key insight: **treat your agent as a state machine**, not just a prompt pipeline.

## Deployment Strategy

### Local Models vs. Cloud APIs

| Factor | Local (Ollama) | Cloud (OpenAI) |
|--------|----------------|------------------|
| Cost | Hardware only | Per-token pricing |
| Latency | Fast (no network) | Network dependent |
| Control | Full | Limited |
| Quality | Varies by model | Generally higher |

For production, I recommend a hybrid approach:
- Use cloud APIs for complex reasoning tasks
- Use local models for simpler, high-volume operations

### Infrastructure Stack

My recommended production stack:

- **FastAPI** - High-performance async API
- **Redis** - Rate limiting and session storage
- **PostgreSQL** - Persistent memory storage
- **Celery** - Background task processing
- **LangGraph** - Agent orchestration

## Key Takeaways

1. **Start simple** - Don't over-engineer your first agent
2. **Measure everything** - You can't improve what you don't track
3. **Plan for failures** - Agents will make mistakes
4. **Design for observability** - Know what your agent is doing
5. **Iterate quickly** - Real-world feedback beats theory

## Next Steps

Ready to build your own agent? Check out these resources:

- [LangGraph documentation](https://langchain-ai.github.io/langgraph/)
- [DeepAgent project on GitHub](https://github.com/aiwithvd/langchain_deepagent)
- [FastAPI best practices](https://fastapi.tiangolo.com/best-practices/)

---

*Have questions about building production LLM agents? [Let's connect](mailto:contact@aiwithvd.com).*

---

### Building a Production-Ready Document Extraction Pipeline with FastAPI and MinerU

URL: https://aiwithvd.com/blog/building-document-extraction-pipeline-fastapi-ollama
Published: 2026-04-09
Tags: FastAPI, MinerU, Ollama, OCR, Document AI, LLM, Celery, Production

# Building a Production-Ready Document Extraction Pipeline with FastAPI and MinerU

Every AI engineer eventually faces the document processing problem: how do you turn a mess of PDFs, scanned images, and unstructured files into clean, structured data that your systems can actually use?

I recently built a [Document Extraction Pipeline](https://github.com/aiwithvd/document-extraction-pipeline) that processes thousands of documents daily. In this post, I'll share the architecture, key decisions, and lessons learned from taking it from prototype to production.

## The Problem

Businesses are drowning in documents:
- **Invoices** that need data extraction for accounting
- **Legal contracts** requiring clause analysis
- **ESG reports** with structured sustainability metrics
- **Forms** that need automated processing

The challenge isn't just extracting text—it's extracting **structured, validated data** that integrates with downstream systems.

## Architecture Overview

Here's the system I built:

![Document Extraction Pipeline architecture — FastAPI + JWT accepts uploads, persists to MinIO, enqueues Celery jobs in Redis. Workers run MinerU OCR and LLM extraction and store JSONB in Postgres](/diagrams/docextraction-architecture.svg)

### Tech Stack

| Component | Technology | Why |
|-----------|------------|-----|
| **API Layer** | FastAPI | Async support, automatic OpenAPI docs, Python-native |
| **Task Queue** | Celery + Redis | Background processing, rate limiting, retries |
| **OCR/Layout** | MinerU (vlm-auto-engine) | Structured Markdown output preserving tables, headers & multi-column layouts |
| **LLM** | OpenAI GPT-4o-mini / Ollama | Cloud or local inference, configurable via env vars |
| **Storage** | MinIO | S3-compatible, self-hosted, fast |
| **Database** | PostgreSQL + JSONB | Flexible schema for varying document types |

### The Processing Pipeline

```
Upload → OCR → LLM Extraction → Validation → Storage
```

Here's the same pipeline as a request life-cycle — the API acknowledges the upload immediately, a worker does the heavy lifting asynchronously, and the client polls for the structured result:

![Document Extraction request flow — POST /upload returns 202 immediately; a Celery worker pulls the file from MinIO, runs MinerU OCR, calls the LLM for schema-driven extraction, and persists JSONB to Postgres. Clients poll GET /result until status is completed](/diagrams/docextraction-flow.svg)

**Step 1: Document Upload**
- Supports PDF, PNG, JPG, TIFF (up to 10MB)
- JWT authentication with per-user rate limits
- Async validation and virus scanning

**Step 2: OCR & Layout Analysis**
- MinerU's vlm-auto-engine converts PDFs and images to structured Markdown
- Tables, headers, and multi-column layouts are preserved (not flattened to raw text)
- Output is normalized and chunked for LLM consumption

**Step 3: LLM-Powered Extraction**
- Ollama runs locally for zero API costs
- Structured output using JSON schemas
- Support for Invoice, Legal, and ESG document types

**Step 4: Data Validation**
- Schema validation with Pydantic
- Business rule validation (e.g., "total must equal sum of line items")
- Confidence scoring for each extracted field

## Key Implementation Details

### Async Processing with Celery

```python
from celery import Celery
from fastapi import BackgroundTasks

app = Celery('document_processor', broker='redis://localhost:6379')

@app.task(bind=True, max_retries=3)
def process_document(self, document_id: str):
    try:
        # 1. Download from storage
        doc = download_document(document_id)
        
        # 2. OCR & layout analysis with MinerU
        markdown_content = extract_with_mineru(doc)  # returns structured Markdown
        
        # 3. LLM structured extraction
        structured_data = extract_with_llm(markdown_content, schema=document.schema)
        
        # 4. Validate and save
        save_extraction_result(document_id, structured_data)
        
    except Exception as exc:
        # Retry with exponential backoff
        raise self.retry(exc=exc, countdown=2 ** self.request.retries)
```

**Why Celery?**
- Documents can take 5-30 seconds to process
- API stays responsive with immediate "processing" response
- Automatic retries handle transient failures
- Horizontal scaling by adding more workers

### Local LLM with Ollama

```python
from langchain_ollama import ChatOllama
from langchain_core.prompts import ChatPromptTemplate

llm = ChatOllama(
    model="llama3.2:3b",
    temperature=0.1,  # Low creativity for extraction
    format="json"     # Force JSON output
)

extraction_prompt = ChatPromptTemplate.from_template("""
Extract structured data from this document text.
Document type: {doc_type}
Schema: {schema}

Text:
{text}

Return ONLY valid JSON matching the schema.
""")

chain = extraction_prompt | llm
```

**Why Ollama?**
- **Zero API costs** - Critical for high-volume processing
- **Data privacy** - Documents never leave your infrastructure
- **Low latency** - No network round-trips to external APIs
- **Customizable** - Fine-tune models for your specific document types

### Flexible Schema with PostgreSQL JSONB

```sql
CREATE TABLE extractions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID REFERENCES documents(id),
    doc_type VARCHAR(50),
    extracted_data JSONB,  -- Flexible schema per document type
    confidence_score FLOAT,
    created_at TIMESTAMP DEFAULT NOW()
);

-- Query examples
SELECT extracted_data->>'total_amount' 
FROM extractions 
WHERE doc_type = 'invoice' 
  AND extracted_data->>'vendor' = 'Acme Corp';
```

**Why JSONB?**
- Invoices have different fields than legal contracts
- No schema migrations when adding new document types
- PostgreSQL indexes JSONB for fast queries
- Native JSON operators for flexible querying

## Production Challenges & Solutions

### Challenge 1: OCR Quality

**Problem:** Scanned documents with poor quality, complex layouts, or multi-column tables.

**Solutions:**
- **MinerU vlm-auto-engine**: Understands document layout natively — no manual deskew or preprocessing needed
- **Structured Markdown output**: Tables and multi-column text are preserved automatically, dramatically improving LLM extraction accuracy
- **Confidence thresholds**: Flag low-confidence extractions for human review
- **Human-in-the-loop**: Manual review queue for uncertain cases

### Challenge 2: LLM Consistency

**Problem:** LLMs sometimes return malformed JSON or hallucinate fields.

**Solutions:**
- **Structured output with Pydantic:** Enforce schema at code level
- **Retry logic:** Regenerate on validation failures
- **Temperature tuning:** Lower temperature (0.1) for more deterministic outputs
- **Few-shot prompting:** Include examples in prompts

### Challenge 3: Rate Limiting & Fair Use

**Problem:** Prevent abuse and ensure fair resource distribution.

**Implementation:**
```python
from slowapi import Limiter, _rate_limit_exceeded_handler
from slowapi.util import get_remote_address

limiter = Limiter(key_func=get_remote_address)

@app.post("/upload")
@limiter.limit("10/minute")  # 10 uploads per minute per IP
async def upload_document(
    request: Request,
    file: UploadFile,
    current_user: User = Depends(get_current_user)
):
    # Per-user limits in addition to IP limits
    check_user_quota(current_user.id)
    ...
```

### Challenge 4: Monitoring & Observability

**What we track:**
- Processing time per document
- OCR confidence scores
- LLM extraction accuracy
- Error rates by document type
- Queue depth and worker utilization

**Tools:**
- Prometheus for metrics
- Grafana for dashboards
- Structured logging with correlation IDs

## Performance Results

After 3 months in production:

| Metric | Before (Manual) | After (Pipeline) | Improvement |
|--------|-----------------|------------------|-------------|
| **Processing Time** | 5-10 min/doc | 15-30 sec/doc | **20x faster** |
| **Accuracy** | 85% | 94% | **+9 points** |
| **Cost per doc** | $0.50 (labor) | $0.02 (compute) | **96% cheaper** |
| **Throughput** | 50/day | 1000+/day | **20x scale** |

## Key Takeaways

1. **Layout-aware OCR wins** - MinerU's structured Markdown output dramatically improves LLM extraction accuracy vs raw text
2. **Async processing is essential** - Don't block APIs on long-running tasks
3. **Local LLMs are viable** - For high-volume, structured tasks, Ollama beats API costs
4. **Schema flexibility matters** - JSONB lets you iterate without migrations
5. **Observability from day one** - You can't improve what you don't measure

## Code & Resources

- **Full source code:** [github.com/aiwithvd/document-extraction-pipeline](https://github.com/aiwithvd/document-extraction-pipeline)
- **Live demo:** Contact me for access
- **Docker setup:** One-command deployment with `docker-compose up`

## What's Next?

Future improvements I'm working on:
- **Fine-tuned models** for specific document types
- **Active learning** pipeline to improve extraction quality
- **Multi-language support** beyond English
- **Integration webhooks** for real-time downstream updates

---

*Need help building document processing pipelines? [Let's connect](mailto:contact@aiwithvd.com). I advise teams on production AI architecture and implementation.*