Document Extraction Pipeline: Scaling AI Workloads with FastAPI and Celery

At a Glance

The Product

Businesses drown in documents that carry structured data trapped in unstructured formats: invoices, contracts, ESG reports. Manual data entry is slow, error-prone, and expensive, and most extraction tools fail the moment a layout changes.

This pipeline turns PDFs and images into clean, structured JSON. Upload a document, get back typed fields matching a schema. OCR handles the reading, an LLM handles the understanding, and the architecture handles the part most AI demos skip: doing this reliably, concurrently, and securely in production.

The engagement centered on four objectives:

• Extract structured data from PDFs and images across multiple document types.
• Keep the API responsive while OCR and LLM inference run for seconds or minutes per document.
• Support both cloud LLMs and fully local inference for cost and privacy flexibility.
• Ship production concerns complete: authentication, rate limiting, storage, and job tracking.

The Challenge

The core tension of this project was heavy AI inference inside a responsive API. OCR and LLM extraction take seconds to minutes per document, and an API that blocks on them collapses under its second concurrent user.

1. Slow work behind a fast interface. A single document can take longer to process than any sane HTTP timeout. The architecture had to accept work instantly and deliver results asynchronously, without losing jobs in between.

2. Layout chaos. Invoices, legal documents, and ESG reports share nothing structurally. Tables, multi-column layouts, scanned images, and mixed quality inputs all had to resolve into the same clean schemas.

3. The cost and privacy fork. Some workloads justify a cloud LLM; others demand data never leaves the machine. One codebase had to serve both without branching into two products.

4. Production is a feature. Authentication, per-user rate limiting, object storage, retry behavior, and job status tracking are where prototypes go to die. They were in scope from the first commit.

My Approach

I built the system async-first, with a clean separation between the API that accepts work and the workers that do it.

FastAPI at the front, Celery at the back. The API validates uploads, persists files to MinIO object storage, enqueues a Celery job through Redis, and returns a job ID immediately. Clients poll for results while workers process in the background. Throughput scales by adding workers, not by making users wait.

A two-stage extraction pipeline. MinerU, running its vlm-auto-engine, converts documents into structured Markdown that preserves tables and layout, the context an LLM needs to extract fields correctly. The LLM stage then maps that Markdown into typed schemas, with pre-built templates for Invoice, Legal, and ESG document types and support for custom extraction templates.

Provider-agnostic LLM layer. The extraction stage runs against OpenAI GPT-4o-mini for managed inference or a local Ollama model for fully private, zero-API-cost processing. Switching is configuration, not code.

Production hardening throughout. JWT authentication with per-user rate limiting, PostgreSQL with JSONB for flexible result storage, Redis doubling as broker and cache, MinIO for S3-compatible storage, and the whole stack containerized with Docker for reproducible deployment.

Engineering Highlights

• Async job architecture. Upload returns a job ID in milliseconds; Celery workers absorb the heavy lifting, and the API never blocks on inference.
• MinerU OCR front end. Layout-aware conversion to structured Markdown, preserving tables that naive OCR flattens into noise.
• Schema-driven LLM extraction. Pre-built Invoice, Legal, and ESG schemas with customizable templates, returning typed JSON instead of free text.
• Cloud or local inference. OpenAI and Ollama behind one interface, letting cost and privacy requirements pick the backend per deployment.
• JSONB result storage. PostgreSQL stores heterogeneous extraction results queryably without schema migrations per document type.
• JWT auth and rate limiting. Per-user access control and request budgets built in, not bolted on.
• S3-compatible object storage. MinIO handles originals and artifacts, keeping the database lean and the files durable.
• Fully containerized. Docker Compose brings up the entire stack, API, workers, Redis, Postgres, and MinIO, in one command.

Implemented Capabilities

Tech Stack and Deliverables

• Platform: FastAPI, Celery, Redis, PostgreSQL (JSONB), MinIO, Docker
• AI layer: MinerU (vlm-auto-engine) OCR, OpenAI GPT-4o-mini, Ollama for local inference
• Deliverables: Async extraction API, worker pipeline, three document schemas with template system, JWT auth and rate limiting, containerized deployment