Hi everyone!
This is the fourth lesson in our six-week course on building a production OCR system.
In Lesson 1 we set out the foundations of Kubernetes, in Lesson 2 we explored ten years of OCR architecture, and in Lesson 3 we deployed a vLLM server on a GPU node and pointed a small FastAPI service at it.
That service is the one we'll be improving today. It's funny, because this service typically receives less attention than anything else in the pipeline, but it's usually the first thing to break under real traffic. If you take a close look, you'll notice that it doesn't do much: it decodes the upload, rasterises the PDF, calls the model, an tidies the output with a few regexes. Twenty lines of code, in whatever web framework the team already used. And the thing is that on a laptop, it works just fine. But under concurrent load? Well, it doesn't, and the reason is more specific than Python just being slow.
This lesson covers, in detail, what is going wrong here (and why a compiled programming language addresses it). As we did with the previous Lesson, we'll also provide the code for the new deployment: a Rust gateway on Axum and Tokio sitting in from of a vLLM server running the one and only baidu/Unlimited-OCR on AKS.
💻 The production OCR code is open-source. Support our work by dropping a friendly ⭐ on the repo!
Ready? Let's go!
How the Lesson 3 gateway fails under load
The Lesson 3 service was a FastAPI app that accepts a base64 payload, decodes it, rasterises any PDF pages with PyMuPDF, encodes the resulting PNGs as data URIs, forwards them to vLLM over the OpenAI-compatible endpoint and returns the assembled Markdown. Two replicas, a liveness probe on /health every ten seconds, one GPU node behind them.
Send it a thirty-page scan and the worker spends the next several seconds decompressing content streams and painting pixels into arrays. That work is pure computation. There is no socket to poll and no descriptor to wait on, so the event loop has nothing to switch to while it runs.
For that whole period the process is unavailable for anything else. Clients streaming tokens from earlier requests stop receiving chunks, new connections sit unaccepted in the kernel backlog, and the liveness probe goes unanswered. After three missed probes kubelet restarts the pod and every request in flight dies with it. The callers retry, the replacement pod comes up, starts rasterising immediately and misses its own probes in turn.
The GPU is idle for all of it. It finishes whatever batch it was given and then waits, because the only component capable of feeding it is being killed and rescheduled.
Two separate problems are tangled together here, and it helps to pull them apart. The restart loop is a probe-configuration problem and you can tune it away by raising failureThreshold. The idle GPU is not tunable, because it follows directly from where the CPU work is being done. Neither problem is well described by "Python is slow".
Why this work is a poor fit for an async runtime
Almost none of the work a document gateway does before the GPU sees anything is I/O, which is the root of the problem.
An incoming request is not a clean list of token IDs. It is a multi-megabyte base64 string that has to be stripped of its data URI prefix and decoded into bytes. Those bytes have to be inspected to find out what they actually are, because the client’s file extension is a suggestion and nothing more. If they turn out to be a PDF, its cross-reference table has to be parsed, its content streams decompressed, and its pages rendered into pixel buffers. Later, when completions come back, the raw text has to be scanned with regular expressions to pull out grounding coordinates.
Every one of those steps is CPU-bound. An async runtime only helps with work that spends its time waiting on something external, and none of this does.
Some of this is fixable in Python, and it's worth being straight about that before arguing for a rewrite.
CPython runs bytecode under the Global Interpreter Lock, so only one thread executes Python instructions at a time. But the two escape hatches are real. Native extensions can release the GIL while they work, and PyMuPDF does exactly that during rendering, so a rasterising worker is not holding the lock the whole time. And FastAPI runs plain def handlers on a threadpool rather than on the event loop, so a synchronous handler doesn't block the loop the way an async def one does.
What those escape hatches do not give you is bounded behaviour under load. You can keep the loop responsive by moving work to threads, and now you have a threadpool whose size you have to tune against a GPU whose throughput you don't control. You can add processes, and now each one carries its own interpreter and its own copy of the heap. You can raise the liveness probe's failureThreshold until the restarts stop, and now genuinely dead pods stay in the service for two minutes.
Each of those fixes is reasonable on its own. Collectively they are a set of workarounds for a runtime that was not designed to hold megabytes of binary data in flight while remaining responsive.
Rendering ten pages at 300 DPI into uncompressed RGB produces something on the order of a hundred megabytes of short-lived buffers, and in Python every intermediate slice, dictionary lookup and regex match on top of that is a separate heap object with its own header. A small integer object is 28 bytes. A page is millions of them.
The allocator fragments. Eventually the runtime stops the world to trace what is still reachable, sweep what isn't, and compact what's left. On our service those pauses landed somewhere between fifty and a few hundred milliseconds, which for a REST API shuffling small JSON objects would be invisible, and which for a client watching tokens arrive one at a time looks like the connection stalling.
Measure this on your own service before rewriting anything rather than taking our figures. Two graphs are enough: resident set size per replica under concurrent load, and the interval between successive SSE flushes. Regular spikes in the second are usually the collector.
What Rust changes
Rust gives you the execution model and memory control of C alongside a type system that rejects most of the mistakes C permits. Three properties matter for a gateway.
Zero garbage collection via compile
Rust decides when memory is released at compile time rather than at runtime. A ten-megabyte byte vector is owned by one binding, that binding has a scope, and the compiler emits the deallocation where the scope ends. This is RAII, and it is the same mechanism that will later clean up our temporary directories without any explicit cleanup code.
There is no tracing phase and no pause during which threads stop so the runtime can determine what is still reachable. One clarification, because it is a common source of confusion when reading dashboards: memory returns to the allocator, not to the operating system, so resident set size will not necessarily fall the moment a buffer is dropped.
Multi-threaded async scheduling
Tokio schedules asynchronous tasks across a pool of OS threads and lets idle threads steal queued work from busy ones. That is a real difference from a single event loop, and it is why a Rust gateway can keep answering health probes while a PDF is being rendered.
It is worth being precise about the scheduling model, though, because the comparison tables you'll find online often call it preemptive and it isn't. Tokio is cooperative: a task yields at an .await point and nowhere else. A task that spends four seconds in a computation with no awaits holds its worker thread for four seconds, and work-stealing cannot reclaim it. This is the one place where the Python failure above can be reproduced in Rust and still compile, and we run into it later when we get to PDF rendering.
Thread safety checked at compile time
The type system tracks which parts of a program can reach which memory and whether they may write to it. If safe Rust compiles, it is free of data races and dangling pointers. Two caveats: unsafe blocks opt out of those checks, and none of it prevents a deadlock.
Rust fundamentals for Python engineers
If you've only written interpreted code, the first week in Rust feels like arguing with the compiler about things you never had to think about. Almost all of that argument is about one question: who owns this, and for how long?
Where data lives
In Python and JavaScript nearly everything is on the heap behind a pointer, tracked by the collector, and you are not expected to care.
Rust makes the split explicit. The stack holds values whose size is known at compile time, that is, integers, floats, booleans, or the fixed-size header of a struct. Allocating there costs a pointer move. The heap holds things that grow, like a String or a Vec<T>; the buffer lives on the heap while a small descriptor of it (pointer, length, capacity) sits on the stack.
Ownership and borrowing
Three rules, and everything else follows from them: every value has exactly one owning variable, there is only ever one owner at a time, and when the owner leaves scope the value is dropped.
In Python, handing a list to another name gives you a second way to reach the same object:
original = ["page1.png", "page2.png"]
alias = original # both names, one list
alias.append("page3.png")
print(len(original)) # 3 — mutating one mutated "both"In Rust, the same assignment moves ownership, and the compiler stops you from using the old name at all:
let original = vec!["page1.png".to_string(), "page2.png".to_string()];
let destination = original; // ownership moves here
// println!("{:?}", original); // compile error: use of moved value
println!("{:?}", destination); // fine: sole ownerThat looks hostile until the first time it saves you. In a gateway, the values being moved around are megabyte-sized buffers, and "who is allowed to mutate this while three other tasks are reading it" stops being a question you answer by reading the code carefully.
When a function only needs to look at data, it borrows a reference instead. An immutable borrow (&T) can be handed out many times at once. A mutable borrow (&mut T) is exclusive: while it exists, nothing else may read or write that memory.
// Reads the bytes without taking ownership and without copying
fn page_count_hint(pdf_bytes: &[u8]) -> usize {
pdf_bytes.len()
}
// Appends in place, exclusively, no reallocation of the caller's string
fn append_batch_separator(document: &mut String) {
document.push_str("\n\n---\n\n");
}Enums that carry data, and matching on them
Python models state with strings, dicts, or class hierarchies. Rust uses structs for data and enums for state, and a Rust enum variant can carry its own payload:
#[derive(Debug, PartialEq, Eq)]
pub enum DocumentType {
Pdf,
Image(String), // the MIME type we detected, e.g. "image/png"
Unknown,
}You take them apart with match, and the compiler refuses to compile a match that doesn't handle every variant. When we add WebP support later, every place that inspects a document type becomes a compile error until we've thought about it:
fn route_document(doc: &DocumentType) -> &'static str {
match doc {
DocumentType::Pdf => "rasterise, then batch",
DocumentType::Image(_) => "single-image path, crop mode",
DocumentType::Unknown => "reject",
}
}No null, and no exceptions
There is no None, null or undefined in safe Rust, which removes an entire family of runtime failures. A value that might be absent is an Option<T>:
let requested_batch: Option<usize> = Some(4);
let batch_size = requested_batch.unwrap_or(4);An operation that might fail returns a Result<T, E>, and rather than wrapping call sites in try/except you propagate with ?, which returns the error to the caller immediately if there is one:
// bytes is the decoded buffer on success; on failure this function returns the error
let bytes = clean_and_decode_base64(&payload.file)?;That single character is most of why Rust error handling reads pleasantly once you're used to it. It also means the error type has to be something real, which we'll come back to when we get to HTTP status codes (because that is the one place our first version cheated).
The model behind the gateway
Before designing the gateway it's worth being specific about the thing on the other side, because this model has opinions and the cost of ignoring them is a gateway that looks like it works.
We are serving baidu/Unlimited-OCR, released in June 2026 under MIT and described in arXiv 2606.23050. It sits in the DeepSeek-OCR lineage discussed in Lesson 2, sharing the gundam vision stack — a SAM-ViT-B plus CLIP-L DeepEncoder — and it adds Reference Sliding Window Attention, which is what the "unlimited" in the name is about. Its pitch is one-shot long-horizon parsing: hand it several pages in a single request and let the model keep them coherent, rather than chunking page by page and stitching the Markdown afterwards.
Two numbers matter for everything that follows. It is 3B parameters in BF16, about 6.8 GB of weights, and its context window is 32,768 tokens.
The 6.8 GB is worth sitting with for a second, because it changes how you should think about the GPU bill. vLLM's own recipe for this model says a single card with 8 GB of VRAM is enough for BF16 inference. If you have been sizing OCR nodes on the assumption that vision models are enormous, they aren't — this one fits on hardware most teams already have, as long as it's Ampere or newer for native BF16.
The required serving configuration
This model ships without a chat template and is trained for a specific prompt and decode setup. Get it wrong and it does not error; it returns nothing, or it loops on coordinate tokens until it hits your token ceiling. The official vLLM recipe spells out four requirements, and all four have to be honoured by whatever sits in front of it:
The server has to register the model's no-repeat-ngram logits processor.
Every prompt has to begin with a literal
<image>marker.Every request has to set
skip_special_tokenstofalse.And each request has to pass the processor's own arguments,
ngram_sizeof 35 with awindow_sizeof 128 for single images and 1024 for multi-page input.
The third requirement deserves particular attention. Special tokens are what the grounding information is made of, so leaving skip_special_tokens at its default means vLLM strips every <|det|> and <|ref|> marker before the response leaves the server. The Markdown that comes back looks almost correct, the bounding-box array is empty, and nothing logs a warning. If boxes come back empty on a page that clearly has structure, check this flag before you go looking at your regex.
The launch command is not vllm serve with defaults. The architecture isn't in a stable pip wheel yet, so it's served from the dedicated image, prefix caching is turned off because OCR requests share no prefixes worth caching, and the logits processor is registered by path:
docker run --rm --gpus all --network host --ipc host \
vllm/vllm-openai:unlimited-ocr \
baidu/Unlimited-OCR \
--trust-remote-code \
--logits_processors vllm.model_executor.models.unlimited_ocr:NGramPerReqLogitsProcessor \
--no-enable-prefix-caching \
--mm-processor-cache-gb 0Crop mode and the cost of batching
One more property, and this is the one nobody tells you until you compare outputs.
A request carrying a single image is processed in gundam mode: the page is cropped into tiles at image_size=640 on top of a 1024-pixel base view. A request carrying several images falls back to base mode, one 1024-pixel view per page, no crops.
That means batch_size is not a free throughput knob. Raising it above one gets you fewer round trips and lets the model reason across consecutive pages, and it pays for that by dropping crop mode, which lowers the effective resolution the model sees per page. On clean laser-printed reports we could not tell the difference in output quality. On a dense financial table set in 8-point type, batching lost us digits. The practical rule is to batch for throughput on ordinary documents and to send pages individually when fine print carries the meaning.
Architecture
The gateway exposes POST /process for a single JSON response and POST /process-stream for Server-Sent Events, and a request moves through four stages.
First it is decoded and identified. The body is parsed into a typed ProcessRequest, the base64 is cleaned and decoded, and the first few bytes of the result are inspected to determine what the payload really is. This costs nanoseconds and happens before any expensive work, which is the whole point of doing it first.
If the payload is a PDF, its page count is checked against a configured limit and then it is rendered to PNGs by pdftoppm in a separate operating system process, writing into a temporary directory.
Those pages are grouped into micro-batches of batch_size, and if there is more than one batch, up to concurrency of them are dispatched to vLLM at once so its continuous batching engine has more than one sequence to work with. The prompt, the window_size and the output ceiling all change depending on whether a request carries one image or several.
Finally the completions are parsed: grounding markers are pulled out into structured boxes and the remaining text is assembled into Markdown, in page order.
