Hi everyone!
This is the second lesson in our six-week course on building a production OCR system.
In Lesson 1 we set out the foundations of Kubernetes. Today's aim is to look in detail at the core workload that is responsible for modern enterprise document pipelines: Optical Character Recognition (OCR) and Visual Document Understanding (VDU).
If you ask a typical software engineer how to handle document ingestion, they'll probably tell you:
"Oh, we just pass the PDF to Tesseract or call a cloud provider's OCR API, get the text string back, and feed it into our RAG vector database."
But, if you've built RAG systems at scale … well, you already know the painful reality:
An agent or RAG pipeline is only as good as the context it consumes.
You're familiar with the adage that garbage in, garbage out? Well, that's exactly what happens in this case. If you give your retriever messy, flat text in which multi-column news articles are turned into single lines, financial tables have their column headers lost, or embedded charts are simply erased and replaced with [IMAGE] placeholders, ... even the future Fable 24 will become a hallucination machine.
Today's document processing is NO LONGER just a simple text-matching utility. It has developed and become more sophisticated, turning into a systems/visual modelling discipline.
The aim of this article is to trace out the full architectural development of OCR over the last ten years, starting with the year 2015 (with heuristic CRNNs) and ending with 2026 (with hybrid VLM routers), and to explain the specific design decisions that underpin our open-source benchmark repository.
💻 The production OCR code is open-source. Support our work by dropping a friendly ⭐ on the repo!
Let's begin.
For many years OCR processed one character at a time, and tools such as the early version of Tesseract, OpenCV template matching, or Sauvola binarization all adhered to about the same method: they first cleaned up the image, divided it into separate character shapes, and then guessed each glyph individually by using hand-written rules or simple classifiers.
It works just fine when applied to a clean scan of a page from a book at a resolution of 300 DPI (dots per inch) using a standard font, but fails when used on the types of documents companies actually work with, for three reasons.
The first point is that genuine documents are in a messy condition in countless ways: they are crooked, stained with coffee, creased, taken in poor light, shadowed because of the phone being used to take the photograph, printed in odd fonts with odd spacing, and also written on by hand. All of these features do not appear in a careful scan.
The other aspect is layout. In actual pages you will find multiple columns, the text arranged in boxes to the side, nested sidebars, notes floating above the header, and reading orders which do not simply go from left to right and top to bottom. A character by character examination has no means of following them.
The third point is that structure has meaning. In a financial table, an invoice, or a receipt, the significance lies in the position of a number on the page. If you convert a five-column table into a single continuous stream of characters, the rows and columns vanish along with the meaning.
Modern systems adopt a different method, usually referred to as Visual Document Understanding or VDU. Rather than searching for individual letters, the system considers the entire page as a single continuous image, with the text, the typography, the layout, the borders of tables, the equations, and the charts all forming part of the same signal and not being separable.
The objective has shifted from that of transcription to one of translation: converting the disordered information on a two-dimensional page into a more structured format, typically Markdown or JSON, in such a way that the meaning is preserved so that later-running LLM agents and vector indexers can read it without losing the context.
The early Deep Learning Era (2015 - 2019)
A breakthrough in deep learning document recognition occurred roughly between 2015 and 2019. Prior to this, neural networks required training data in which each character was labeled with a bounding box, which meant that a great deal of work had to be carried out by human annotators. This process was slow and tedious.
Shi, Bai and Yao introduced the Convolutional Recurrent Neural Network (CRNN) at that time, it being the first approach capable of being trained end to end in order to recognize a whole sequence at once.
How CRNN is built?
The work is divided by CRNN into two stages: one section looks and the other reads.
The portion responsible for image analysis is a convolutional network, for example something similar to VGG or ResNet. If you provide it with a narrow strip of an image showing a single line of text, it will return a compressed version of that strip in which all the visually relevant features, such as the strokes, edges, and curves, have been identified.
The bridge between the two stages then appears. The compressed strip is cut into thin vertical columns from left to right, just as you would cut a loaf of bread. Each slice is narrower than a character, so one character usually covers several of the columns. The image is now in the form of a sequence.
The reading section consists of a stack of bidirectional LSTMs, which simultaneously scans the slices in both directions (one from left to right and the other from right to left). This means that whenever it makes a judgment regarding any particular slice, it already has information about what came before and what comes after. A half-blurred letter is considerably easier to identify when you can see its neighbors.
If you think about it, this process is very similar to speech recognition. I mean, speech is a constant flow. People speak quickly and then slowly, they pause at odd times, and their pitch varies.
A transcription model has to produce the correct words without being told precisely when each sound begins and ends.
A single line of text is the same kind of puzzle, but instead of stretching it out over time it is laid out across a page. Some of the letters are narrower while others are wider and in certain cases the letters are pressed together. The model has to produce the correct characters without being told exactly which pixels belong to which letter.
When it had been realised that the two problems were the same shape, the solution from speech could be directly applied to OCR. And that's how CTC enters the picture.
CTC: Connectionist Temporal Classification
The solution that was borrowed from speech recognition is known as Connectionist Temporal Classification or CTC and was published by Alex Graves and his colleagues in 2006; it was this that enabled people to stop drawing boxes around individual letters.
The idea is that the network makes an estimate for each individual slice, and in addition to the usual characters it can also guess a special "nothing here" symbol, known as the blank. For example, with the word "cat" it could generate something such as "c c blank a a blank t", or "c blank blank a blank t t", or a number of other variations depending on the width of the letters.
The same two rules are used to deal with all of them: first squeeze together any adjacent repeats and then discard the blanks. Each of those messy guesses is transformed back into cat.
Incidentally, it is the
blankthat enables this procedure to work; it is the way the model indicates a true double letter, sincel blank lremains asll, whereasl lbecomes a singlel.
And here is the part that actually solves the problem. Nobody knows which of those messy sequences is the "right" one, and the model does not need to pick. During training it adds up the probability of every sequence that would clean up into cat, and pushes that combined total higher. The model is free to spread each letter across however many slices it likes, as long as the final answer comes out correct.
You might think that it would be hopeless to add up all the possible sequences since there are incredibly many of them; but there is an efficient algorithm which divides the work among the overlapping possibilities, so that the total sum is easy to compute.
The entire breakthrough consists in the fact that if you give the model a segment of text that has been cropped and the words it is supposed to produce, and nothing more, it will work out the alignment on its own and there will thus be no longer any need for the people who previously had to draw boxes around each individual letter.
The Transformer Shift (2020 - 2023)
Although CRNN had line-level recognition for a long time, it came with an inherent speed limitation. Since LSTMs processed a sequence one step at a time and each step relied on the previous one, training could not be carried out over the entire line at the same time as modern hardware requires. Long lines also led to a second issue: the farther apart two characters were, the weaker the signal linking them, until the early part of the line no longer had any influence on the later part.
From 2020 to 2023, Transformers replaced nearly all of that.
Introducing the Vision Transformer (ViT)
In 2020 Dosovitskiy and his colleagues came out with a paper having the memorable title An Image is Worth 16x16 Words and demonstrated that the convolutional network could be completely discarded.
The idea is rather simple in a basic way: take the image of the document and divide it up into a grid of small square tiles, typically 16 by 16 pixels, with no overlaps. Convert each tile into a list of numbers and then reduce that list to a fixed length so that each tile becomes a single token. Include a special token at the beginning to enable the model to summarise the entire image, and attach to each tile details regarding its position on the page, since otherwise the model would not know whether a tile was in the top-left or bottom-right area. Finally, feed all the tiles into a standard Transformer and let the mechanism of attention decide which tiles are important for each other.
There are no convolutions and no sliding filters. Just tiles being handled in the same way as words in a sentence.
TrOCR: Transformers on both ends
In 2021 Microsoft went all the way with TrOCR by getting rid of both the CNN and the LSTM.
The encoder is a Vision Transformer that has already been pre-trained and which converts a cropped line of text into tile embeddings, and the decoder is a text Transformer that has also been pre-trained, belonging to the same family as RoBERTa or BART, writing out the answer one token at a time by referring to what the encoder saw. Since both parts were already trained (one on images and the other on language) TrOCR was able to start from a significantly better position than if the model had had to learn everything from scratch. It achieved better results than the previous best ones on both printed and handwritten text lines.
However, it inherited the actual limitation of CRNN. Although TrOCR processes one line at a time, another system has to identify those lines and extract them, typically using a separate detector such as CRAFT or DBNet. The pipeline was simply made shorter, not eliminated.
Donut, and the case against pipelines
When NAVER CLOVA launched Donut in 2022, the change that it caused was greater than simply improving accuracy.
Detection, cropping, recognition and parsing: four stages, each of which required computing power and each of which had the possibility of failing. Failures build up in only one direction. If the detector fails to detect a block of text or combines two columns into one, nothing that comes afterwards can make up for it since the recognizer never sees what has been lost. It faithfully reads the incorrect crop.
Donut omits the middle stage. Instead, the Swin Transformer looks at the whole page and the mBART decoder produces the answer directly. There is no detection stage, no bounding boxes, and no cropping.
You will already know how Donut handles pages if you know how Whisper deals with audio, since it is based on the same architecture.
Whisper creates a transcript directly from an unsegmented audio recording, which means that it does not begin by dividing the audio into individual words. The output produced by Whisper is determined by a prompt token, such as one that instructs it to transcribe or another that tells it to translate.
Donut accepts an image of a whole document and generates structured output in the form of JSON or Markdown instead of plain text.
It does not begin by identifying the text regions. Once again, a prompt token determines the task, the model being instructed to read the page as a receipt or as an invoice, which in turn determines the structure of what comes out.
In each instance the crude intermediate stage which all people had thought was necessary proved to be optional.
The Modern Frontier (2024 - 2026)
At this stage it ceases to be sensible to regard character recognition, layout analysis, and general AI as distinct fields. The reality is that they have now merged together. The companies that are operating nowadays make use of Vision Language Models (VLMs) which have been designed for documents and have been trained on hundreds of millions of pages. The present situation can be divided into three main types.
The first method involves having a teacher and a student. You train an extremely large, slow and costly multimodal ensemble offline, after which you use it to train a much smaller model (often one with under one billion parameters) which is the one that is actually put into production. The large model itself never handles any requests; it only serves an educational role. Nvidia's Nemotron OCR v2 operates in this manner.
The other approach is for one model to carry out all the tasks. A single vision-language backbone reads the page and generates structured Markdown directly, with no steps in between. DeepSeek-OCR-2 and Chandra OCR 2 use this method.
The third approach is a hybrid one: a highly fast layout parser first scans the page to determine what type of region each section belongs to, before passing each of those regions on to a specialised engine (one for tables, one for equations, etc.). PaddleOCR-VL-1.6 and GLM-OCR are the ones that fall into this category.
Originally, OCR was evaluated according to its error rate. To calculate this, count the number of characters the machine got wrong (whether it substituted the wrong character, omitted one, or created one) and then divide that number by the total number of characters that were actually present. Carry out the same procedure with whole words and you will obtain the word error rate.
On clean synthetic text strips this was a reasonable measure, but in actual production it tells you almost nothing. A model might transcribe paragraphs with 99.5% accuracy and yet be useless, since it had swapped the values in two columns of a financial audit table. All the characters would be correct. The document would then contain a false statement.
Benchmarks such as OmniDocBench and Real-OmniDocBench make use of actual scans and examine five things.
We must first verify that the structure has survived by checking whether the table tags are valid, whether the nested lists are correctly nested, and whether the Markdown is in fact parsed.
Second, whether or not the reading order makes sense. In the case of a page that has multiple columns, did the model read them in an order that a human would normally use, or did it interleave two columns in a way that makes no sense for the LLM to then treat as a single argument?
The third point concerns mathematics: the mathematics must be correct and the equations are checked one token at a time against the correct LaTeX, since a formula which is almost correct is still just wrong.
Even if the charts and figures had been described usefully, there is no text in a bar chart, so the model must explain what the chart means and this explanation is then judged for accuracy.
And finally, regarding affordability (take into account how quickly it runs, how much VRAM it uses, and how many pages you get per dollar of GPU cost) a model that performs best in all of the other four criteria and needs one H100 per document remains unsold.
Now that we have a historical perspective of OCR, let's explore in more detail the 1-stage approach (DeepSeek-OCR-2) and the 2-stage approach (PaddleOCR / GLM-OCR).
2-Stage Strategy
The first architecture family achieves its speed by dividing up two questions: what is on this page and where, and then what it says, and answering them one at a time using different tools.
Take that approach instead, which involves feeding an entire 2048 by 2048 page into a large autoregressive decoder and asking it to deal with all of it. It does work, but most of the page consists of margins and gaps, and you are having to pay premium rates for the model to examine empty space.
Here's how the two-stage version works. A small and fast vision model looks at the page once and draws out the boundaries of each area it detects, identifying each one as either a header, a title, a paragraph, a table, a chart, or a footer. The page is then cut along these boundaries into individual blocks. The blocks are collected into a batch and sent off at the same time to specialized models such as GLM-OCR, with several regions being transcribed simultaneously rather than one after the other.
The heavyweight model always avoids using a pass for whitespace and ceases to be a bottleneck since the processing takes place in parallel.
Why upright rectangles are not good enough
Older types of detector (those based on Faster R-CNN or YOLOv8) specify each region using an upright rectangle defined by four numbers: left, top, right, and bottom. Rotation is not permitted.
It's all right on a flat page, but otherwise it fails. If the pages are folded or viewed at an angle using a phone, or if they're curved at the spine, or if they're fed in a bit crookedly, then imagine a block of text tilted by a few degrees and a rectangle having to fit perfectly square and contain it. The only way to cover the entire block is to enlarge the rectangle, and as soon as it has grown it begins to include strips from the adjacent column. If you then crop it, the transcriber ends up with two columns intertwined, which is precisely what is returned.
Baidu's PaddlePaddle team addressed this in PP-DocLayoutV3 by outlining regions with polygons rather than rectangles. With enough points, the outline can lean with a tilted block or curve along a bent line of text, taking in what belongs and nothing else.
The reason this stage costs almost nothing is that PP-DocLayoutV3 has no decoder. It is encoder only, around 33 million parameters, which by current standards is tiny.
Since there is no need to generate one token at a time, a full high-resolution page is completed in less than 8 milliseconds on a T4, and that is not a more recent card. Within those few milliseconds the model has not only identified all the different regions but also determined the order in which they should be read. When compared with the transcription that comes next, the layout pass is almost imperceptible.
1-Stage Strategy
The other architectural family does the opposite. You give the entire page image to a single specialised vision-language model, with the number of its parameters ranging from under one billion up to three or five billion, and ask it to output structured Markdown directly. This can be done with just one model and a single pass, without any layout stage or cropping.
The design is cleaner as well as one that hits a wall during production, and that wall is memory.
Consider the situation when a Vision Transformer processes a page. The image is divided up into tiles that are 16 pixels by 16 pixels and each of these tiles then becomes a token. A page that is 1024 pixels by 1024 pixels results in 64 tiles going across and 64 going down, making a total of 4,096 tokens. If you increase the resolution or include the high-detail tiles that these models usually use in areas requiring a high level of detail, then a single page typically ends up as five or six thousand visual tokens.
Everything of that kind is there before the model has put out a single word.
The inference then takes place in two stages. In the first stage the model reads the six thousand tokens all at once and establishes its key-value cache, which is the working memory it will refer to when generating text. In the second stage it writes one token at a time, consulting that cache with each token it produces.
It is the reading phase that presents the problem. Although the size of the cache increases as the number of tokens increases, the amount of attention work required to set it up increases with the square of that number, which means that six thousand tokens is not just six times as bad as one thousand. The VRAM gets full and then the model stays there for a noticeable period of time before the first character shows up.
The authors of DeepSeek-OCR-2 deal with this by using a mechanism which they refer to as Visual Causal Flow, and the rationale for this is difficult to contest.
Almost the entire area of a document page contains no information, with the margins being empty and the backgrounds consisting of a single solid colour; the main text repeatedly uses the same small number of characters at the same size. Such features deserve no special status and should not compete with all the other elements for attention, yet a simple tiled grid treats them exactly as if they did.
A compression encoder is placed in front of the decoder and reduces those roughly six thousand tile tokens to about a thousand more dense ones, each of which carries more meaning than the tiles it has replaced; it is only the compressed set that gets to the part of the model which generates text.
Cutting the visual prompt by around 83 percent changes the hardware picture substantially.
The amount of cache memory used per request drops from about 12 GB to less than 2.2 GB, which is the difference between a single request taking over a GPU and several requests sharing it comfortably.
The speed of the reading phase increases by more than four and a half times, which means that the delay before the first output token is mostly eliminated.
The kind of bottleneck involved changes. Previously, the GPU was stuck carrying out the attention calculations on the prompt. Now, the limiting factor is the speed at which the card can transfer the cache between memory while decoding, and this is a far better kind of bottleneck since it is the stage for which hardware has already been designed to handle efficiently. As a result, the throughput per node increases significantly.
Systems Engineering in Production
It isn't until it is being run on actual hardware that any of this becomes relevant; if you deploy a VLM on the GPUs that businesses actually have, for example a T4, an A10G, an L4, or an A100, the simple configuration will leave most of the capacity on those cards unused.
A GPU works best when its tensor cores are given a large, uniform block of matrix multiplication to process; if you provide it with a lot of work of the same shape all at once, it will then run at full capacity.
Observe what takes place when an API gateway takes a 20-page PDF and sends each page through one at a time, treating each page as a separate full-page prompt.
The first page shows genuinely heavy work and causes the tensor cores to be used for a moment. After that the writing begins, one token at a time, and everything changes. The process of generating one token needs almost no arithmetic; what it does require is retrieving the whole cache from VRAM, so the card spends its time waiting for memory rather than carrying out computations. Tensor core usage drops below 15 per cent. The computing hardware, which is the more expensive part, is not being used at all.
It continues to do nothing until the first page is finished since the second page has not yet been sent. Twenty pages, twenty periods during which the card is idle.
The aspect of the hybrid design is entirely concerned with systems engineering and has nothing to do with accuracy.
The regions into which PP-DocLayoutV3 divides a page are not processed in order. Instead, an asynchronous worker pool collects crops from many pages at the same time, so rather than handling a single large sequential job it is holding an increasing number of separate small jobs.
The crops are bundled into batches on the go and then given to an inference engine that has been designed specifically for this purpose, such as vLLM, SGLang, or TensorRT LLM. This engine makes use of continuous batching, meaning that as soon as one crop finishes generating, another waiting crop takes over its position rather than the batch waiting until all the crops have completed their generation. PagedAttention keeps the memory for all these concurrent sequences in order without wasting VRAM on padding.
The result is that decoding ceases to be a period of inactivity, since there is always more work in flight than the card can deal with at one time, tensor core usage remains above 90 per cent rather than falling into the teens and memory bandwidth is used continuously rather than in bursts. The same GPU achieves several times the number of pages per hour.
Next Wednesday in Lesson 3, we'll study how vLLM works from ground up, and provide a clean example of how to deploy a synchronous API for specialised VLMsfor OCR in Kubernetes.
See you in Sunday's live office hours! 🚀
The layout-aware polygon approach you describe for PP-DocLayoutV3 maps almost exactly onto what we see in visual quality inspection on production lines - tilted or curved parts break rigid bounding-box detectors the same way skewed pages break OCR. The bigger parallel is the hybrid 2-stage vs 1-stage tradeoff: we hit the same VRAM/latency wall running one heavy VLM per frame instead of a lightweight detector feeding specialized models. Curious whether you've seen similar throughput gains from continuous batching in real-time inspection pipelines, not just batch document processing.
Great insights Victor! In the 2-step scenario, the biggest bottleneck is how fast your orchestrator feeds the vLLM server with many small crops. That's why you don't need a powerful gpu, but a gpu equipped with enough CPU cores