Pixel-Native RAG: Build a Visual Document Search Pipeline
A practical guide to building a pixel-native RAG pipeline that retrieves documents as images using SigLIP, FAISS, BM25, and reciprocal rank fusion. No text parsing needed.
A new tutorial from Marktechpost walks through building a pixel-native retrieval-augmented generation (RAG) pipeline entirely from scratch. Instead of extracting text from HTML or PDFs, the system renders documents as images, splits them into overlapping 1024×1024 pixel tiles, embeds those tiles with SigLIP or CLIP, and stores the vectors in a FAISS index. OCR-based BM25 scoring and reciprocal rank fusion improve retrieval quality, and a FastAPI endpoint exposes everything as a search service.
What happened
| Component | Detail |
|---|---|
| Tile size | 1024 x 1024 pixels, 128-pixel overlap |
| Default embedding model | google/siglip-base-patch16-224 |
| Optional backends | CLIP, Qwen3-VL-Embedding-2B |
| Vector store | FAISS (IVF threshold 2000, nprobe 16) |
| Sparse retrieval | OCR + BM25 via rank-bm25 |
| Fusion method | Reciprocal rank fusion (k=60) |
| API layer | FastAPI on port 8000 |
| Evaluation metrics | Recall@k, mean reciprocal rank |
| Optional VLM answer step | Qwen/Qwen2.5-VL-3B-Instruct |
The tutorial addresses a real limitation in standard RAG systems: they rely on clean text extraction, which fails badly on scanned PDFs, tables, diagrams, and visually structured pages. The pixel-native approach skips that dependency entirely by treating each page as a screenshot.
Pages are captured using a headless Playwright Chromium browser. Each page is capped at 24,000 pixels tall, split into tiles with a 128-pixel overlap so content at tile boundaries is not missed, and any blank tiles (detected by a standard deviation threshold of 6.0) are discarded. Near-duplicate tiles are also removed using Hamming distance, keeping the index lean.
How the retrieval stack works
Each tile is passed through a vision encoder (SigLIP by default) to produce a dense embedding vector. Those vectors go into a FAISS index for approximate nearest-neighbour search. At query time the system also runs OCR on tiles and scores them with BM25, a classic sparse text-matching algorithm. The two ranked lists are then merged using reciprocal rank fusion, which combines rankings without needing to tune score scales between the two systems.
Tile-level results are aggregated up to document-level results, so the final output is “Document X is the best match” rather than “Tile 7 of Document X is the best match.” The top tiles from the winning documents can optionally be passed to Qwen2.5-VL-3B-Instruct to generate a grounded natural-language answer.
Training and evaluation
The pipeline includes a lightweight residual adapter that can be trained with contrastive learning to improve embedding quality for a specific document corpus. Seven evaluation queries are baked in, each paired with a known source document, covering topics from photosynthesis to transformer attention mechanisms. Retrieval quality is measured by Recall@k and mean reciprocal rank (MRR), giving an honest signal on whether the system is actually finding the right pages.
For anyone building pipelines that need to retrieve from visual content, this kind of evaluation discipline matters. It is easy to ship a RAG system that looks good on a demo but has no baseline numbers. For context on how businesses are thinking about this, our overview of open-weight AI models for business covers when self-hosted models like Qwen are the right call.
Why it matters
Most document search tools assume clean, extractable text. That assumption breaks across a wide range of real-world documents: architectural drawings, product spec sheets, scanned invoices, slide decks, and any PDF where the visual layout carries meaning. A pixel-native approach handles all of these without custom parsers for each file type.
The hybrid dense-plus-sparse design is also significant. Pure vector search misses exact keyword matches. Pure BM25 misses semantic paraphrases. Combining them with reciprocal rank fusion is a well-established pattern that consistently outperforms either alone. If you are building or evaluating an AI integration that includes document retrieval, this architecture is worth understanding as a baseline.
Our take
The pixel-native framing is genuinely useful, not just a novelty. We have seen clients struggle with RAG over PDFs because their extraction pipeline silently drops tables, headers, and multi-column layouts. Rendering to images sidesteps that entire class of bugs.
That said, the cost trade-offs are real. Embedding 1024×1024 tiles is slower and more memory-hungry than embedding a text chunk. The tutorial defaults to a batch size of 8, which hints at how carefully you need to manage GPU memory at scale. For most small-to-medium document corpora (a few thousand pages), this is totally workable. For millions of pages, you will want to think carefully about tile count per document (the config caps at 12 tiles per doc) and whether a lighter encoder is acceptable.
The optional Qwen3-VL backend is the most interesting path for production use. Vision-language models that also produce embeddings are converging fast, and tying retrieval and answer generation to the same model family should reduce the gap between what the retriever “sees” and what the generator can reason over. Keep an eye on that space. If you want to explore what a system like this could look like for your own content, talk to the Lumien team about scoping a pilot.
What to do about it
- Run the tutorial in a Colab environment to get a working baseline before touching your own documents.
- Swap the default SigLIP model for Qwen3-VL-Embedding-2B if your documents are heavy on mixed text and visuals.
- Enable OCR hybrid mode and confirm Tesseract is installed; dense-only retrieval will underperform on text-heavy pages.
- Check your Recall@k score before deploying. Anything below 0.7 at k=5 warrants tuning the tile overlap or training the residual adapter.
- If passing tiles to the VLM answer step, gate it behind a confidence threshold so low-quality retrievals do not produce confident-sounding wrong answers.
The practical takeaway: if your RAG pipeline is choking on PDFs, rendering pages as images and embedding tiles is a reliable fix worth prototyping this week.
Frequently asked questions
What is pixel-native RAG?
Pixel-native RAG is a retrieval-augmented generation approach that renders documents as images instead of extracting text. Pages are split into image tiles, embedded with a vision model, and searched via vector similarity, making it robust to PDFs, scanned documents, and visually structured layouts that break standard text parsers.
Which embedding models work with this pipeline?
The tutorial supports three backends: SigLIP (google/siglip-base-patch16-224 by default), CLIP, and Qwen3-VL-Embedding-2B. The Qwen backend is optional and suited to documents mixing text and complex visuals.
How does the hybrid retrieval work?
Dense vector search via FAISS is combined with OCR-based BM25 sparse scoring. The two ranked lists are merged using reciprocal rank fusion with k=60, giving a final ranking that benefits from both semantic similarity and exact keyword matching.
How is retrieval quality measured in this pipeline?
The pipeline evaluates retrieval using Recall@k and mean reciprocal rank (MRR) against seven built-in test queries, each mapped to a known source document covering topics like vector databases, transformers, and photosynthesis.
