RAG Preprocessing: Why Clean Markdown Beats Raw PDF Chunks

A retrieval-augmented generation system is only as good as its chunks. Most RAG tutorials start at 'load PDF, split text' and never mention that the split text is broken. That single skipped step is why so many RAG demos shine on clean sample files and fall apart on real ones.

Garbage in, garbage retrieved

If you split a raw PDF straight into chunks, you embed the noise too: broken words, headers landing mid-sentence, tables flattened into unreadable rows. The embedding model encodes that mess, so a well-phrased query retrieves the wrong chunk, or the right chunk arrives too corrupted for the LLM to use. No reranker or prompt tweak fixes bad input.

Why Markdown is a better substrate

• Structure survives. Headings become section boundaries you can chunk on, so a chunk is a coherent unit instead of an arbitrary slice that cuts a sentence in half.
• Tables stay tables. A Markdown table keeps rows and columns aligned, so a question about a number retrieves a chunk where that number still has its label.
• Less noise per token. Removing page furniture means more of each chunk is real content, so embeddings are sharper and you fit more meaning per chunk.

Chunk on structure, not character count

Fixed-size character chunking is the default because it is easy, not because it is good. With clean Markdown you can chunk on headings and paragraphs, keep related content together, and prepend the section heading path to each chunk. Retrieval improves immediately because each chunk becomes self-describing.

A practical preprocessing order

1. Convert the source (PDF, DOCX, XLSX) to clean Markdown, removing boilerplate and fixing broken text.
2. Recover scanned pages with OCR so image-only PDFs are not silently empty.
3. Chunk on headings and sections, and prepend the heading path to each chunk as context.
4. Embed the clean chunks, and keep the source Markdown so you can show citations.

The highest-leverage change

Fixing the input is the single most valuable thing you can do before touching your retriever, reranker or prompt. It is also the least glamorous, which is exactly why it is usually skipped.

You cannot embed your way out of bad input. Clean the document first, and every downstream step gets easier.

PackForAI turns real-world PDFs, Office files and spreadsheets into clean Markdown built for exactly this. Convert one through the API or the app at packforai.com and feed the output straight into your chunker.