Why Flat Text Breaks RAG — and How Markdown-Aware Chunking Fixes It
How preserving document hierarchy transforms retrieval accuracy
When humans read a webpage, we don’t just consume words — we perceive structure. Titles tell us where we are, subheadings group related ideas, and nested lists give us relationships. This hierarchy is what makes web pages intuitive.
Now imagine stripping that away. If you scrape a page’s text without preserving its headings, you’re left with a flat blob of words. To a machine, “Payment Options” looks no different from “Refund Policy.” That loss of hierarchy can seriously degrade the accuracy of downstream tasks like Retrieval-Augmented Generation (RAG), search, and summarization.
The Problem: Flat Text Is Contextless
A human instantly recognizes that “Credit Cards” and “PayPal” are children of “Payment Options.” But to a machine, unless we explicitly encode that relationship, they’re just three unrelated strings. Without structure, a RAG system might retrieve “PayPal” for a query about “Refund Policy” — simply because both sections mention “payment.”
Two Approaches: Plain Text vs. Markdown-Aware Chunking
| Approach | Pros | Cons |
|---|---|---|
| Plain-Text Chunking Split into fixed-size blocks (e.g. 1,000 chars) |
Simple, fast, works without formatting | No hierarchy; mixes unrelated concepts; may cut headings mid-way |
| Markdown-Aware Chunking Uses #, ##, ### to rebuild structure |
Preserves parent-child context; more accurate retrieval | Requires markdown/structure in source data |
Our Solution: Contextual Chunking Pipeline
To overcome context loss, we built a pipeline with four stages:
1. Parse Markdown Headings
- Each heading is analyzed based on the number of
# - Hierarchy rebuilt:
#is top-level,##is nested, and so on - Skipped levels (e.g. jumping from
#to###) are handled gracefully
2. Build Sections with Context + Content
- Each section carries its full heading path, so it always “knows who its parent is”
3. Combine Sections into Chunks
- Small sections share a chunk if they fit within the size limit
- Larger sections stand alone
- Oversized sections are split into sub-chunks, each retaining the same context
4. Output Context-Aware Chunks
- Final chunks preserve both content and structural path, ready for embedding
Why This Approach Works
- Better Retrieval — queries like “What payment methods are available?” correctly surface “Credit Cards” and “PayPal” under “Payment Options”
- Reduced Noise — unrelated content no longer groups together just because it fits a fixed-size chunk
- Scalable Handling of Large Content — long policies, manuals, or docs split into smaller, still-linked chunks
- Improved RAG Accuracy — models perform best on chunks that mirror human-readable sections
Practical Example
Question: “Do you accept American Express?”
❌ Plain-Text Chunking
Retrieves a mixed chunk: “We offer several ways to pay… Refund Policy…” → inaccurate context, risk of hallucination.
✅ Contextual Chunking
Retrieves the correct “Payment Options → Credit Cards” chunk → clear, accurate answer aligned with source.
Conclusion
Scraping web data is easy. Making that data useful is the real challenge. Plain-text chunking is quick, but it strips away the navigational cues humans rely on. Markdown-aware contextual chunking restores hierarchy, letting downstream AI systems reason about data the way humans naturally do.
In practice, this has been transformative for RAG workflows:
- Retrieval is more precise
- Responses are more contextually accurate
- Large documents remain manageable
“Good AI doesn’t just process data. It understands its structure.”