Loaders, chunking and retrievers with LangChain

Operations finally sent Arjun the real thing: crumb-store-ops-handbook-v7.pdf. Two hundred pages, edited by four different store managers over six years. Some pages were tidy paragraphs. Some were tables of oven temperatures. Page 3 was a scanned photo of a signed fire-safety certificate. Every page ended with the same footer: “Crumb & Co. Confidential | Store Operations Handbook v7 | Page 64 of 200”.

Arjun was in a hurry. He used a PDF library to pull all the text into one long string, then cut it every 500 characters, like slicing a long loaf into equal pieces. Forty lines of code, 410 chunks, done by lunch.

At 4 p.m., a trainee at the Baner store asked the bot: “What’s on the closing checklist?” The bot answered confidently with a three-step checklist: send the end-of-day sales photo to the WhatsApp group, check that the doors are locked, set the alarm. The real checklist has eight steps. Steps 1 to 5, including “move all fresh-cream products to the walk-in chiller”, were missing. The trainee closed the store without refrigerating ₹6,000 worth of cream cakes.

Meera got the waste report the next morning. “The bot told her to lock up and leave,” she said. “It forgot the part about the cakes.”

Priya pulled up the chunk the bot had used. It began: “safe. 6. Send the end-of-day sales photo…” She turned the screen towards Arjun. “You cut the cake with a ruler,” she said, “instead of between the layers.”

Why the checklist lost its first five steps

“Walk me through what your 500-character knife did to page 64,” Priya said.

Page 64 is the start of Chapter 9, “Closing the store”. After extraction it is 896 characters long: a chapter title, a short section on who closes, the eight-step checklist, a short section on leftover stock, and the footer. Arjun’s knife cut at character 500 and character 1,000, no matter what was there. Here is what came out:

Chunk	Characters	Starts with	Ends with
1	0 to 500	“Chapter 9: Closing the store. 9.1 Who closes…”	“…5. Count the cash drawer and seal it in the”
2	500 to 896	“safe. 6. Send the end-of-day sales photo…”	“…Page 64 of 200”

The checklist was sliced between “seal it in the” and “safe”. Steps 1 to 5 went into a chunk that was mostly about who is allowed to close. Steps 6 to 8 went into a chunk that also held leftover stock rules and a footer. When the trainee asked about the checklist, search picked chunk 2, and the model, doing exactly what it was told (“answer only from the context”), reported the three steps it could see.

“The model was faithful,” Arjun said, slowly. “Faithful to a broken chunk.”

“So what should a chunk be?”

He thought about it. “One complete thing. The whole checklist, together. Not half of it glued to half of something else.”

“That’s the rule. Each chunk should be a passage you could hand to a new employee on a sticky note, and it would make sense on its own.” She drew a layered cake on the board. “Recipes come in layers: chapters, sections, paragraphs, lines, words. A ruler ignores the layers. A good knife finds the gap between them.”

Where the analogy breaks: real documents do not always have clean layers. A PDF’s “paragraphs” are sometimes only line breaks where the page ran out of width, and tables have no paragraphs at all. The knife can only follow the layers the loader managed to preserve, which is why loading comes first.

Loaders: getting honest text out of files

A document loader is a small piece of code that reads one kind of source (PDF, Word, web page, Notion, a database table) and returns text in a standard shape. LangChain, a popular open-source Python library of building blocks for LLM applications, ships hundreds of them. They all return the same thing: a list of Document objects, each with two fields.

• page_content: the text.
• metadata: a dictionary of labels, such as the source file and the page number.

That standard shape is the whole point. Once everything is a Document, the rest of your pipeline does not care whether the text came from a PDF, a web page or a spreadsheet. You can swap the loader and keep everything else.

# uv add langchain-community langchain-text-splitters pypdf
from langchain_community.document_loaders import PyPDFLoader

pages = PyPDFLoader("crumb-store-ops-handbook-v7.pdf").load()   # one Document per page

print(len(pages))                        # 200
print(pages[63].metadata)                # includes 'source' and 'page': 63 (counts from 0)
print(len(pages[63].page_content))       # 896
print(pages[2].page_content[:100])       # page 3: '' or almost nothing

You should see 200 documents, page 64 (index 63, because Python counts from zero) with 896 characters, and an empty or nearly empty page 3.

Page 3 is empty because it is a scanned image. PyPDFLoader reads the text layer of a PDF; a photo of a certificate has no text layer. Reading text out of images needs OCR (optical character recognition), a separate tool. The dangerous part is that nothing errors. The page loads “successfully” with no content, and the fire-safety certificate silently never becomes searchable.

So the first habit of anyone who builds RAG for a living: open the loader’s output and read it. Print a few pages from the start, middle and end of each file, and one page that has a table. You will nearly always find something to fix.

For Crumb, Arjun found two things worth fixing straight away: the footer on every page, and the fact that pages had no idea which chapter they belonged to. Both are easy to fix in code before splitting.

import re

FOOTER = re.compile(r"Crumb & Co\. Confidential \| .*? \| Page \d+ of \d+")
CHAPTER = re.compile(r"^Chapter \d+: .+$", flags=re.MULTILINE)

current_chapter = "Front matter"
for page in pages:
    page.page_content = FOOTER.sub("", page.page_content).strip()   # drop the footer
    found = CHAPTER.search(page.page_content)
    if found:                                                         # a new chapter starts here
        current_chapter = found.group(0)
    page.metadata["chapter"] = current_chapter                        # every page remembers it

After this loop, page 64’s text no longer ends in the footer, and its metadata says "chapter": "Chapter 9: Closing the store". Page 65, which continues the chapter without repeating the title, gets the same label because current_chapter carries over.

Following page 64 through a better knife

Chunking is splitting long text into smaller passages that each get their own embedding (a list of numbers that captures the passage’s meaning). Why split at all? Because one embedding is a summary of meaning. A page about who closes, the checklist and leftover stock gets a single blurred vector that is a little bit about everything. A short passage about one thing gets a sharp vector that search can find.

Watch a document being split into overlapping chunks:

The splitter you will reach for most is LangChain’s RecursiveCharacterTextSplitter. It takes two settings.

• chunk_size: the maximum length of each chunk, counted in characters (not words, not tokens).
• chunk_overlap: how much text, at most, the start of one chunk may repeat from the end of the previous one. Overlap is insurance: if a passage must be cut somewhere awkward, the sentences around the cut appear whole in at least one chunk.

Here is page 64, now cleaned (822 characters without the footer), through RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=80):

Chunk	Characters	Contents
1	0 to 200	Chapter title, “9.1 Who closes” and its paragraph
2	202 to 668	“9.2 Closing checklist” and all eight steps
3	670 to 822	“9.3 Leftover stock” and its paragraph

Every chunk is a complete section. The checklist is whole, 466 characters, all eight steps. Same text, same maximum size, completely different result.

“How did it know where the sections were?” Arjun asked.

How the recursive splitter decides where to cut

“Recursive” describes its strategy. It has an ordered list of separators, from the biggest natural boundary to the smallest: a blank line ("\n\n", usually between paragraphs or sections), then a single line break ("\n"), then a space, then, as a last resort, between any two characters.

1. It first splits the text at every blank line. Page 64 breaks into four pieces: the chapter title (28 characters), “Who closes” (170), the checklist (466) and “Leftover stock” (152).
2. It then packs those pieces back together, in order, into chunks as large as possible without going over 500. The title and “Who closes” fit together (200 characters, counting the blank line between them). Adding the checklist would make 668, too big, so a new chunk starts. The checklist alone is 466; adding “Leftover stock” would make 620, over 500, so another chunk starts.
3. If any single piece were longer than 500 on its own, say a 1,200-character paragraph, the splitter would recurse: split that piece at the next separator down (line breaks), and pack again. And so on down the list.

That is “cutting between the layers”: it only uses a smaller boundary when a bigger one cannot do the job.

You may notice that the three chunks above do not overlap at all, even though you asked for 80 characters. That surprises nearly everyone, and it is not a bug. Overlap is built from whole pieces: when the splitter starts a new chunk, it carries over trailing pieces from the previous chunk only if they fit within the overlap. Every piece of page 64 was longer than 80 characters, so nothing carried over. Overlap kicks in when the splitter has to cut inside a long paragraph, at line or word level, which is exactly where you need it.

The twist: the right chunk still lost

Arjun re-indexed with the recursive splitter and asked the question again, printing similarity scores. He expected the checklist chunk to win easily. It did not.

Chunk	Cosine similarity to “What is on the closing checklist?”
1: Chapter title + who closes	0.364
2: The eight-step checklist	0.338
3: Leftover stock	0.109

(Cosine similarity scores how closely two embeddings point the same way: 1 is identical meaning, 0 is unrelated.)

“The checklist chunk doesn’t say ‘closing the store’,” Priya said. “Only ‘9.2 Closing checklist’ and a list of tasks. The chunk that has the chapter title, ‘Closing the store’, looks more about closing. The embedding model doesn’t know they belong together.”

The fix is called contextual headers: before embedding, add a line to the top of every chunk saying where it lives. Arjun prefixed each chunk with “Store Operations Handbook > Chapter 9: Closing the store”. The new scores:

Chunk	Without header	With header
1: Who closes	0.364	0.381
2: The checklist	0.338	0.405
3: Leftover stock	0.109	0.250

The checklist now wins. For a more natural question, “What do I do when I close the store at night?”, the checklist chunk scored 0.668 with the header against 0.563 without. One line of context per chunk, a big jump in retrieval.

Note the honest lesson inside the twist. Good chunking was necessary but not sufficient. Search is a chain, and you only find out which link is weak by printing the scores.

The splitting code, step by step

Step 1: split, and check the result.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=120)
chunks = splitter.split_documents(pages)       # metadata is copied onto every chunk

print(len(chunks))                              # around 500 to 700 for this handbook
print(chunks[300].metadata)                     # still has source, page and chapter

You should see several hundred chunks, and each one still carries the source, page and chapter labels from the page it came from. split_documents copies metadata for you, which is why you added the chapter to the pages first.

Why 800 here, after the 500 in the trace? For the real handbook, Arjun tested both (see “Try it yourself”); 800 kept more multi-paragraph procedures together while staying well under the embedding model’s limit (800 characters is about 130 words, roughly 170 to 200 tokens).

Step 2: add contextual headers.

for chunk in chunks:
    header = f"Store Operations Handbook > {chunk.metadata['chapter']}"
    chunk.page_content = f"{header}\n{chunk.page_content}"

Step 3: embed and store. This uses a free embedding model that runs on your machine, so no API key is needed.

# uv add langchain-chroma langchain-huggingface sentence-transformers
from langchain_chroma import Chroma
from langchain_huggingface import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
vectordb = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    collection_name="ops_handbook",
    persist_directory="./chroma_handbook",      # saved to disk for next time
)

The first run downloads the model and embeds every chunk, which takes a minute or so on a laptop.

Retrievers: one plug that fits every socket

A retriever is anything that takes a question and returns a list of relevant Documents. In LangChain it is a standard interface with one main method, invoke. The most common retriever wraps a vector store, but in the next lesson you will build retrievers that use keyword search, combine several searches, or rerank results. Because they share the same interface, you can swap one for another without touching the code that builds prompts.

Think of it like an electrical socket. Your prompt-building code is the appliance. It does not care what is behind the wall, as long as the plug fits.

retriever = vectordb.as_retriever(search_kwargs={"k": 4})

for doc in retriever.invoke("What is on the closing checklist?"):
    print(doc.metadata["page"], "|", doc.page_content[:90].replace("\n", " "))

You should see four chunks, with the checklist chunk from page 64 near the top. Generation from here is the same as in the pipeline lesson: number the chunks, build a prompt, call Claude.

Two retriever options you will use constantly:

• Metadata filters restrict search to chunks whose labels match, for example search_kwargs={"k": 4, "filter": {"chapter": "Chapter 9: Closing the store"}}. When Crumb adds the HR policies to the same index, a filter on source is how store staff are kept away from the salary bands.
• MMR (maximal marginal relevance), turned on with search_type="mmr", picks chunks that are relevant and different from each other, so four near-identical copies of the same paragraph do not crowd out everything else.

Library APIs change quickly. If a name has moved, check the docs linked below.

Where it breaks

A few weeks of real use taught Arjun the rest of the list.

• Tables. The oven temperature table on page 112 came out of the loader as a stream of numbers with no column names: “Deck oven 1 220 180 12”. Search could not find it, and when it did, the model could not tell which number was which. Options: convert tables to sentences or Markdown before splitting (“Deck oven 1: bread at 220 °C, pastry at 180 °C, 12 minutes”), or use a layout-aware PDF parser.
• Two-column pages. Some loaders read straight across both columns, interleaving unrelated sentences. Spot-check any multi-column page.
• Scanned pages. Empty text, no error. Run OCR, or at least log every page with fewer than, say, 50 characters so you know what is missing.
• Characters versus tokens. LangChain’s character splitters count characters. Other libraries (LlamaIndex, later in this module) count tokens. A chunk_size of 800 means very different things in each.
• Chunk size is a trade-off, not a constant. Small chunks (300 characters) are sharp for short facts but scatter multi-step procedures. Large chunks (1,500) keep procedures whole but blur short facts and risk the embedding model’s length limit. Test with your questions.

Strategies beyond the recursive splitter, for when you need them:

Strategy	How it splits	Use when
Fixed size	Every N characters	Almost never; it is the ruler
Recursive character	Paragraphs, then lines, then words	The default for prose
By headings	At Markdown or HTML headings	Well-structured docs (convert the PDF to Markdown first)
Semantic	Where the topic shifts, measured with embeddings	Long, rambling text; slower and costs more to index

Try it yourself

Find the best chunk size for Crumb’s handbook, or any long PDF you have.

1. Write five test questions whose answers you know, with the page number: two short facts, two multi-step procedures, one involving a table.
2. Build three separate Chroma collections with chunk_size 300, 800 and 1,500 (overlap about 15%), with contextual headers.
3. For each question and each collection, record whether the correct page appears in the top 4.

Show what you will probably find

At 300 characters, short factual questions (“What temperature should the chiller be?”) do well, but procedures get scattered across several chunks and a step or two falls outside the top 4. At 1,500, procedures stay whole but short facts are buried in chunks about several topics, and with all-MiniLM-L6-v2 the end of each chunk is not even embedded. Something between 600 and 1,000 characters usually wins for handbook-style prose. The table question probably fails at every size until you convert the table to sentences, which tells you the problem was loading, not chunking. The exact winner depends on your documents. That is why you test rather than copy a number from a blog post.

Common mistakes

• Cutting by fixed size. It ignores structure and slices lists and rules in half. Start with the recursive splitter.
• Never reading the loader’s output. Empty scanned pages, scrambled tables and repeated footers are invisible until you look.
• Chunks longer than the embedding model can read. The excess is silently dropped. Check the model’s maximum input length.
• Chunks with no context. A checklist that does not say what it is a checklist for can lose to a weaker chunk. Add a header line.
• Relying on the prompt to protect sensitive chunks. Filter by metadata in code so they are never retrieved.
• Dropping metadata. You need source, page and section for citations, filters and debugging.

Recap

The waste report had made Meera nervous, so Arjun explained the fix to her in the store, using an actual cake.

• The loader reads the PDF into text plus labels like the page and chapter. We check what it produced, because scanned pages and tables come out broken without any warning.
• The splitter cuts the text into passages. The old version cut every 500 characters, like a ruler, and sliced the closing checklist in half.
• The new splitter cuts between sections first, then paragraphs, then lines, so each passage is one complete idea.
• Each passage now starts with a line saying which chapter it is from, which helps the search pick the right one.
• The retriever is the plug the rest of the app uses: question in, best passages out. We can upgrade what is behind it without rewiring anything else.

“So the checklist can’t be cut in half any more?” Meera asked.

“Not unless one section is longer than the knife,” Arjun said. “And then it cuts at the line breaks, between steps, with some overlap.”

The next morning a store manager typed “What is SOP-14?”, the code everyone at Crumb uses for the allergy incident procedure. The retriever returned a chunk about oven maintenance. The embedding model had never seen “SOP-14” and had no idea what it meant.

“Meaning-based search is great at meaning,” Priya said, “and bad at exact codes. Time to bring back the keyword search you threw away, and make the two work as a team.”

Go deeper

• LangChain documentation: https://python.langchain.com
• Chroma documentation: https://docs.trychroma.com
• Hugging Face documentation (Sentence Transformers models): https://huggingface.co/docs