LlamaIndex fundamentals

On Saturday, Arjun went to an AI meetup in Baner. The speaker built a chatbot over a folder of company PDFs, live on stage, in about fifteen lines of Python. Someone asked how. “LlamaIndex,” she said, as if that explained everything.

On Sunday night Arjun tried it. He dropped the store operations handbook and the HR policies into a folder called crumb_docs, copied the fifteen lines from the meetup slides, and ran them. It worked first time. “What’s on the closing checklist?” came back with all eight steps.

On Monday morning he showed Priya, half proud and half deflated. “Fifteen lines. My pipeline is four files and two weeks of work. Did I waste my time?”

“Run your golden set on it,” Priya said.

He did. Faithfulness fell from 0.93 to 0.81. Correct “I don’t know” answers fell from 8 out of 8 to 4. SOP-14 was wrong again. And then he noticed something worse in his billing dashboard: a few hundred calls to OpenAI’s API from Sunday night. He had never written a line of code that called OpenAI.

Priya smiled. “You didn’t waste two weeks. You spent two weeks learning what those fifteen lines do, so now you can see what they did without asking you. Let’s open the box.”

What the fifteen lines did without asking

Here are the meetup lines, the way Arjun ran them on Sunday:

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex

documents = SimpleDirectoryReader("crumb_docs").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
print(query_engine.query("What's on the closing checklist?"))

“Point at each line,” Priya said, “and tell me which step of the pipeline it is. You know them all.”

Arjun went slowly. “SimpleDirectoryReader… that’s the loader. from_documents must be chunking, embedding and storing, all at once. as_query_engine is retrieval plus the prompt plus the LLM call.”

“Good. Now, which embedding model did it use?”

He looked. Nothing in the code said. “…I don’t know.”

“Which LLM wrote the answers?”

“Claude? No. I never told it Claude.” He looked at the billing dashboard again. “OpenAI.”

“So frameworks are bad?” Arjun asked.

“No. Frameworks are good at the boring 80%. The trap is believing the defaults were chosen for you.” Priya drew a box on the whiteboard. “Think of moving into a furnished flat. Bed, sofa, kitchen, all there on day one. Lovely. But someone else chose the mattress. If you never check, you’ll sleep badly for a year and blame yourself.”

Where the analogy breaks: in a flat you can see the furniture. In a framework, the defaults are invisible until you look them up or print them. Every serious LlamaIndex user learns to make the important choices explicit in code, even when the default happens to be fine, so the next reader can see them.

The five building blocks, named

A framework is a library that gives you a ready-made structure to fill in, instead of loose tools you assemble yourself. LlamaIndex (it began life in 2022 as “GPT Index”) focuses on one job: getting your data into a shape an LLM can use, and getting the right pieces back out at question time. Here is how its vocabulary maps onto the pipeline you built by hand.

flowchart LR
  A["Reader (data connector)"] --> B["Documents"]
  B --> C["Node parser"]
  C --> D["Nodes"]
  D --> E[("Index")]
  E --> F["Retriever"]
  F --> G["Query engine"]
  G --> H["Response with sources"]

1. Readers, also called data connectors. A data connector reads one kind of source and returns documents. SimpleDirectoryReader reads a whole folder of PDFs, Word files, Markdown and CSVs, choosing the right method by file extension. Hundreds more (Notion, Google Drive, Slack, databases) are available as separate packages through LlamaHub, LlamaIndex’s catalogue of integrations. This is the same idea as LangChain’s loaders.

2. Documents. A Document is the text of one source, or one page of a PDF, plus metadata (labels such as the file name and page label).

3. Nodes. A node is LlamaIndex’s word for a chunk: a short passage that gets its own embedding (the list of numbers that captures its meaning). A node parser (LlamaIndex’s name for a splitter) makes nodes from documents. Unlike a plain chunk, each node remembers its relationships: which document it came from, and which nodes come immediately before and after it. That lets LlamaIndex do things like fetch a neighbouring node for extra context when a match lands at the edge of a section.

4. Indexes. An index is a data structure built over nodes that makes them findable. The one you will use nearly all the time is VectorStoreIndex: it embeds every node and stores the vectors so the nearest ones can be found quickly. Others exist for special jobs. SummaryIndex, for instance, does not search at all; it walks through every node, which suits “summarise this whole policy” requests where picking the top few chunks makes no sense.

5. Retrievers and query engines. A retriever takes a question and returns the most relevant nodes, each with a similarity score. A query engine wraps three parts: a retriever, optional node postprocessors (steps that filter or rerank the retrieved nodes), and a response synthesiser (the part that builds the prompt and calls the LLM). You ask a query engine a question; you get back an answer and the source nodes it used.

Following one question through LlamaIndex’s machinery

Priya had Arjun re-run Sunday’s demo with printing turned on at every stage, to see the defaults with their real values. Here is the trace for “What’s on the closing checklist?”.

Reading. SimpleDirectoryReader found two PDFs and returned 230 Document objects: one per PDF page (200 handbook pages plus 30 HR pages). Page 64’s document had metadata like {"file_name": "crumb-store-ops-handbook-v7.pdf", "page_label": "64", ...}. Note that LlamaIndex used the printed page label, “64”, where LangChain’s loader gave a zero-based index of 63. Same page, different convention: check before you cite.

Parsing. With no node parser specified, from_documents used the default SentenceSplitter with a chunk size of 1,024 tokens (the word-pieces models read; roughly 750 English words) and an overlap of 200 tokens. Most handbook pages are shorter than that, so most pages became exactly one node. Result: about 240 nodes, many of them whole pages about several topics. That is the blurry-vector problem from the chunking lesson, back again.

Embedding and storing. Each node’s text went to the default OpenAI embedding model, and the vectors were kept in memory. Nothing was saved to disk, so every run of the script re-embedded all 240 nodes. That accounted for many of the mystery API calls.

Retrieving. The query engine’s retriever embedded the question and returned the top 2 nodes, the default similarity_top_k. For the checklist question, the top node was page 64 (score about 0.82 with OpenAI’s embeddings; different models produce different score ranges, so do not compare these with the MiniLM scores from earlier lessons).

Synthesising. The response synthesiser, in its default compact mode, stuffed both nodes into one prompt using LlamaIndex’s built-in question-answering template. That template reads, in essence:

Context information is below.
---------------------
{the two retrieved nodes}
---------------------
Given the context information and not prior knowledge, answer the query.
Query: What's on the closing checklist?
Answer:

A decent general prompt. But it has no “if the answer is not there, say exactly this” rule, no citation instruction, and nothing about Crumb. That one template explained most of the drop in abstentions.

The response. A Response object: str(response) is the answer text, and response.source_nodes is the list of the two nodes it used, each with its score and metadata.

“So it’s the same six steps as my version,” Arjun said. “Just with somebody else’s choices at every step.”

“And now you know where each choice lives, so you can make your own.”

Opening the hood: how a query engine turns nodes into an answer

The response synthesiser has a response_mode setting, and it changes how many LLM calls a question costs. Suppose the retriever returns 6 nodes.

• compact (the default) packs as many nodes as fit into each prompt. Six short nodes usually fit in one prompt: one LLM call.
• refine sends the first node and asks for an answer, then sends the next node with the answer so far and asks the model to improve it, and so on: six LLM calls, one after another. Thorough, slow and about six times the cost.
• tree_summarize answers from groups of nodes, then summarises those answers, layer by layer, like a knockout tournament. Useful for “summarise everything about leave policies” across many nodes; overkill for a single fact.

For Crumb’s short factual questions, compact is right. The point is to know that “query engine” is not one fixed thing: it is a small pipeline whose cost can vary by a factor of six depending on one setting.

Building it properly, one step at a time

Here is the same system with every important choice made explicitly. Each step runs on its own.

Step 1: choose the models.

# uv add llama-index llama-index-llms-anthropic llama-index-embeddings-huggingface python-dotenv
from dotenv import load_dotenv
from llama_index.core import Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.anthropic import Anthropic

load_dotenv()                                           # ANTHROPIC_API_KEY
Settings.llm = Anthropic(model="claude-opus-5-5", max_tokens=1000)
Settings.embed_model = HuggingFaceEmbedding(
    model_name="sentence-transformers/all-MiniLM-L6-v2"  # free, runs locally
)

Settings is LlamaIndex’s global configuration object. Anything you do not pass explicitly to a later function is taken from here. Setting both models at the top of the file means no silent fallback to OpenAI, ever.

Integration packages like llama-index-llms-anthropic sometimes lag behind brand-new model names. If you get an “unknown model” error, upgrade the package. If that does not help, use LlamaIndex for retrieval only (step 5) and call Claude yourself with the Anthropic SDK.

Step 2: read and inspect.

from llama_index.core import SimpleDirectoryReader

documents = SimpleDirectoryReader("crumb_docs").load_data()
print(len(documents))                                   # 230
print(documents[63].metadata)                           # file_name, page_label "64", ...
print(documents[63].text[:200])

You should see 230 documents and page 64’s text. Same rule as with any loader: read the output, look for empty scanned pages and scrambled tables.

Step 3: parse into nodes, and label them.

from llama_index.core.node_parser import SentenceSplitter

parser = SentenceSplitter(chunk_size=256, chunk_overlap=32)   # TOKENS, not characters
nodes = parser.get_nodes_from_documents(documents)

for node in nodes:
    is_hr = "hr" in node.metadata.get("file_name", "").lower()
    node.metadata["department"] = "hr" if is_hr else "operations"

print(len(nodes))                                        # roughly 700
print(nodes[200].metadata)                               # file_name, page_label, department

SentenceSplitter tries to keep sentences whole, similar in spirit to LangChain’s recursive splitter. The units differ: 256 tokens here is roughly 1,000 to 1,200 characters, and it fits inside the 256-token limit of the MiniLM embedding model. The department label is your own metadata, which you will use to keep HR questions in HR documents.

Step 4: build the index and save it.

from llama_index.core import StorageContext, VectorStoreIndex, load_index_from_storage

index = VectorStoreIndex(nodes)                          # embeds every node
index.storage_context.persist(persist_dir="./storage")   # write to disk once

# In app.py, later: load instead of re-embedding
index = load_index_from_storage(StorageContext.from_defaults(persist_dir="./storage"))

Indexing happens once; the app loads the saved index in a second or two. This alone removed most of Sunday’s mystery API calls.

Step 5: a retriever, on its own.

retriever = index.as_retriever(similarity_top_k=8)
for result in retriever.retrieve("What is SOP-14?"):
    print(round(result.score, 3), result.node.metadata["page_label"],
          result.node.get_content()[:70].replace("\n", " "))

Each result is a node with its score (higher is more similar). From here, you could take the node texts and run your own hybrid search, reranking, abstention rules and Claude call, exactly as in the previous lessons. Many teams use LlamaIndex this way: for loading and indexing, with their own code for the answer.

Step 6: a query engine with Crumb’s prompt and a reranker.

from llama_index.core import PromptTemplate
from llama_index.core.postprocessor import SentenceTransformerRerank, SimilarityPostprocessor

crumb_prompt = PromptTemplate(
    "You answer questions for Crumb & Co. store staff.\n"
    "Context:\n{context_str}\n\n"
    "Answer using only the context. If it does not contain the answer, reply "
    "exactly: I don't know based on Crumb's documents.\n"
    "Question: {query_str}\nAnswer: "
)

query_engine = index.as_query_engine(
    similarity_top_k=20,                                   # cast a wide net
    node_postprocessors=[
        SimilarityPostprocessor(similarity_cutoff=0.3),    # drop weak matches
        SentenceTransformerRerank(model="cross-encoder/ms-marco-MiniLM-L-6-v2", top_n=4),
    ],
    text_qa_template=crumb_prompt,                         # replace the default prompt
)

response = query_engine.query("What's on the closing checklist?")
print(response)
for source in response.source_nodes:
    print(round(source.score, 3), source.metadata["page_label"])

That is the “retrieve 20, rerank to 4” pattern from the hybrid search lesson, plus your abstention rule, in about fifteen lines. {context_str} and {query_str} are placeholders LlamaIndex fills in. Note that after reranking, source.score is the cross-encoder’s score, not the cosine similarity.

Arjun re-ran the golden set with this version: faithfulness 0.92, correct abstentions 7 of 8, SOP-14 still shaky. That last one makes sense: there is no keyword search in this pipeline. LlamaIndex does offer BM25 and fusion retrievers as add-on packages; he decided to keep his own hybrid search code instead, which the next lesson discusses.

To store vectors in a real vector database instead of local files, LlamaIndex has integrations for Chroma, Pinecone, pgvector and others: you create the store, wrap it in a StorageContext, and pass that to VectorStoreIndex. The rest of your code does not change.

Library APIs change quickly. If a name has moved, check the docs linked below.

Where it breaks

• Silent defaults. The default LLM, embedding model, chunk size, top-k and prompt are all reasonable for a demo and probably wrong for you. Set them explicitly at the top of the file.
• Invisible prompts. When an answer is strange, you need to see the exact prompt that was sent. LlamaIndex has debugging and tracing hooks (and integrates with observability tools like Langfuse, which you will meet in Module 22). Turn one on before you need it.
• Version churn. LlamaIndex is split into a core package and many small integration packages, and imports have moved between versions. Pin versions in your lock file and upgrade on purpose.
• Units. Token-based chunk sizes here, character-based in LangChain’s common splitters. Copying “800” between them gives very different chunks.
• A framework is not an evaluation. It wired the pipeline. Only the golden set and RAGAS can tell you if it is any good.

Try it yourself

Using the step-by-step version:

1. Build the index twice, with chunk_size=256 and chunk_size=1024.
2. Ask both: “List every step of the closing checklist.” Count the source nodes and the number of checklist steps in each answer.
3. Add a metadata filter so HR questions only search HR nodes. Hint: look up MetadataFilters in the docs.

from llama_index.core.vector_stores import ExactMatchFilter, MetadataFilters

hr_engine = index.as_query_engine(
    filters=MetadataFilters(filters=[ExactMatchFilter(key="department", value="hr")]),
)
print(hr_engine.query("How many casual leaves do I get per year?"))

Common mistakes

• Shipping the defaults. Set the LLM, embedding model, chunk size, top-k and prompt yourself.
• Leaving extra API keys lying around in .env. They let defaults “work” silently with services you did not choose.
• Rebuilding the index on every run. Persist it, or use an external vector database.
• Ignoring source_nodes. They are your citations and your best debugging tool.
• Mixing up token and character chunk sizes when moving between libraries.
• Assuming the framework improved quality. Measure with the same golden set you used before.

Recap

Meera wandered over while Arjun was re-running the golden set. “So did the fifteen lines replace your two weeks?”

He explained it to her the way Priya had explained it to him.

• LlamaIndex is a framework that already has a slot for every step we built by hand: reading files, cutting them into passages (it calls them nodes), indexing them, finding the best ones, and asking the AI.
• Out of the box, it makes every choice for you, including which AI company to use. Ours quietly used a different one, because a key was lying around.
• Its default instructions to the AI didn’t include “say I don’t know”, so the bot started guessing again.
• Once we set our own models, chunk size, instructions and reranking, it scored about the same as my hand-built version, with much less code.
• It doesn’t make the answers smarter. It makes the plumbing shorter. The tests still decide if it’s good.

“So which one do we keep?” Meera asked. “Yours, or theirs?”

Priya looked up from her desk. “That’s a real decision,” she said, “and people on the internet argue about it a lot more than they should.”

Go deeper

• LlamaIndex documentation: https://docs.llamaindex.ai
• Hugging Face documentation (embedding and reranker models): https://huggingface.co/docs